or
or
By clicking below, you agree to our terms of service.
New to HackMD? Sign up
Syntax | Example | Reference | |
---|---|---|---|
# Header | Header | 基本排版 | |
- Unordered List |
|
||
1. Ordered List |
|
||
- [ ] Todo List |
|
||
> Blockquote | Blockquote |
||
**Bold font** | Bold font | ||
*Italics font* | Italics font | ||
~~Strikethrough~~ | |||
19^th^ | 19th | ||
H~2~O | H2O | ||
++Inserted text++ | Inserted text | ||
==Marked text== | Marked text | ||
[link text](https:// "title") | Link | ||
![image alt](https:// "title") | Image | ||
`Code` | Code |
在筆記中貼入程式碼 | |
```javascript var i = 0; ``` |
|
||
:smile: | Emoji list | ||
{%youtube youtube_id %} | Externals | ||
$L^aT_eX$ | LaTeX | ||
:::info This is a alert area. ::: |
This is a alert area. |
On a scale of 0-10, how likely is it that you would recommend HackMD to your friends, family or business associates?
Please give us some advice and help us improve HackMD.
Syncing
xxxxxxxxxx
Data Transfer Stack Design
Background
Problem Definition
A data transfer stack coordinates finding and downloading free or paid content that is hosted one or more locations which support one or more transport protocols.
We have already built several low-level components that make up the data transfer stack. This includes two transport protocols (Bitswap/Graphsync), various content routing mechanisms (Bitswap Want-Have Protocol, IPFS DHT, in progress Filecoin indexer), a control protocol for managing transfers (go-data-transfer), and an initial payment and fair exchange system (go-fil-markets retrieval market/estuary fil-client). We believe we will build more of these components in the future and improve upon existing components.
This document is focused building an extensible architecture for coordinating all these components so users can express high level requests for content and our software can determine the best way to get it. The content may be ultimately be retrieved through one or more individual underlying transfers over one or more protocols from one or more providers.
This is document is designed to help us align on high level vision for the long term. We expect that data transfer stacks we build in the short term will not immediately align with this high level vision, but will move towards this over time.
Sample Use Cases
These are some sample scenarios we might encounter retrieving content on the DWeb. The intent here is to think about desired outcomes.
Assume for all of these use cases that the client does not know ahead of time where any of the data they are downloading is hosted.
Prior Art
go-data-transfer
, despite its name, is NOT an orchestration layer of the type we're discussing here.go-data-transfer
is fundamentally a control protocol that sits on top of a transport – it performs access control logic, restarts, pauses and resumes, and implementing hooks to do payment and exchange. It is not an orchestration layer.Terminology
TODO: Choose a single term for each of these
Client or Requestor- person requesting content
Provider or Responder - person making content available
Transport or Transport Protocol - A single mechanism for transferring content (ex: Bitswap / Graphsync / HTTP)
Transfer - a single request over a single transport protocol between a single client and a single provider
Control Protocol - An abstract protocol that sits on top of a transport to manage the control flow of a transfer – authorizing requests, pausing and resuming, restarting, and communicating with an exchange protocol. The current implementation of a control protocol is go-data-transfer (poorly named)
Exchange Protocol - an abstract protocol that sits on top of a control protocol to manage the fair exchange of data for payment. (current implementation is go-fil-markets retrieval protocol, retrieval client implementation in Estuary)
Content Request or Retrieval Request – a high level request to find content in the network, choose which protocols and peers to retrieve it with, and retrieve it through one or more transfers
Design And Scoping
Top Level Interface
A user downloading content can do the following:
All of these operations effectively compress down to a single operation. A request for a single block is effectively a request for a DAG matching matching a CID and a selector that matches only the first block. And a request for a single CID + selector is effectively creating a session with a single request.
Each operation can take additional parameters to modify policy decisions about where to retrieve content from (i.e. pricing, geography, etc)
Core Agreements
These are some core design principles for the data transfer stack. They define the approach and scope of the core problem we're trying to address.
Three core operations in retrieval of content
Ultimately serving a content request consists of three basic operations:
Asynchronous Operation Flow
The complexity of content retrieval emerges from two places:
However, in practice, the results of later operations flow back into previous operations. A typical content request may require several iterations of finding content, choosing peers, and transfering.
Content Routing
Inputs:
Outputs:
Content routing is the processing of find content through a set of disperate routing mechanisms.
Even in today's go-ipfs data transfer stack we look for content in several places:
Places to look for content tend to sit on a spectrum from "very fast but not a lot of content" to "very slow but lots of content":
Going forward, we can only imagine more places to look for content:
It's an open question if all the places we want to find content will eventually use our future Routing Language, but it's likely we will need to support lots of legacy protocols for some time.
Since there may be a non zero cost to a content routing query, we may want to sequence requests to prioritize places that are likely to respond faster:
Content Routing Coordination vs Content Routing Requests
Content routing really involves two layers operation:
When we think about extensibility, we likely want to provide mechanisms to substitute third party implementations at both layers:
Policy
Inputs:
Outputs:
Transfer
Inputs:
Outputs:
Units Of Transfer
The core units for a single transfer are:
When retrieving a block, providers should send a complete response.
When retrieving a DAG, providers may send only a partial response.
Architectural Kernel
A full data transfer stack will have lots of different components performing all or part of the three core operations involved in retrieving content. Some of these components may need to live out of process or even on another machine. We want these components generally to be pluggable and replacable.
Perhaps the most important design question is how we orchestrate all of these components, and allow them to be pluggable.
Traditional Golang coordination primitives and extensibility mechanisms are insufficient
Golang provides the following mechanisms for coordination among parallel operations:
Golang's primary mechanism of extensibility is interfaces. Interfaces can wrap HTTP APIs to support out of process calls. But they require explicit configuration.
We have used these tools extensively throughout our stack to try to coordinate many parallel processes and components involved in retrieval. Generally they have proven difficult to manage, inflexible to modify or extend, and racy. Most critically, we have no way to insure a single order of operations for a complicated stack of communication between transfer components. Even when we carefully manage synchronization in a single component, we often encounter ordering errors across components.
Finally, channels are golang specific.
Proposal
The core data primitive for the transfer stack should be a per-session single threaded event bus with total producer-consumer ordering.
The most important reason for using an event bus is decoupling producers of events from their consumers. Currently, we implement some extensibility informally in via interfaces, registered hooks, and occasionally HTTP APIs. An event bus bakes extensibility into the architecture.
At the same time, while not a property of all event buses, certain event buses can gaurantee a single ordering of events among components. To avoid races, we want total producer-consumer ordering: everyone receives and consumes events in the exact same order.
We have built a prototype interface for such a bus here: https://github.com/protocol/hack-the-bus (implementation still in progress)
In the proposed bus, components can:
Proposed Event Types
The following initial events would enable all three core operations, as well as provide mechanisms for payment in transfer across multiple protocols.
User Content Routing Request
a. Parameters: CID / Selector / policy preferences
b. Producer: Top Level Interface
c. Consumer: Content Routing Coordinator
Source Specific Content Routing Request
a. Parameters: CID / Selector / policy preferences & a specific content source
b. Producer: Content Routing Coordinator
c. Consumer: Request Executor for Content Routing Source
Content Routing Result
a. Parameters: Peer / Content found / Protocols Supported / Exchange Mechanisms Supported
b. Producer: Request Executor for Content Routing Source
c. Consumer: Policy Engine, Content Routing Coordinator (important for planning requests to other sources)
Transfer Request
a. Parameters: peer / transport protocol / data requested / payment mechanism and parameters
b. Producer: Policy Engine
c. Consumer: Control protocol implementation (see go-data-transfer), Content Routing Coordinator (important to know what has already been requested)
Initial Voucher Request
a. Parameters: peer / data requested / payment mechanism and parameters
b. Producer: Control protocol implementation
c. Consumer: Exchange protocol / Wallet
Initial Voucher
a. Parameters: payment voucher
b. Producer: Exchange protocol / Wallet
c. Consumer: Control protocol implementation
Initiate Transfer
a. Parameters: peer / data requested / payment voucher
b. Producer: Control protocol implementation
c. Consumer: Transport implementation
Followup Voucher Request
a. Parameters: peer / data requested / data received, payment mechanism and parameters
b. Producer: Control protocol implementation
c. Consumer: Exchange protocol / Wallet
Followup Voucher
a. Parameters: payment voucher
b. Producer: Exchange protocol / Wallet
c. Consumer: Control protocol implementation
Send voucher and resume
a. Parameters: peer / data requested / followup payment voucher
b. Producer: Control protocol implementation
c. Consumer: Transport implementation
Transfer result
a. Parameters: success / failure / partial success (with information about what was retrieved)
b. Producer: Transport implementation
c. Consumer: Policy engine, Content Routing coordinator
Miscellaneous Concerns
Identifying Peers
We need a way to identify peers in this framework. We should pass information in a form that can help us identify both libp2p peers and traditional HTTP peers
Some possible formats include:
See authenticated records: https://github.com/libp2p/specs/pull/217
Identifying protocols
We need to define a format for identifying protocols:
/libp2p/bitswap
/libp2p/graphsync
Strings? Multicodec? Something else?
Bitswap vs Graphsync
Figuring out how graphsync and bitswap interact and are prioritized is a whole large seperate discussion, that merits its own product review.
Miscellaneous Improvement needs
Optimization Scenarios
Payment mechanisms To Figure Out
These are "out of band" problems (they are the implementation problems of additional exchange protocol providers) but still worth mentioning.