Document design for the DAL node

# Document design for the DAL node :warning: THIS DOCUMENT IS A DRAFT. Everything is subject to change. :warning: :warning: THIS DOCUMENT MAY NOT BE UP TO DATE. :warning: ## Overview The DAL node is the main component of the Data-Availibility Layer (DAL). Objectives of the DAL node can be split into two: - a P2P protocol to communicate - an API to communicate with the various users The users of the DAL node can be: - The rollup nodes to produce slots and reconstruct slots - The endorser to assert the availibility of data - Other DAL nodes to exchange information about the availibility of data or to get new data of interest This document will not talk about the P2P interaction. To simulate a P2P interaction, we will rely on the API. The goal of this document is therefore to describe a backend of the DAL node and an API that is adapted to the different users described above. [toc] ## High-level description of the DAL node use-cases We describe in this section the various use-cases for the DAL node: - The rollup node must be able to get pages corresponding to slot headers that are confirmed - The rollup node must be able to request the proof of some specific page - The attestor must be able to know whether the shards he was assigned to per slot are confirmed - A slot producer must be able to compute a commitment and its proof - A slot reconstructor must be able to compute a slot and the corresponding pages if enough of shards are known - Another DAL node can request the shards which are known - Another DAL node must be able to get shards and their corresponding proofs :::warning We do not tackle the problem of how-long those data must be made available by the DAL node. ::: This notion of various use-case leads to a notion of profile as described below: ## Data-structures Several data-structures are involved with the DAL node. We give a rough description of those data here. The DAL node stores raw data that we call `slot`. A slot will have a fixed size. This slot can be split in two ways: - By `pages` which are chunks of the original `slot` - By `shard` which encodes a chunk of the original `slot` with an erasure code. The main property of shards is that any sufficiently large subset of the shards can be used to reconstruct the original slot. In the DAL nodes, those notions are abstract: ```ocaml= type slot = bytes type shard type page ``` Only the cryptographic primitives know how to interpret them. To prevent spamming we must be able to prove that a `shard` or `page` is related to the original slot. To do so, there is a notion of `commitment` and proofs. ```ocaml= type commitment type shard_proof type page_proof ``` Again, those data-types can only be interpreted by the cryptographic primtives. ## Cryptographic primitives The DAL node must interact with the cryptographic primitive to provide the various services mentioned above. Cryptographic primitives are parameterized by some constants: - `slot_size` which is the size of a slot - `number_of_shards` which is the number of shards per slot - `number_of_pages` which is the number of pages per slot - `redundancy_factor` which is the erasure-code factor used for the shard. Only $\frac{\text{number_of_shards}}{\text{redundancy_factor}}$ shards are necessary to reconstruct the whole slot. There is a dependency between those parameters, and the data stored by the DAL node as well as the validation of those data. Consequently, it is essential to always be able to retrieve those parameters when using the API. :::info It is important for this design to assume that those parameters can change. In particular because some parameters are correlated to other parameters of the economic protocol. For example, the `slot_size` depends on the `time_between_blocks`. ::: # Storage diagram The following diagram gives an overview of the storage (viewed as a tree) of the DAL node. Whenever there is an item starting with `<` and ending with `>` it is a variable whose range belongs to the corresponding datatype. Hence `<commitment>` means it can be any valid representation (as an OCaml string) of a commitment. Only leaves contain datatypes. All datatypes have a fixed length. ```mermaid graph R[ ]--- |commitments|C[ ]; R[ ]--- |levels|L[ ]; C[ ]--- |"<commitment>"|CC[ ]; CC[ ]--- |"headers"|CCH[ ]; CCH[ ]--- |"&ltslot_id&gt"|CCHH[empty, slot_header in path?]; CC[ ]--- |"<slot_size>"|CCSIZE[ ]; CCSIZE[ ]--- |"slot"|CCSIZEE[slot]; CC[ ]--- |"proof"|CCPF[commitment_proof]; CC[ ]--- |"shards"|CCS[ ]; CCS[ ]--- |"<number_of_shards,redundancy_factor&gt"|CCSS[ ]; CCSS[ ]--- |"index"|CCSSI[ ]; CCSSI[ ]--- |"<i&gt"|CCSSn[ ]; CCSSn[ ]--- |"shard"|CSSnS[shard]; CCSSn[ ]--- |"proof"|CSSnP[shard_proof]; CC[ ]--- |"pages"|CCPPn[ ]; %% CCP[ ]--- |"<number_of_pages&gt"|CCPP[ ]; %% CCPP[ ]--- |"index"| CCPPI[ ]; %% CCPPI[ ]--- |"<i&gt"|CCPPn[ ]; CCPPn[ ]--- |"page"|CPPnS[page]; CCPPn[ ]--- |"proof"|CPPnP[page_proof]; L[ ]--- |"<level>"|LL[ ]; %%LL[ ]--- |"parameters"|LLP[Cryptobox.parameters]; LL[ ]--- |"slot_indices"|LLSI[ ]; LLSI[ ]--- |"<slot_index>"|LLSII[ ]; LLSII[ ]--- |"accepted"|LLSIIA[ ]; LLSIIA[ ]--- |"commitment"|LLSIIAC[commitment]; LLSIIA[ ]--- |"status"|LLSIIAS[status]; LLSII[ ]--- |"others"|LLSIIO[ ]; LLSIIO[ ]--- |"<commitment>"|LLSIIOC[status]; ``` This storage is an over-approximation of what could be actually stored. For example, it may be faster to recompute the `commitment_proof` than to store it (same argument could be done for `page`). A benchmark should allow to determine whether this is useful. For performance reasons, it may also be interesting to provide a cache for the various data stored. We can expect that shards related to a slot header that is `Waiting_for_attestations` will be asked many times in a relatively short amount of time. Finally, concerning the concurrency, we should ensure that the writings are done atomically and the node can support several readers and writers (each connection can write/read data). ## Rest API The DAL node provides en API so that we can interact with it. This API can be used by external users like `rollup nodes` or `attestors` but could also be used internally via the notion of `profile` as explained in the next section. - [x] `POST /commitments` -> rename `/slots -> /commitments` - **Input**: `slot` with length of exactly `1`MiB - **Output**: the slot's commitment with its proof - **Description**: Returns when the slot is stored by the DAL node. - *Usage*: submit a new slot to the rollup node. The cryptobox parameters used are the one corresponding to the current L1's head. - *Errors*: Fails if the slot is not of the expected size. - :::success - Implemented in [!6922](https://gitlab.com/tezos/tezos/-/merge_requests/6922) as `POST /slots`? - Improved/renamed in [!7262](https://gitlab.com/tezos/tezos/-/merge_requests/7262) ::: - [x] `PATCH /commitments/<commitment>` - **Input**: ```ocaml= type input = { published_level : level; slot_index : slot_index } ``` - **Output**: unit - **Description**: Associate a commitment to a level and a slot index. - :::success - Implemented in [!6922](https://gitlab.com/tezos/tezos/-/merge_requests/6922) as `PATCH /slots/<commitment>` ? - Improved/renamed in [!7262](https://gitlab.com/tezos/tezos/-/merge_requests/7262) - Follow-up: [#4528](https://gitlab.com/tezos/tezos/-/issues/4528) ::: - [x] `GET /commitments/<commitment>` - **Input**: None - **Output**: `slot` - **Description**: returns the slot corresponding to `<commitment>``. - *Usage*: allow to get the content of a slot given a `<commitment>` - *Errors*: - Fails if `<commitment>` is not known. - Fails if the slot is not known. Note: If the slot could be recomputed from the shards, once must explicit do it using `PUT /slots/<slot_id>`. - :::success - Implemented in [!6926](https://gitlab.com/tezos/tezos/-/merge_requests/6926) as `GET /slots/<commitment>` ? - Improved/renamed in [7265](https://gitlab.com/tezos/tezos/-/merge_requests/7265) ::: - [x] `GET /commitments/<commitment>/proof` - **Input**: None - **Output**: `commitment_proof` - **Description**: returns the proof that the `<slot>` corresponding to `<commitment>` is below the size of the commitment. - :::success - Implemented as `GET /slots/<commitment>/proof` in [!7008](https://gitlab.com/tezos/tezos/-/merge_requests/7008)? - Improved/renamed in [7265](https://gitlab.com/tezos/tezos/-/merge_requests/7265) ::: - [x] `GET /commitments/<commitment>/headers` - **Input**: None - **Query parameters** - Level : filter the slot header returned with an L1 level - Slot index : filter the slot index - **Output**: slot header list ```ocaml= type attestation_status = | Waiting_for_attestations (* The slot header was published onto the L1 but remains to be confirmed. *) | Attested (* The slot header was published and confirmed onto the L1. *) | Unattested (* The slot header was published but not confirmed onto the L1. *) type status = | [ attestation_status ] | Not_selected (* The slot header was published onto the L1 but was not selected as the slot header for this index. *) | Unseen (* The slot header was never seen in a block. This can happen if the RPC `PATCH /slots/<commitment>` was called but the corresponding slot header was never included into a block. *) type slot_header = { level : level; commitment : commitment; slot_index : slot_index; status : status; } ``` - **Description**: returns the slot headers associated with the given `<commitment>` that were published onto the L1. Considering one `<commitment>` it can be only has one status at time that may change depending on the L1. - *Usage*: Get the status of a slot header given its commitment - *Errors*: Fails if the `<commitment>` is unknown. - :::success - Implemented in [!7049](https://gitlab.com/tezos/tezos/-/merge_requests/7049) - Needs to be revisited in follow-up: [#4528](https://gitlab.com/tezos/tezos/-/issues/4528) and [#4541](https://gitlab.com/tezos/tezos/-/issues/4541) ::: - [x] `GET /levels/<published_level>/slot_indices/<slot_index>/commitment` - **Input**: None - **Output**: commitment - **Description**: return the commitment associated to the given `<published_level>` and `<slot_index>`. This commitment must have the status `Confirmed` or `Not_selected`. - *Usage*: Get the commitment associated to an index. - *Errors*: Fails if there is no commitment at the level `<level>` and slot index `<slot_index>`. - :::success - Implemented in [!7121](https://gitlab.com/tezos/tezos/-/merge_requests/7121) - Currently `Not_selected` not handled (returns 404 if not waiting for attestation. Fix it! ::: - [x] `GET /levels/<published_level>/headers` - **Input**: None - **Query parameters:** the expected status (optional) - **Output**: slot_header list - **Description**: returns all the slot headers associated for the given level. - *Usage*: Can be used to know all the slot headers published at some level. - *Errors*: - Should fail if the `<level>` is in the future? - :::success - Implemented in [!7285](https://gitlab.com/tezos/tezos/-/merge_requests/7285) - Check the expected status - Should fail if the <level> is in the future?: not handled yet. ::: - [ ] `GET /levels/<published_level>/<slot_index>/status` - **Input**: `None` - **Output**: ```ocaml= type status = attestation_status = | Waiting_for_attestations | Attested | Unattested ``` - **Description**: - *Usage*: Get the status of a slot. - [ ] `GET /slots/<published_level>/<slot_index>/slot` - **Input**: `None` - **Output**: slot - **Description**: - *Usage*: Get a slot. - *Remarks*: The call to this RPC can be long. - *Errors*: - Fails if `<slot_id>` is not known. - Fails if there is not enough shards which are known. - Fails if the slot is already present. - [ ] `POST /commitments/<commitment>/shards` - **Input**: ```ocaml type input = { with_proof : bool } ``` - **Output**: None - **Description**: computes the shards associated to the given slot. If `None` is given as input, then all the shards are computed. If `with_proof` is given, the proofs associated to each given shards are also computed. - *Usage*: Should be used before advertising shards onto the network. - *Errors*: - Fails if the the slot associated to `<commitment>` is not known. - :::success - implemented in: [!7294](https://gitlab.com/tezos/tezos/-/merge_requests/7294) ::: - [ ] `GET /slots/<published_level>/<slot_index>/shards/<shard_index>` - **Input**: None - **Output**: shard - **Description**: returns the shard with the given index for the given slot. - *Usage*: Use to get a particular shard - *Errors*: - Fails if `<slot_id>` is not known. - Fails if `<shard_index>` is not in the correct range. - Fails if the shard associated to `<shard_index>` is not known. - [ ] `GET /slots/<published_level>/<slot_index>/shards/<shard_index>/proof` - **Input**: None - **Output**: shard_proof - **Description**: returns the proof of the shard associated to the shard index and the corresponding slot id. - *Usage*: Use to get a particular proof of a shard - *Errors*: - Fails if `<slot_id>` is not known. - Fails if `<shard_index>` is not in the correct range. - Fails if the proof of shard `<shard_index>` is not known. - [ ] `GET /slots/<published_level>/<slot_index>/page/<page_index>` - **Input**: None - **Output**: page - **Description**: returns the page with the given index for the given slot. - *Usage*: Use to get a particular page - *Errors*: - Fails if `<slot_id>` is not known. - Fails if `<page_index>` is not in the correct range. - [ ] `GET /slots/<published_level>/<slot_index>/page/<page_index>/proof` - **Input**: None - **Output**: page_proof - **Description**: returns the proof of the page associated to the page index and the corresponding slot id. - *Usage*: Use to get a particular proof of a page - *Errors*: - Fails if `<slot_id>` is not known. - Fails if `<page_index>` is not in the correct range. - Fails if the proof of page `<page_index>` is not known. - [ ] `GET /level/<published_level>/confirmed_level` - **Input**: None - **Output**: level - **Description**: return the confirmed level associated to a published level. - *Usage*: Usage to compute the confirmed level given a published level. - *Errors*: - Fails if `<published_level>` is too far in the future. - [x] `PATCH /profiles {profile}` - **Input**: ```ocaml= type profile = | Attestor of public_key_hash | Consumer of slot_index | Producer of slot_index | Observer of [`Slot of slot_index | `Shards of public_key_hash] | Archive ``` - **Output**: None - **Description**: updates the list of profiles tracked by the DAL node - **Usage**: This can be used to know which shards needs to be downloaded and which slots needs to be tracked. - **Errors**: - Fails if `<slot_index>` is not in the correct range - :::success Implemented in [!7104](https://gitlab.com/tezos/tezos/-/merge_requests/7104) for `Attestor` profile ::: - [x] `GET /profiles` - **Input**: None - **Output**: profile list - **Description**: Provides the list of current profiles tracked. - :::success Implemented in [!7104](https://gitlab.com/tezos/tezos/-/merge_requests/7104) ::: - [x] `GET /profiles/<public_key_hash>/attested_levels/<level>/assigned_shard_indices` - **Input**: None - **Output**: shard_index list - **Description**: Returns the shard indices associated to this public key hash for the given level. The association is given by the DAL committee which depends on the L1 economic protocol. - :::success Implemented in [!7207](https://gitlab.com/tezos/tezos/-/merge_requests/7207) Updated path in [!7314](https://gitlab.com/tezos/tezos/-/merge_requests/7314) ::: - [ ] `GET /profiles/<public_key_hash>/<slot_id>` - **Input**: None - **Output**: bool - **Description**: Returns `true` if the DAL node has the corresponding shards associated to this public key hash for the given slot. - [ ] `GET /profiles/<public_key_hash>/attested_levels/<level>/attestable_slots` - **Input**: None - **Output**: `Attestable_slots of slot_set | Not_in_committee` - **Description**: Returns the set of currently attestable slots. A slot is attestable if it published and *all* the shards assigned at the given level to the given public key hash are available in the DAL node's store. - :::success Implemented in: [!7314](https://gitlab.com/tezos/tezos/-/merge_requests/7314) Updated return value in [!7477](https://gitlab.com/tezos/tezos/-/merge_requests/7477) ::: - [ ] `GET /monitor/slot_headers` - **Description**: Produces a stream of slot headers which are known by the node - **Input**: None - **Query parameters** - Slot index : filter the slot index - attested : monitor only the attested slot headers - from level : monitor slot headers from a specific level - **Output**: slot_header stream - [ ] `GET /monitor/<slot_id>/shards` - **Input**: None - **Output**: shards stream - **Description**: Produces a stream of shards known by the DAL node. - [ ] `GET /limit/<level>` - **Input**: None - **Output**: ```ocaml= type output = { slot_size : int; number_of_shards : int; number_of_slots : int; page_size : int; shard_size : int; number_of_pages : int; endorsement_lag : int; shard_proof_size : int; page_proof_size : int; commitment_size : int; } ``` - **Description**: returns the limits associated to the cryptographic primitives. - **Usage**: To get the maximum number of pages, shards, etc... - **Errors**: if `<level>` is in the future or too far in the past. The following RPCs can be used to delete the corresponding ressources: - [ ] `DELETE /slots/<slot_id>/pages/<page_index>/proof` - [ ] `DELETE /slots/<slot_id>/pages/<page_index>` - [ ] `DELETE /slots/<slot_id>/pages` - [ ] `DELETE /slots/<slot_id>/shards/<shard_index>/proof` - [ ] `DELETE /slots/<slot_id>/shards/<shard_index>` - [ ] `DELETE /slots/<slot_id>/shards` - [ ] `DELETE /slots/<slot_id>` - [ ] `DELETE /slots/<level>` - [ ] `DELETE /profile/<public_key_hash>` Note: This interface could be refined depending on the road blocks that can be identified while executing end-to-end tests. All those RPCs are not strictly needed to implement a demo. # Profiles using this API We describe below the various use cases of this API depending on the various `profiles` defined below. Each profile can be implemented using the API provided above. For each profile below, except the archive profile, `DELETE` RPCs can be called for slot headers which are too old. - Slot consumer profile aims to be used by a rollup node in "observer" mode - Slot producer profile aims to be used by a rollup node in "batcher" mode - Endorser profile aims to be used by an endorser - Observer profile aims to be used as intermediate nodes in the DAL network - Archiver profile aims to be used for DAL indexers Some of those profiles could be provided directly via the DAL node such as the slot producer or the slot consumer while other profiles could be implemented externally. ## Slot producer Producing a SLOT could be implemented as: - [x] `POST /slot {SLOT}` --> returns a `commitment` - :::success See/replaced by `POST /commitments` ::: - [x] `PATCH /slot/<commitment> {level;slot_index}` --> associates a `slot header` to the `commitment` - :::success See/replaced by `PATCH /commitments/<commitment>` ::: - [x] `GET /slot/<commitment>/proof` --> returns a `commitment_proof` - :::success See/replaced by `GET /commitments/<commitment>/proof` ::: - publish the `slot_header` with the commitment proof onto the L1 - for all shards, `POST /slot/<slot_id>/shards {with_proof=true}` . This call can be costly (between `5` and `10` seconds). ## Slot consumer A slot consumer tracking SLOT_INDEX could be implemented as: - [ ] `GET /monitor/slot_headers?slot_index=SLOT_INDEX&confirmed=true` --> returns a stream of slot headers - For each slot headers `SH` produced by the stream: - [x] `GET /slots/<commitment>/slot` - :::success - See/replaced by `GET /commitments/<commitment>` - TODO: we must handle the reconstruction of a slot from a shard properly ::: - [ ] `GET /slots/<published_level>/<slot_index>/pages/<page>` to get a page of a slot requested by the PVM ## Attestor An attestor with the public key hash PUBLIC_KEY_HASH can be implemented as: - [ ] `GET /monitor/slot_headers` --> returns a stream of slot headers - For each slot headers `SH` produced by the stream: - [ ] `GET /slots/SH/info` - For all slot headers that needs to be confirmed: - [ ] `GET /profile/PUBLIC_KEY_HASH/SH` (TODO: The attestor must do this request at the appropriate moment) ## Observer An observer can observe slot headers via the `GET /monitor/slot_headers`. ## Archiver An archiver aims to archive everything that went through the DAL. It can be implemented as: - [ ] `GET /monitor/slot_headers` --> returns a stream of slot headers - For each slot headers `SH` produced by the stream: - [ ] `GET /slots/SH/info` - for all the shard indices known: - [ ] `GET /slots/SH/shards/<shard_index>` - [ ] `GET /slots/SH/shards/<shard_index>/proof` - [ ] `PUT /slots/SH/content {reconstruct=true}` - for all the page indices known: - [ ] `GET /slots/SH/shards/<page_index>` - [ ] `GET /slots/SH/shards/<page_index>/proof` - It remains unclear how a DAL node that is run with the `archiver` profile will be able to know old slot headers. # Handling switching of protocols There are mainly two difficulties to handle properly the switching of protocol: 1. DAL parameters can change, which means the cryptographic primitives may change too. Except for the `archive` profile, we only need to deal with at most 2 (maybe 3 protocols switching). The design ensures this is doable even though for the demo it is not a necessity. In particular, it requires some anticipation to publish a slot header for the first level of a protocol because it requires to know the DAL parameters for this protocol in advance. 2. The DAL node uses a plugin mechanism to get some data which are protocol specific such as the DAL parameters. We muse ensure we can switch from one plugin to another. This should be similar to what is currently done for the prevalidator. However, the DAL node must still be able to answer to request for a previous protocol for some amount of time. :::warning This document does not aim to solve this problem. Instead it aims to provide a design that allows to solve those two problems without having to redesign the internals of the DAL node. :::