Document design for the DAL node

THIS DOCUMENT IS A DRAFT. Everything is subject to change.

THIS DOCUMENT MAY NOT BE UP TO DATE.

Overview

The DAL node is the main component of the Data-Availibility Layer (DAL). Objectives of the DAL node can be split into two:

• a P2P protocol to communicate
• an API to communicate with the various users

The users of the DAL node can be:

• The rollup nodes to produce slots and reconstruct slots
• The endorser to assert the availibility of data
• Other DAL nodes to exchange information about the availibility of data or to get new data of interest

This document will not talk about the P2P interaction. To simulate a P2P interaction, we will rely on the API.

The goal of this document is therefore to describe a backend of the DAL node and an API that is adapted to the different users described above.

• Document design for the DAL node
  • Overview
  • High-level description of the DAL node use-cases
  • Data-structures
  • Cryptographic primitives
• Storage diagram
  • Rest API
• Profiles using this API
  • Slot producer
  • Slot consumer
  • Attestor
  • Observer
  • Archiver
• Handling switching of protocols

High-level description of the DAL node use-cases

We describe in this section the various use-cases for the DAL node:

• The rollup node must be able to get pages corresponding to slot headers that are confirmed
• The rollup node must be able to request the proof of some specific page
• The attestor must be able to know whether the shards he was assigned to per slot are confirmed
• A slot producer must be able to compute a commitment and its proof
• A slot reconstructor must be able to compute a slot and the corresponding pages if enough of shards are known
• Another DAL node can request the shards which are known
• Another DAL node must be able to get shards and their corresponding proofs

We do not tackle the problem of how-long those data must be made available by the DAL node.

This notion of various use-case leads to a notion of profile as described below:

Data-structures

Several data-structures are involved with the DAL node. We give a rough description of those data here.

The DAL node stores raw data that we call slot. A slot will have a fixed size. This slot can be split in two ways:

• By pages which are chunks of the original slot
• By shard which encodes a chunk of the original slot with an erasure code.

The main property of shards is that any sufficiently large subset of the shards can be used to reconstruct the original slot.

In the DAL nodes, those notions are abstract:

type slot = bytes

type shard

type page

Only the cryptographic primitives know how to interpret them.

To prevent spamming we must be able to prove that a shard or page is related to the original slot. To do so, there is a notion of commitment and proofs.

type commitment

type shard_proof 

type page_proof

Again, those data-types can only be interpreted by the cryptographic primtives.

Cryptographic primitives

The DAL node must interact with the cryptographic primitive to provide the various services mentioned above. Cryptographic primitives are parameterized by some constants:

• slot_size which is the size of a slot
• number_of_shards which is the number of shards per slot
• number_of_pages which is the number of pages per slot
• redundancy_factor which is the erasure-code factor used for the shard. Only
  $\frac{\text{number\_of\_shards}}{\text{redundancy\_factor}}$ shards are necessary to reconstruct the whole slot.

There is a dependency between those parameters, and the data stored by the DAL node as well as the validation of those data. Consequently, it is essential to always be able to retrieve those parameters when using the API.

It is important for this design to assume that those parameters can change. In particular because some parameters are correlated to other parameters of the economic protocol. For example, the slot_size depends on the time_between_blocks.

Storage diagram

The following diagram gives an overview of the storage (viewed as a tree) of the DAL node. Whenever there is an item starting with < and ending with > it is a variable whose range belongs to the corresponding datatype. Hence <commitment> means it can be any valid representation (as an OCaml string) of a commitment. Only leaves contain datatypes. All datatypes have a fixed length.

This storage is an over-approximation of what could be actually stored.

For example, it may be faster to recompute the commitment_proof than to store it (same argument could be done for page). A benchmark should allow to determine whether this is useful. For performance reasons, it may also be interesting to provide a cache for the various data stored. We can expect that shards related to a slot header that is Waiting_for_attestations will be asked many times in a relatively short amount of time.

Finally, concerning the concurrency, we should ensure that the writings are done atomically and the node can support several readers and writers (each connection can write/read data).

Rest API

The DAL node provides en API so that we can interact with it. This API can be used by external users like rollup nodes or attestors but could also be used internally via the notion of profile as explained in the next section.

• POST /commitments -> rename /slots -> /commitments

  • Input: slot with length of exactly 1MiB
  • Output: the slot's commitment with its proof
  • Description: Returns when the slot is stored by the DAL node.
    • Usage: submit a new slot to the rollup node. The cryptobox parameters used are the one corresponding to the current L1's head.
    • Errors: Fails if the slot is not of the expected size.
  • • Implemented in !6922 as POST /slots?
    • Improved/renamed in !7262

• PATCH /commitments/<commitment>

  • Input:
  ​​​​    type input = 
  ​​​​    {
  ​​​​        published_level : level;
  ​​​​        slot_index : slot_index
  ​​​​    }
  

  • Output: unit
  • Description: Associate a commitment to a level and a slot index.
  • • Implemented in !6922 as PATCH /slots/<commitment> ?
    • Improved/renamed in !7262
    • Follow-up: #4528

• GET /commitments/<commitment>

  • Input: None
  • Output: slot
  • Description: returns the slot corresponding to `<commitment>``.
    • Usage: allow to get the content of a slot given a <commitment>
    • Errors:
      • Fails if <commitment> is not known.
      • Fails if the slot is not known. Note: If the slot could be recomputed from the shards, once must explicit do it using PUT /slots/<slot_id>.
  • • Implemented in !6926 as GET /slots/<commitment> ?
    • Improved/renamed in 7265

• GET /commitments/<commitment>/proof

  • Input: None
  • Output: commitment_proof
  • Description: returns the proof that the <slot> corresponding to <commitment> is below the size of the commitment.
  • • Implemented as GET /slots/<commitment>/proof in !7008?
    • Improved/renamed in 7265

• GET /commitments/<commitment>/headers

  • Input: None
  • Query parameters
    • Level : filter the slot header returned with an L1 level
    • Slot index : filter the slot index
  • Output: slot header list
  ​​​​type attestation_status =
  ​​​​    | Waiting_for_attestations (* The slot header was published onto the L1 but remains to be confirmed. *)
  ​​​​    | Attested (* The slot header was published and confirmed onto the L1. *)
  ​​​​    | Unattested (* The slot header was published but not confirmed onto the L1. *)
  
  ​​​​type status = 
  ​​​​   | [ attestation_status ]
  ​​​​   | Not_selected  (* The slot header was published onto the L1 but was not selected as the slot header for this index. *)
  ​​​​   | Unseen (* The slot header was never seen in a block. This can happen if the RPC `PATCH /slots/<commitment>` was called but the corresponding slot header was never included into a block. *)
  
  ​​​​type slot_header = 
  ​​​​    {
  ​​​​        level : level;
  ​​​​        commitment : commitment;
  ​​​​        slot_index : slot_index;
  ​​​​        status : status;
  ​​​​    }
  

  • Description: returns the slot headers associated with the given <commitment> that were published onto the L1. Considering one <commitment> it can be only has one status at time that may change depending on the L1.
    • Usage: Get the status of a slot header given its commitment
    • Errors: Fails if the <commitment> is unknown.
  • • Implemented in !7049
    • Needs to be revisited in follow-up: #4528 and #4541

• GET /levels/<published_level>/slot_indices/<slot_index>/commitment

  • Input: None
  • Output: commitment
  • Description: return the commitment associated to the given <published_level> and <slot_index>. This commitment must have the status Confirmed or Not_selected.
    • Usage: Get the commitment associated to an index.
    • Errors: Fails if there is no commitment at the level <level> and slot index <slot_index>.
  • • Implemented in !7121
    • Currently Not_selected not handled (returns
      404 if not waiting for attestation. Fix it!

• GET /levels/<published_level>/headers

  • Input: None
  • Query parameters: the expected status (optional)
  • Output: slot_header list
  • Description: returns all the slot headers associated for the given level.
    • Usage: Can be used to know all the slot headers published at some level.
    • Errors:
      • Should fail if the <level> is in the future?
  • • Implemented in !7285
    • Check the expected status
    • Should fail if the <level> is in the future?: not handled yet.

• GET /levels/<published_level>/<slot_index>/status

  • Input: None
  • Output:
    ​​​​​​​​type status = attestation_status =
    ​​​​​​​​   | Waiting_for_attestations 
    ​​​​​​​​   | Attested 
    ​​​​​​​​   | Unattested 
    

  • Description:
    • Usage: Get the status of a slot.

• GET /slots/<published_level>/<slot_index>/slot

  • Input: None
  • Output: slot
  • Description:
    • Usage: Get a slot.
    • Remarks: The call to this RPC can be long.
    • Errors:
      • Fails if <slot_id> is not known.
      • Fails if there is not enough shards which are known.
      • Fails if the slot is already present.

• POST /commitments/<commitment>/shards

  • Input:
    ​​​​​​​​    type input = {
    ​​​​​​​​        with_proof : bool
    ​​​​​​​​    }
    

  • Output: None
  • Description: computes the shards associated to the given slot. If None is given as input, then all the shards are computed. If with_proof is given, the proofs associated to each given shards are also computed.
    • Usage: Should be used before advertising shards onto the network.
    • Errors:
      • Fails if the the slot associated to <commitment> is not known.
  • • implemented in: !7294

• GET /slots/<published_level>/<slot_index>/shards/<shard_index>

  • Input: None
  • Output: shard
  • Description: returns the shard with the given index for the given slot.
    • Usage: Use to get a particular shard
    • Errors:
      • Fails if <slot_id> is not known.
      • Fails if <shard_index> is not in the correct range.
      • Fails if the shard associated to <shard_index> is not known.

• GET /slots/<published_level>/<slot_index>/shards/<shard_index>/proof

  • Input: None
  • Output: shard_proof
  • Description: returns the proof of the shard associated to the shard index and the corresponding slot id.
    • Usage: Use to get a particular proof of a shard
    • Errors:
      • Fails if <slot_id> is not known.
      • Fails if <shard_index> is not in the correct range.
      • Fails if the proof of shard <shard_index> is not known.

• GET /slots/<published_level>/<slot_index>/page/<page_index>

  • Input: None
  • Output: page
  • Description: returns the page with the given index for the given slot.
    • Usage: Use to get a particular page
    • Errors:
      • Fails if <slot_id> is not known.
      • Fails if <page_index> is not in the correct range.

• GET /slots/<published_level>/<slot_index>/page/<page_index>/proof

  • Input: None
  • Output: page_proof
  • Description: returns the proof of the page associated to the page index and the corresponding slot id.
    • Usage: Use to get a particular proof of a page
    • Errors:
      • Fails if <slot_id> is not known.
      • Fails if <page_index> is not in the correct range.
      • Fails if the proof of page <page_index> is not known.

• GET /level/<published_level>/confirmed_level

  • Input: None
  • Output: level
  • Description: return the confirmed level associated to a published level.
    • Usage: Usage to compute the confirmed level given a published level.
    • Errors:
      • Fails if <published_level> is too far in the future.

• PATCH /profiles {profile}

  • Input:
  ​​​​type profile = 
  ​​​​  | Attestor of public_key_hash
  ​​​​  | Consumer of slot_index
  ​​​​  | Producer of slot_index
  ​​​​  | Observer of [`Slot of slot_index | `Shards of public_key_hash]
  ​​​​  | Archive
  

  • Output: None
  • Description: updates the list of profiles tracked by the DAL node
    • Usage: This can be used to know which shards needs to be downloaded and which slots needs to be tracked.
    • Errors:
      • Fails if <slot_index> is not in the correct range

  • Implemented in !7104 for Attestor profile

• GET /profiles

  • Input: None
  • Output: profile list
  • Description: Provides the list of current profiles tracked.

  • Implemented in !7104

• GET /profiles/<public_key_hash>/attested_levels/<level>/assigned_shard_indices

  • Input: None
  • Output: shard_index list
  • Description: Returns the shard indices associated to this public key hash for the given level. The association is given by the DAL committee which depends on the L1 economic protocol.

  • Implemented in !7207
    Updated path in !7314

• GET /profiles/<public_key_hash>/<slot_id>

  • Input: None
  • Output: bool
  • Description: Returns true if the DAL node has the corresponding shards associated to this public key hash for the given slot.

• GET /profiles/<public_key_hash>/attested_levels/<level>/attestable_slots

  • Input: None
  • Output: Attestable_slots of slot_set | Not_in_committee
  • Description: Returns the set of currently attestable slots. A slot is attestable if it published and all the shards assigned at the given level to the given public key hash are available in the DAL node's store.

  • Implemented in: !7314
    Updated return value in !7477

• GET /monitor/slot_headers

  • Description: Produces a stream of slot headers which are known by the node
  • Input: None
  • Query parameters
    • Slot index : filter the slot index
    • attested : monitor only the attested slot headers
    • from level : monitor slot headers from a specific level
  • Output: slot_header stream

• GET /monitor/<slot_id>/shards

  • Input: None
  • Output: shards stream
  • Description: Produces a stream of shards known by the DAL node.

• GET /limit/<level>

  • Input: None
  • Output:
  ​​​​    type output = {
  ​​​​        slot_size : int;
  ​​​​        number_of_shards : int;
  ​​​​        number_of_slots : int;
  ​​​​        page_size : int;
  ​​​​        shard_size : int;
  ​​​​        number_of_pages : int;
  ​​​​        endorsement_lag : int;
  ​​​​        shard_proof_size : int;
  ​​​​        page_proof_size : int;
  ​​​​        commitment_size : int;
  ​​​​    }
  

  • Description: returns the limits associated to the cryptographic primitives.
    • Usage: To get the maximum number of pages, shards, etc…
    • Errors: if <level> is in the future or too far in the past.

The following RPCs can be used to delete the corresponding ressources:

• DELETE /slots/<slot_id>/pages/<page_index>/proof
• DELETE /slots/<slot_id>/pages/<page_index>
• DELETE /slots/<slot_id>/pages
• DELETE /slots/<slot_id>/shards/<shard_index>/proof
• DELETE /slots/<slot_id>/shards/<shard_index>
• DELETE /slots/<slot_id>/shards
• DELETE /slots/<slot_id>
• DELETE /slots/<level>
• DELETE /profile/<public_key_hash>

Note: This interface could be refined depending on the road blocks that can be identified while executing end-to-end tests. All those RPCs are not strictly needed to implement a demo.

Profiles using this API

We describe below the various use cases of this API depending on the various profiles defined below. Each profile can be implemented using the API provided above. For each profile below, except the archive profile, DELETE RPCs can be called for slot headers which are too old.

• Slot consumer profile aims to be used by a rollup node in "observer" mode
• Slot producer profile aims to be used by a rollup node in "batcher" mode
• Endorser profile aims to be used by an endorser
• Observer profile aims to be used as intermediate nodes in the DAL network
• Archiver profile aims to be used for DAL indexers

Some of those profiles could be provided directly via the DAL node such as the slot producer or the slot consumer while other profiles could be implemented externally.

Slot producer

Producing a SLOT could be implemented as:

• POST /slot {SLOT} –> returns a commitment

  • See/replaced by POST /commitments

• PATCH /slot/<commitment> {level;slot_index} –> associates a slot header to the commitment

  • See/replaced by PATCH /commitments/<commitment>

• GET /slot/<commitment>/proof –> returns a commitment_proof

  • See/replaced by GET /commitments/<commitment>/proof

• publish the slot_header with the commitment proof onto the L1
• for all shards, POST /slot/<slot_id>/shards {with_proof=true} . This call can be costly (between 5 and 10 seconds).

Slot consumer

A slot consumer tracking SLOT_INDEX could be implemented as:

• GET /monitor/slot_headers?slot_index=SLOT_INDEX&confirmed=true –> returns a stream of slot headers

• For each slot headers SH produced by the stream:

  • GET /slots/<commitment>/slot
    • • See/replaced by GET /commitments/<commitment>
      • TODO: we must handle the reconstruction of a slot from a shard properly

• GET /slots/<published_level>/<slot_index>/pages/<page> to get a page of a slot requested by the PVM

Attestor

An attestor with the public key hash PUBLIC_KEY_HASH can be implemented as:

• GET /monitor/slot_headers –> returns a stream of slot headers
• For each slot headers SH produced by the stream:
  • GET /slots/SH/info
  • For all slot headers that needs to be confirmed:
    • GET /profile/PUBLIC_KEY_HASH/SH (TODO: The attestor must do this request at the appropriate moment)

Observer

An observer can observe slot headers via the GET /monitor/slot_headers.

Archiver

An archiver aims to archive everything that went through the DAL. It can be implemented as:

• GET /monitor/slot_headers –> returns a stream of slot headers
• For each slot headers SH produced by the stream:
  • GET /slots/SH/info
  • for all the shard indices known:
    • GET /slots/SH/shards/<shard_index>
    • GET /slots/SH/shards/<shard_index>/proof
  • PUT /slots/SH/content {reconstruct=true}
  • for all the page indices known:
    • GET /slots/SH/shards/<page_index>
    • GET /slots/SH/shards/<page_index>/proof
• It remains unclear how a DAL node that is run with the archiver profile will be able to know old slot headers.

Handling switching of protocols

There are mainly two difficulties to handle properly the switching of protocol:

1. DAL parameters can change, which means the cryptographic primitives may change too. Except for the archive profile, we only need to deal with at most 2 (maybe 3 protocols switching). The design ensures this is doable even though for the demo it is not a necessity. In particular, it requires some anticipation to publish a slot header for the first level of a protocol because it requires to know the DAL parameters for this protocol in advance.

2. The DAL node uses a plugin mechanism to get some data which are protocol specific such as the DAL parameters. We muse ensure we can switch from one plugin to another. This should be similar to what is currently done for the prevalidator. However, the DAL node must still be able to answer to request for a previous protocol for some amount of time.

This document does not aim to solve this problem. Instead it aims to provide a design that allows to solve those two problems without having to redesign the internals of the DAL node.