# Blob Sidecar handling in Prysm ## Background In EIP4844, [`BlobsSidecar`](https://github.com/ethereum/consensus-specs/blob/dev/specs/eip4844/p2p-interface.md#blobssidecar) was introduced to accept "blobs" of data to be persisted in the beacon node for period of time. These blobs bring rollup fees down by magnitude and enable Ethereum to remain competitive without sacrificing decentralization. Blobs are pruned after ~1 month. Available long enough for all actors of a L2 to retrieve it. The blobs are persisted **in beacon nodes**, not in the execution engine. Alongside these blobs, [`BeaconBlockBody` contains a new field `blob_kzg_commitments`](https://github.com/ethereum/consensus-specs/blob/dev/specs/eip4844/beacon-chain.md#beaconblockbody). It's important that these kzg commitments match blob contents. Note that on the CL, the blobs are only referenced in the `BeaconBlockBody`, and not encoded in the `BeaconBlockBody`. Instead of embedding the full contents, the contents are propagated separately in `BlobsSidecar` above. The CL must do these three things correctly: * Beacon chain: process updated beacon blocks and ensure blobs are available * P2P network: gossip and sync updated beacon block types and new blobs sidecars * Honest validator: produce beacon blocks with blobs, publish the blobs sidecars In this doc, we explore the spec dependencies between `BlobsSidecar` and `BeaconBlockBody` loosely coupled and implementation complexity due to them being loosely coupled. # Beacon chain processing As part of block processing, a new function [`process_blob_kzg_commitments`](https://github.com/ethereum/consensus-specs/blob/dev/specs/eip4844/beacon-chain.md#blob-kzg-commitments) is added. It ensures kzg commitments between the `BeaconBlockBody` matches hashes in `SignedBlobTransaction` defined in the block transaction. Given raw transactions, a node should be able to peek inside for blob versioned hashes given the field offset. In this validation, `BlobsSidecar` is not required. The beacon node can process the block "optimistically”. # P2P network processing We'll look at `beacon_block` and `blobs_sidecar` gossip topics separately. In `blobs_sidecar`, the node will reject if the blobs are malformed (i.e. the `BLSFieldElement` in an invalid range), and the kzg proof is incorrectly encoded as a compressed BLS G1 point. Lastly, it will reject if the sidecar signature is invalid. Before accepting the blobs, one last step is to run [validate_blobs_sidecar](https://github.com/ethereum/consensus-specs/blob/dev/specs/eip4844/validator.md#validate_blobs_sidecar) which verifies the aggregated proof within the side car object. Also before that, one quick verification to ensure that KZG proof is a correctly encoded compressed BLS G1 point. In `beacon_block`, the node will look at `blob_kzg_commitments` in `BeaconBlockBody` and reject the block if the kzg commitments are incorrectly encoded as compressed BLS G1 points and the commitments don't correspond to the versioned hashes in the transaction list # Forkchoice processing As we can see from the Beacon chain and P2P network processing, either object could be processed independently. There’s no strict dependency on each other. This changes in the fork choice land. Today, client implementations use forkchoice store to cache useful status such as `VALID`, `OPTIMISTIC`, `INVALID`… etc With EIP4844, It’ll also be useful for forkchoice store to mark the block (i.e. a node in protoarray term) whether the data is available (i.e has a valid sidecar). `is_data_available` is a piece of important information and is meant to change with later sharding upgrades. `is_data_available` retrieves the matching `BlobsSidecar` of a given `blob_kzg_commitments` and validates the sidecar is sound. It's important to note that a block can not be `VALID` until `is_data_available` returns `true`. Until then, the block should only be treated as `OPTIMISTIC`. Given such constraints, we analyze two scenarios 1. Block is received before sidecar ![](https://i.imgur.com/FdTQ97I.png) This is a happy case. `block` (blue) is received before `sidecar` (red). `block` arrives in p2p service where the node validates the block passes p2p conditions before gossip to its peers and sending it to blockchain service where node validates the block according to consensus rule then send it to forkchoice store and DB for persistent storage. Once the block has been cached in forkchoice store, the node can validate `sidecar` and then send it to forkchoice store to make the block has a valid sidecar and `VALID` before storing it to DB for persistent storage. The total completion time depends on the time gap between the two objects, which should be minimized for performance. 2. Sidecar is received before block ![](https://i.imgur.com/wxqDxbV.png) This is a tricky case. `sidecar` (grey) is received before `block` (blue). In this case, we store `sidecar` in a pending queue until the corresponding block is received, processed, and stored in forkchoice store. The total completion time depends on when the block can arrive Loosely coupled `sidecar` and `block` increases complexity for the scenario above. # Further considerations - With checkpoint sync, a node will likely have to request `BeaconBlocksByRange` for backtracking. In a tightly coupled world, we don't need to call `BlobsSidecarsByRange` separately - Anything else?
Last changed by  
 
Prysmatic Labs
eth2 development team
1793

Read more

Migrating Prysm to Slog

Contextual logging:

12/6/2023
Running BOLD Challenges on Sepolia

This guide will set up

11/17/2023
Generalized History Commitment

Today in BOLD, challenges can have subchallenges, and we have a total of 3 levels. We do this because at each challenge, validators have to commit to N state roots of a block’s history, which can be very expensive to compute. For example, computing 2048 machine hashes for the execution of a block and collecting them takes around 5 minutes on a modern MacBook Pro.

8/16/2023
If I Had a Reset Button...

If you knew what you know now, and you were starting a new CL from scratch, what structures would you use? Let me start myself to set the tone, you can make it language specific (for example Go and avoid templated methods) but I am not really interested in the small details but rather the main components of the critical parts of code. Typical questions I’m thinking about areWould you use a custom allocator for the beacon-state/validators/attestations structures?Would you use a struct like object for them or would you use a functional style object. The typical situation is this:We have a buffer with 8192 roots in the BeaconState, but these roots are sequential, if state for N had up to root r, the state for N+1 will share essentially all of the roots as that for N except at one point where it would have changed one. If you store vectors and then you want to share the common part then you are lead to structures like the multi-value slice or similar structures, where you have a base slice and then somewhere else you store the diffs (or some mathematically equivalent statement). Another way to store the object is as a linked list (with some additions to make it into a round buffer) and when you change one element you just change a single node on that list, someone that holds the pointer to the current node will resolve to the current full round buffer, while someone that holds the pointer to the previous-to-modified node, will resolve (by traversing the links from it) to the previous full round buffer. If you have a fork this still works, you just need to add a few pointers on either side of the fork. The point is that from the point of view of the structure itself, there’s no “base state” and some diffs, and rather you can just delete the pointer and let GC in Go, or the Destructuror in C++ deal with the rest. No need to be pruning yourself from a multi-value slice for example. On the other side, the multi-value slice would incur in way less memory fragmentation if you are dealing with a single state in the happy case.There are variations of this for all structures in the Beacon State and there are benefits and cons for each one of them.What sort of thread management would you use? (I think here there won’t be a disagreement and just relying on the Go runtime is fine)What sort of fork management would you use? I originally thought about templating, something like having the Fork being the template parameter for a BeaconBlock structure. In C++ at least, the best structure seems to be vertical polymorphism in the structures themselves like having class AltairBlock : public Phase0Block but not use methods of these structures, but rather pure functions like attestations(block Block) []Attestations and just let AltairBlock and Phase0Block satisfy the concept Block, which in this case would mean they both have a member attestations so you can just simply implement that function as return block.attestations. In Go this seems to be quite complicated, templates as far as I could read do not allow us to do this, and interfaces themselves we have them, but we still are forced to repeat the underlying structures completely for AltairBlock and Phase0Block instead of invisibly inheriting and obscuring the changed ones.Would you trade security and correctness for performance? I think this is a funny one: if you’re coding for 0.1% of the market, then you can do lots of shit like only keeping the last two states and break if there’s a depth 3 reorg. That’s an extreme case, but that’s probably not so far away from what Erigon’s CL may be, I would bet that they just follow blindly whatever is voted. But besides the extreme cases, how about Optimistic sync? I am pretty sure we are the only ones that considered a recursive system in which a bunch of branches turned out to be invalid. I am convinced that you can simplify a lot the optimistic status dealing if you just ignore correctness in a bunch of cases, with no real danger to the network.How would you deal with the engine? Would you change the interaction with the engine if you knew that ALL your users are using MEV-Boost? would you optimize the builder code in detriment of the local code? for example, it could be perfectly valid to not wait for local blocks and not request them in advance until it’s late in the block in case we know we are not resource-heavy. Not wait for the result of FCU is also good, since we are guaranteed already from a VALID response from NewPayload (this is something we could do right now on Prysm).

8/2/2023
Read more from Prysmatic Labs
Published on HackMD