2025-07-02 Lighthouse PeerDAS Performance Test

# Lighthouse PeerDAS Performance Test and Analysis ## Set up - Local Kurtosis with 2 supernodes - 10 logical cores - 30G RAM - 36/48 blobs per block - spamoor throughput 36 blob txs / 200 eox txs per slot **NOTE**: This set up is *very* different to a realistic network, as each node only has a single peer. However, this still give us some preliminary idea of how things look like at high blob count. ## Summary of Findings - Block production and publishing times increase as blob count increases (up to 2.5s at 48 blobs). - Beacon processor bottlenecks observed in `api_request_p0` and gossip data column handling. - Sinlge KZG verification on gossip columns contributes to the attestation delays - SSZ is 30–35% faster than JSON for block production payloads ## Optimisations - Switch to SSZ for block production: `get_validator_blocks_v3_ssz` - Low hanging fruit, this shaves `~20ms` off block production time - Queue and batch verify gossip columns - This could shave up to `~240ms` off block availability time (for supernode) - Potentially more savings if we batch verify the headers too - Remove KZG verification on blobs from the local EL (both `getPayloadV5` and `getBlobsV2`) - This potentially reduces block production time by `~500ms` - For block publishing, we could remove the check also for locally built blocks, and gives us another `~500ms` savings! - Similar on `getBlobsV2`, for supernodes, we could reduce `~500ms` on block availability time. - Immeidately make block available after receiving the block and 64 valid columns (https://github.com/sigp/lighthouse/pull/7598) ## Metrics ### `http_api_path_times` #### Block Production Block production time (`eth/v3/validator/blocks`) increased to up to 1s: ![image](https://hackmd.io/_uploads/rkG6D8GSll.png) We could try switching over to SSZ (`get_validator_blocks_v3_ssz`) to compare results. #### Block Publishing Processing time for `eth/v2/beacon/blocks` endpoint has increased: - Electra: ~250ms (max 6/9 blobs) - Fulu: up to 1s (max 6/9 blobs) - BPO1: up to 2.5s (max 36/48 blobs) ![image](https://hackmd.io/_uploads/HyHXNLGHee.png) And this matches what we see in `beacon_block_delay_attestable_slot_start` as well ![image](https://hackmd.io/_uploads/HJevBIGSge.png) We also see an increase of `beacon_block_delay_imported_time` to ~600ms ![image](https://hackmd.io/_uploads/BJ7TU8zrle.png) ### `beacon_processor_worker_time` We see quite a significant amount of time spent on processing `api_request_p0` and processing gossip data columns: ![image](https://hackmd.io/_uploads/HkUZwUGSgl.png) #### `gossip_block` Interestingly, the `gossip_block` task time for nethermind node remains unchanged however for the geth node it spiked up to 240ms+ after BPO fork1: ![image](https://hackmd.io/_uploads/HkUec8Grxg.png) #### `gossip_data_column` ~~Surprising result here, gossip data column performance remains pretty much unchanged after max blob count increased from 9 to 48. This suggests that the KZG verification cost increase is quite minimal, and the gossip verification cost is potentially elsewhere:~~ ^ This is false, as KZG verification takes more time at higher blob count. ![image](https://hackmd.io/_uploads/rk2ocUfSex.png) I've done some more testing and I'm seeing numbers between 8ms and 18ms (`time.busy`), so the metrics were actually not very accurate due to bad prometheus buckets: ``` 2025-07-06T14:59:16.779858Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=1}: beacon_chain::data_column_verification: close time.busy=8.45ms time.idle=941ns 2025-07-06T14:59:16.815840Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=15}: beacon_chain::data_column_verification: close time.busy=10.4ms time.idle=1.38µs 2025-07-06T14:59:16.816173Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=9}: beacon_chain::data_column_verification: close time.busy=11.0ms time.idle=1.44µs 2025-07-06T14:59:16.816432Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=14}: beacon_chain::data_column_verification: close time.busy=11.1ms time.idle=1.44µs 2025-07-06T14:59:16.818649Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=24}: beacon_chain::data_column_verification: close time.busy=11.6ms time.idle=1.38µs 2025-07-06T14:59:16.819102Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=12}: beacon_chain::data_column_verification: close time.busy=13.9ms time.idle=1.81µs 2025-07-06T14:59:16.819959Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=13}: beacon_chain::data_column_verification: close time.busy=14.7ms time.idle=927ns 2025-07-06T14:59:16.822445Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=10}: beacon_chain::data_column_verification: close time.busy=17.2ms time.idle=1.71µs 2025-07-06T14:59:16.822823Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=8}: beacon_chain::data_column_verification: close time.busy=17.7ms time.idle=1.25µs 2025-07-06T14:59:16.823140Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=23}: beacon_chain::data_column_verification: close time.busy=16.1ms time.idle=1.22µs 2025-07-06T14:59:16.826797Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=27}: beacon_chain::data_column_verification: close time.busy=10.9ms time.idle=840ns 2025-07-06T14:59:16.826937Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=33}: beacon_chain::data_column_verification: close time.busy=10.4ms time.idle=892ns 2025-07-06T14:59:16.827017Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=28}: beacon_chain::data_column_verification: close time.busy=10.8ms time.idle=725ns 2025-07-06T14:59:16.831967Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=25}: beacon_chain::data_column_verification: close time.busy=15.7ms time.idle=961ns 2025-07-06T14:59:16.832092Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=41}: beacon_chain::data_column_verification: close time.busy=12.1ms time.idle=733ns 2025-07-06T14:59:16.832256Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=38}: beacon_chain::data_column_verification: close time.busy=13.5ms time.idle=983ns 2025-07-06T14:59:16.834612Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=46}: beacon_chain::data_column_verification: close time.busy=11.7ms time.idle=912ns 2025-07-06T14:59:16.835096Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=43}: beacon_chain::data_column_verification: close time.busy=12.6ms time.idle=791ns 2025-07-06T14:59:16.835584Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=40}: beacon_chain::data_column_verification: close time.busy=16.4ms time.idle=927ns 2025-07-06T14:59:16.845956Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=54}: beacon_chain::data_column_verification: close time.busy=8.95ms time.idle=1.02µs 2025-07-06T14:59:16.849790Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=53}: beacon_chain::data_column_verification: close time.busy=13.9ms time.idle=1.25µs 2025-07-06T14:59:16.850050Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=57}: beacon_chain::data_column_verification: close time.busy=11.2ms time.idle=724ns 2025-07-06T14:59:16.851004Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=59}: beacon_chain::data_column_verification: close time.busy=11.1ms time.idle=939ns 2025-07-06T14:59:16.851240Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=52}: beacon_chain::data_column_verification: close time.busy=16.4ms time.idle=1.16µs 2025-07-06T14:59:16.852728Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=61}: beacon_chain::data_column_verification: close time.busy=11.8ms time.idle=1.00µs 2025-07-06T14:59:16.852814Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=56}: beacon_chain::data_column_verification: close time.busy=14.9ms time.idle=1.13µs 2025-07-06T14:59:16.855261Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=63}: beacon_chain::data_column_verification: close time.busy=12.4ms time.idle=1.07µs 2025-07-06T14:59:16.858261Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=62}: beacon_chain::data_column_verification: close time.busy=16.3ms time.idle=1.23µs 2025-07-06T14:59:16.859613Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=65}: beacon_chain::data_column_verification: close time.busy=10.8ms time.idle=1.04µs 2025-07-06T14:59:16.859971Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=68}: beacon_chain::data_column_verification: close time.busy=10.1ms time.idle=711ns 2025-07-06T14:59:16.860254Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=70}: beacon_chain::data_column_verification: close time.busy=9.16ms time.idle=762ns 2025-07-06T14:59:16.863504Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=64}: beacon_chain::data_column_verification: close time.busy=17.4ms time.idle=713ns 2025-07-06T14:59:16.865082Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=69}: beacon_chain::data_column_verification: close time.busy=14.9ms time.idle=722ns 2025-07-06T14:59:16.866071Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=71}: beacon_chain::data_column_verification: close time.busy=14.7ms time.idle=827ns 2025-07-06T14:59:16.881093Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=73}: beacon_chain::data_column_verification: close time.busy=13.4ms time.idle=972ns 2025-07-06T14:59:16.882666Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=75}: beacon_chain::data_column_verification: close time.busy=13.7ms time.idle=967ns 2025-07-06T14:59:16.883891Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=76}: beacon_chain::data_column_verification: close time.busy=13.6ms time.idle=811ns 2025-07-06T14:59:16.889338Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=85}: beacon_chain::data_column_verification: close time.busy=12.9ms time.idle=898ns 2025-07-06T14:59:16.890130Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=77}: beacon_chain::data_column_verification: close time.busy=13.7ms time.idle=1.20µs 2025-07-06T14:59:16.892505Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=79}: beacon_chain::data_column_verification: close time.busy=16.1ms time.idle=1.83µs 2025-07-06T14:59:16.893451Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=84}: beacon_chain::data_column_verification: close time.busy=17.0ms time.idle=1.67µs 2025-07-06T14:59:16.894242Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=81}: beacon_chain::data_column_verification: close time.busy=17.8ms time.idle=1.85µs 2025-07-06T14:59:16.894441Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=87}: beacon_chain::data_column_verification: close time.busy=18.0ms time.idle=2.09µs 2025-07-06T14:59:16.894665Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}: beacon_chain::data_column_verification: close time.busy=18.3ms time.idle=729ns 2025-07-06T14:59:17.078836Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=91}: beacon_chain::data_column_verification: close time.busy=11.2ms time.idle=838ns ``` And a breakdown shows out of 18.3ms - 13.8ms was spent on kzg verification - 2.24ms was spent on proposer signature verification (we can optimise this via caching or batch verification) ``` 2025-07-06T14:59:16.876406Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_data_column_sidecar: beacon_chain::data_column_verification: close time.busy=339ns time.idle=828ns 2025-07-06T14:59:16.876420Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_index_matches_subnet: beacon_chain::data_column_verification: close time.busy=401ns time.idle=574ns 2025-07-06T14:59:16.876426Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_sidecar_not_from_future_slot: beacon_chain::data_column_verification: close time.busy=823ns time.idle=641ns 2025-07-06T14:59:16.876463Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_slot_greater_than_latest_finalized_slot: beacon_chain::data_column_verification: close time.busy=923ns time.idle=1.54µs 2025-07-06T14:59:16.878366Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_is_first_sidecar: beacon_chain::data_column_verification: close time.busy=627ns time.idle=1.06µs 2025-07-06T14:59:16.878392Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_column_inclusion_proof: beacon_chain::data_column_verification: close time.busy=15.7µs time.idle=718ns 2025-07-06T14:59:16.878401Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_parent_block_and_finalized_descendant: beacon_chain::data_column_verification: close time.busy=1.23µs time.idle=758ns 2025-07-06T14:59:16.878415Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_slot_higher_than_parent: beacon_chain::data_column_verification: close time.busy=373ns time.idle=794ns 2025-07-06T14:59:16.880665Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_proposer_and_signature: beacon_chain::data_column_verification: close time.busy=2.24ms time.idle=1.16µs 2025-07-06T14:59:16.894646Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_kzg_for_data_column: beacon_chain::data_column_verification: close time.busy=13.8ms time.idle=1.60µs 2025-07-06T14:59:16.894656Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:observe_slashable: beacon_chain::data_column_verification: close time.busy=3.01µs time.idle=737ns 2025-07-06T14:59:16.894662Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:observe_gossip_data_column: beacon_chain::data_column_verification: close time.busy=629ns time.idle=632ns 2025-07-06T14:59:16.894665Z INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}: beacon_chain::data_column_verification: close time.busy=18.3ms time.idle=729ns ``` ### KZG computation and verification #### `beacon_data_column_sidecar_computation_seconds` Seeing 30-38ms on averagae, this is acceptable. ![image](https://hackmd.io/_uploads/H1PvDP7Bxe.png) #### KZG verification (single vs batch) These metrics look pretty bad and hitting the max bucket, which isn't expected. In unit tests, single column verifcation takes only ~5ms and the data conversion overhead is minimal. This needs further investigation. `beacon_kzg_verification_data_column_single_seconds` ![image](https://hackmd.io/_uploads/ryk4uwXSxl.png) `beacon_kzg_verification_data_column_batch_seconds` ![image](https://hackmd.io/_uploads/HJBSuDXHge.png) More on KZG verification in the KZG section below. ### Other observations No reconsturction performed, suggesting the recent optimisation to delay reconstruction worked. We should try running a network withholding some columns to see how reconstruction affects performance. ## KZG verify proof time investigation I wrote a test to measure kzg verification time for 48 blobs. The vec conversion overhead in our code is quite minimal (in microseconds), the library's `verify_cell_kzg_proof_batch` method is taking the bulk of the verificaiton time - below are numbers for 48 blobs: - 1 column batch: 5.76ms - 8 columns batch: 34.3ms - 64 columns batch : 257ms - 128 columns batch: 508ms if we don't batch them, 128 columns is expected to take 5.76 * 128 = 737ms, and then there's also overhead on verifying the signed block header. ``` col_count: 1 validate_data_columns:validate_data_columns:preprocess_columns: beacon_chain::kzg_utils: close time.busy=44.2µs time.idle=500ns validate_data_columns:validate_data_columns:verify_kzg_batch:verify_cell_kzg_proof_batch: kzg: close time.busy=5.75ms time.idle=209ns validate_data_columns:validate_data_columns:verify_kzg_batch: beacon_chain::kzg_utils: close time.busy=5.76ms time.idle=249ns validate_data_columns:validate_data_columns: beacon_chain::kzg_utils: close time.busy=5.97ms time.idle=250ns validate_data_columns: beacon_chain::kzg_utils: close time.busy=6.04ms time.idle=27.7µs col_count: 8 validate_data_columns:validate_data_columns:preprocess_columns: beacon_chain::kzg_utils: close time.busy=15.5µs time.idle=166ns validate_data_columns:validate_data_columns:verify_kzg_batch:verify_cell_kzg_proof_batch: kzg: close time.busy=34.3ms time.idle=208ns validate_data_columns:validate_data_columns:verify_kzg_batch: beacon_chain::kzg_utils: close time.busy=34.3ms time.idle=250ns validate_data_columns:validate_data_columns: beacon_chain::kzg_utils: close time.busy=34.3ms time.idle=167ns validate_data_columns: beacon_chain::kzg_utils: close time.busy=34.4ms time.idle=458ns col_count: 64 validate_data_columns:validate_data_columns:preprocess_columns: beacon_chain::kzg_utils: close time.busy=108µs time.idle=166ns validate_data_columns:validate_data_columns:verify_kzg_batch:verify_cell_kzg_proof_batch: kzg: close time.busy=257ms time.idle=458ns validate_data_columns:validate_data_columns:verify_kzg_batch: beacon_chain::kzg_utils: close time.busy=257ms time.idle=208ns validate_data_columns:validate_data_columns: beacon_chain::kzg_utils: close time.busy=257ms time.idle=166ns validate_data_columns: beacon_chain::kzg_utils: close time.busy=257ms time.idle=917ns col_count: 128 validate_data_columns:validate_data_columns:preprocess_columns: beacon_chain::kzg_utils: close time.busy=255µs time.idle=209ns validate_data_columns:validate_data_columns:verify_kzg_batch:verify_cell_kzg_proof_batch:verify_cell_kzg_proof_batch_preprocess_columns: kzg: close time.busy=2.67µs time.idle=126ns validate_data_columns:validate_data_columns:verify_kzg_batch:verify_cell_kzg_proof_batch:verify_cell_kzg_proof_batch_verify_kzg_batch: kzg: close time.busy=508ms time.idle=584ns validate_data_columns:validate_data_columns:verify_kzg_batch:verify_cell_kzg_proof_batch: kzg: close time.busy=508ms time.idle=166ns validate_data_columns:validate_data_columns:verify_kzg_batch: beacon_chain::kzg_utils: close time.busy=508ms time.idle=166ns validate_data_columns:validate_data_columns: beacon_chain::kzg_utils: close time.busy=508ms time.idle=209ns validate_data_columns: beacon_chain::kzg_utils: close time.busy=508ms time.idle=1.21µs ``` ## Measuring SSZ vs JSON serde Costs for `SignedBlockContents` `BlockContents` and `SignedBlockContents` is used between the VC and BN during block production, and the overhead in serde will impact overall block production performance. ``` blob count: 9 json: ser 6.38975ms de 4.65625ms json total: 11.046ms ssz: ser 2.59725ms de 4.453083ms ssz total: 7.050333ms blob count: 18 json: ser 13.165875ms de 8.765542ms json total: 21.931417ms ssz: ser 5.59925ms de 8.58475ms ssz total: 14.184ms blob count: 24 json: ser 16.810292ms de 11.476958ms json total: 28.28725ms ssz: ser 7.12375ms de 11.616333ms ssz total: 18.740083ms blob count: 48 json: ser 33.965ms de 23.722291ms json total: 57.687291ms ssz: ser 14.088833ms de 23.015667ms ssz total: 37.1045ms ``` See also [Benchmarking Serde Costs in getPayloadV5 Engine API](https://hackmd.io/@jimmygchen/HygYqsmSle).

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.