Cumulus - HackMD

--- title: Cumulus tags: Polkadot, parachains --- # Cumulus - Obtaining BABE header ## Aim of this write up We want to find a way to use relay validators' BABE VRF output as a source of randomness in the consensus of the parachain. This write up aims to make it easier to understand the codebase of Cumulus (parachain framework build on substrate). ## BABE VRF output Validators on the relay chain are randomly selected for block production using BABE and the output is included in the relay chain blocks (relay_parent). BABE Header of block B consist of: $(d, \pi, j, s)$ where $\pi$, $d$ are the results of the block lottery for slot $s$ $j$: index of thge block producer producing the block in the current authority directory of the current epoch $s$: is the slot at which the block is produced Details in 62.2. Block Production Lottery of the [Polkadot Host Specs](https://github.com/w3f/polkadot-spec/releases/tag/v0.1.0-pre) The BABE Header is included in as a `DigestItem` of the `preRuntime` type in the relay_parent header, `Digest` field ( defined in `primitives/runtimes/src/digest`, a struct contains a Vec of `DigestItem<T>`). ## Cumulus The current version of the parachain testnet is Rococo. I.e. there is a relay chain network (polkadot binary) that have been configured to be able to add / validate parachains. The parachains is in `cumulus/rococo-collator` which uses all the other crates in the cumulus repo. Terminology: Personally got quite confused with the use of the word collator which seems to generally mean the parachain full node, i.e. Backend (DB), Client (consensus, block production, other services) and Runtime. However, in the code, the repo seperates out crates like Collator, Consensus and Runtime. So here we just use *parachain node* as the binary that runs everything. > TODO: wait, so what does the collator have? > Would it not be easier to run BABE in the parachain for FC::ExpectedConsensus? > ### Consensus This is the consensus engine and it follows the relay chain node that is also running. The parablock is finalised when the finalised relay_parent block contains the a blockhash for the parablock for that para_id. Similar to new best relay_parent block which also sets the new best paraBlock. The consensus engine starts with `run_parachain_consensus` function when the client the parachain node starts (collator or not). And should not return unless it fails. There are 2 streams that is polled, one for block import and the other for finalisation, the engine just see what it can do when each of the streams below has something to make progress on: 1. `follow_new_best` 2. `follow_finalized_head` Note: the `.fuse()` is called on the `Stream` to be used in the `select!` macro to satisfy trait bound for `Fuse`. > TODO: do we use new best or just finalised as randomness beacon #### `follow_new_best` This sets the new best block for the parachain node following the relay chain's new best head. This may be the best blocks from relay chain / other parachain nodes. `Stream` are set up in this function to poll both sources and make progress on whichever is available: - the relay chain node `relay_chain.new_best_heads()` uses `handle_new_best_parachain_head()` or - `new_best_heads()` is a fn impl of the RelayChain Trait defined in this crate, which basically uses `import_notification_stream()` just like the parachain - `handle_new_best_parachain_head()` ensures that the block is already known `BlockStatus::InChainWithState` to that parachain node before importing and announcing to the rest of the parachain nodes. Otherwise, it records the relay_parent block hash as the parachain node's local `unset_best_header`, to be read in `handle_new_block_imported` - from an announcement made by other parachain node `parachain.import_notification_stream()` uses `handle_new_block_imported()` - `import_notification_stream()` is a fn impl of `BlockchainEvents` trait for the parachain node client, which gets block import event stream - `handle_new_block_imported` checks if the `unset_best_header` was set > (TODO: not sure why the other parachain node will have `InChainWithState`) #### `follow_finlized_head` This sets the finalized block for the parachain from the finalised relay_parent block. Any block before this are also considered finliased. #### Import Queue When the parachain node imports a block, it goes through an abstract queue (same for all substrate nodes). For the current implementation of Cumulus, it is a `BasicQueue` (as opposed to block production specific `BabeImportQueue`)where information comes in, put into a pool, checked for validity and imported. The `Verifier`, used in this `BasicQueue`, checks the `Inherent` in the Runtime (Inherent is the data in the block, a subset of InherentData produced by the client on block production). > TODO: runtime says `data.check_extrinsics()` for checking inherent.. what does this do? Finally, when the queue is ready for the block to be imported, client will import the block as it (or a struct wrapping it to add checks) imports the `BlockImport` trait ### Collator Collator produces collation for a single parachain. Collator is started when the collator flag is passed for the parachain node. In `start_collator()`, a `CollationGenerationConfig` is created and pass as the Initilise message to the collation generation subsystem. Within the config, there is a produce_candidate function which is called to produce parachain candidate. > TODO: overseer also oversee the parachain and polkadot? different overseer? Currently, a new block block (best / finalised?) causes the collator who initially prepared the block to announce it to the other parachain nodes Collation Subsystem [details](https://w3f.github.io/parachain-implementers-guide/node/collators/collation-generation.html) ### Service > `start_collator()` starts collator and consensus with `polkadot_full_node.client.execute_with()` - polkadot runs these?