HC Rust implementation / onboarding

HC Rust implementation / onboarding === ###### tags: `HC`, `B`, `B3` _June 2022_ > The goal of this document is to give a quick walk-through of the implementation of HC with pointers to the right places in the code (of the MVP). ## Notes on current implementation > Disclaimer: While we were attempting to create code as close to production as possible, the code is in an MVP stage. We had to de-risk several parts of the design to evaluate if _"it was actually possible"_. Adding to this the lack of detailed reviews/audits of the code, we may have left horrible things behind so please feel free to suggest any changes, improvements and report potential bugs or flaws. - Decoupling consensus interface: https://github.com/filecoin-project/eudico/blob/eudico/chain/consensus/iface.go - All consensus implementations for subnets should share the same consensus interface. ----- - Basic commands for HC: - Subnet commands: https://github.com/filecoin-project/eudico/blob/eudico/cmd/eudico/subnet.go - Checkpointing commands: - Atomic execution commands: https://github.com/filecoin-project/eudico/blob/eudico/cmd/eudico/atomic-exec.go - It can be a good way to understand all the features, functionalities, and relevant functions exposed by the API related to HC. ---- - The core of HC is implemented here: https://github.com/filecoin-project/eudico/tree/eudico/chain/consensus/hierarchical - `actors` include the Go implementation of SCA and the subnet actor as `spec_actors`. With FVM these are replaced by (which are the ones to be used for the Rust implementation of HC): - SCA (Rust): https://github.com/adlrocha/builtin-actors/tree/master/actors/hierarchical_sca - Subnet actor (Rust): https://github.com/adlrocha/hc-subnet-actor - `atomic` includes all the code related to locking primitives for the implementation of actors that support atomic executions: https://github.com/filecoin-project/eudico/tree/eudico/chain/consensus/hierarchical/atomic - This is not yet implemented in Rust and it'll (probably) become an independent crate or part of builtin-actors and SCA. - `checkpoints` includes the checkpointing type and related functions. This has changed a lot for the [Rust SCA](https://github.com/adlrocha/builtin-actors/blob/master/actors/hierarchical_sca/src/checkpoint.rs) so it'll have to also change accordingly here: https://github.com/filecoin-project/eudico/tree/eudico/chain/consensus/hierarchical/checkpoints - `subnet/manager` includes all of the logic for the main process that handles the lifecycle of subnets in the peer (instantiating the stack for a new subnet, listening to events of a subnet and accessing the state, etc.): https://github.com/filecoin-project/eudico/tree/eudico/chain/consensus/hierarchical/subnet/manager - _along with the actors this is probably the key part of the HC implementation in the peer_. - `subnet/resolver` implements the libp2p protocol responsible for resolving objects from the state of other subnets: https://github.com/filecoin-project/eudico/tree/eudico/chain/consensus/hierarchical/subnet/resolver > Since the Go MVP, some parts of the spec have slightly change so when facing _inconsistencies_ between the spec and the implementation trust what the spec and the Rust implementation of the actors say. > As we develop the Rust code we will have to also propagate changes to the Go implementation (so we'll be developing some of these parts together). ---- There are slight changes throughout the base code to support HC: - Like the way cross-net messages are executed: https://github.com/filecoin-project/eudico/blob/eudico/chain/consensus/common/executor.go - The validation of messages: https://github.com/filecoin-project/eudico/blob/eudico/chain/consensus/common/cns_validations.go - Or the proposal of cross-net messages when assembling blocks: https://github.com/filecoin-project/eudico/blob/4d729dbc7f4bd702196a09c804180a207d63ad25/chain/consensus/tspow/mine.go#L73 Finally, we keep a list of consensus implementations for subnets (mainly, `mir`, `dummy`, `tspow`, `tendermint`, `delegated`): https://github.com/filecoin-project/eudico/tree/eudico/chain/consensus There are other slight changes on builtin-actors for the operation of HC, but as we are sharing actos between the two implementations you shouldn't worry about it. ## HC Rust implementation Plan 1. Implementation of delegated consensus to decouple the consensus interface and see how we plug in different implementations. 2. Join and spawn a subnet. Validate messages in parallel from the two subnets. 3. End-to-end lifecycle of a subnet (joining, sending messages within the subnet, leaving, syncing). 4. Cross-net messages between different subnets. 5. Integrate Narwhal and Tusk into HC. 6. (BONUS) Atomic execution.