# Forest School

The purpose of this document is to figure out how to add the machinery needed for [Hierarchical Consensus](https://github.com/protocol/ConsensusLab/blob/hc/spec/specs/hierarchical_consensus.md) to [Forest](https://github.com/ChainSafe/forest).

__NOTE__: This was made on version 0.4 of the Forest codebase. It is currently 0.6 any some of the descriptions below no longer apply, for example Bitswap is used slightly differently.

## Code Exploration

The following is a DFS walk of the major components, starting from the `daemon` entry point.

daemon.rs:
- generate keys
- setup KeyStore
- setup prometheus
- setup the DB (RocksDB or Sled):
- the DB implements __``BlockStore``__
- `BlockStore` stores low level IPLD data
- setup the __`ChainStore`__:
- uses DB
- functions:
- cached block headers per epoch
- selects the heavies `Tipset`
- track subscriptions for `Tipset` changes
- finds recent beacon entries
- restore full block from header
- loads the StateTree from a `Tipset`
- loads an actor by address from the StateTree from a `Tipset`
- exports a tipset to a CAR
- walk the DAG backwards for a few epochs
- get a tipset publisher (includes rollbacks) from `ChainStore`
- read Genesis and set on `ChainStore`
- setup the __`StateManager__`:
- uses the `ChainStore`
- publicly exposes `BlockStore` and `ChainStore`
- has an __FVM Wasm engine__
- can be used to:
- calculate and cache projected state for tipsets
- load an actor from a given StateTree root
- serve randomness from the Beacon or the Chain (both done by `ChainRand`)
- consult the Power Actor to check if some miner is slashed in a given state
- consult the Power Actor to check the power of any Actor
- consult any Miner Actor to get the worker address
- apply a whole tipset worth of messages onto some parent state:
- starts with applying cron for all skipped epochs
- creates a VM for each epoch based on what version applies
- saves receipts and flushes the VM in the end; these are the two results
- apply a single message on a tipset without persisting the results
- checks the eligibility of a miner to mine blocks:
- checks if the miner has minimum power
- checks fee debts the miner has
- checks any consensus faults, which elapse after some epochs
- loops to find messages and message receipts in past blocks
- wait for a message to appear on chain and be finalized
- find BLS or Secp256k1 address of an account in StateTree
- get the balance of an Actor at a given state
- look up the balances of an address in the Market Actor escrow and locked tables
- validate that parent states match up in a chain
- get the circulating supply from Genesis
- sync from a snapshot file
- uses the `StateManager`
- does Full verification
- fetch and verify proof parameters
- takes bootstrap peers from config or genesis
- setup __`Libp2pService`__:
- uses the keys
- uses the `ChainStore`
- emits __NetworkEvent__ to app:
* `PubsubMessage`:
* `GossipBlock`: Block header and vector of BLS and Secp256k1 message CIDs
* `SignedMessage`: Message intended for an Actor
* `HelloRequest`: Contains the heaviest tip and genesis info of the peer
* `ChainExchangeRequest`: A list of CIDs to start from, and an epoch length to serve; blocks and/or messages
* `PeerConnected`
* `PeerDisconnected`
* `BitswapBlock`
- consumes __NetworkMessage__ from app:
* `PubsubMessage`: arbitrary message going into a topic
* `ChainExchangeRequest`
* `HelloRequest`
* `BitswapRequest`: advertise the need for a CID
* `JSONRpcRequest`: libp2p node related requests (addr, peers, connect, disconnect)
- builds a libp2p transport:
- TCP and WebSockets
- Yamux: Parity multiplexer
- Mplex: libp2p multiplexer
- builds a __`Swarm`__:
- uses the transport
- uses the __ForestBehaviour__:
- uses the network keys
- handles all Filecoin protocols
- derives NetworkBehaviour for:
* `Gossipsub`: emit event
* `Discovery`: connect bitswap, emit event
* `Ping`: just trace
* `Identify`: just trace
* RequestResponse of Hello:
* Request: respond and emit event
* Response: complete waiting request by ID
* RequestResponse of ChainExchange:
* Request: emit event with a response channel
* Response: complete waiting request by ID
* Bitswap: emit want/receive; ignore cancel
- has:
- event queue
- RequestResponse response channel tracker
- can be used to:
- bootstrap peers via discovery
- return discovered peers and their addresses
- publish to gossipsub
- subscribe to gossipsub
- send a `Hello` request to a peer
- send a `ChainExchange` request to a peer
- send a Bitswap block
- ask for a CID via Bitswap
- subscribes the `Swarm` to block and msg pubsub topics
- bootstraps the `Swarm` with `Kademlia`
- limits incoming/outgoing connections
- listens to outgoing `NetworkMessage` and relays them to the `Swarm`
- listens to __`SwarmEvent`__ which has the following ForestBehaviour variants, mostly emits `NetworkEvent`
* `PeerConnected`: emit
* `PeerDisconnected`: emit
* GossipMessage: turns into `PubsubMessage`
* `GossipBlock`: emit
* `SignedMessage`: emit
* `HelloRequest`: emit
* `ChainExchangeRequest`: spawns a task to respond from the `ChainStore`; can get arbitrary number of blocks
* BitswapReceivedBlock: puts data into the Blockstore, (can be invalid CID, maybe unsolicited), then notifies any subscriber, and emits event
* BitswapReceivedWant: immediately looks up data in Blockstore and sends it via `Swarm`
- setup __``MpoolRpcProvider``__:
- uses the tipset publisher of the `ChainStore`
- uses the `StateManager`
- can be used to:
- subscribe to chain `HeadChange` events
- get heaviest chain (from the `StateManager` exposing the `ChainStore`)
- add a `ChainMessage` (signed or unsigned) to the `Blockstore`
- get the state of an arbitrary Actor at a `Tipset`
- get the messages in a block header
- get the messages in a `Tipset`
- load a `Tipset` by keys
- compute the base fee by `Tipset`
- setup __`MessagePool`__:
- uses the `MpoolRpcProvider`
- uses the NetworkMessage sender of the `Libp2pService
- it's full of public mutable fields >_< :
- tracking pending messages and required funds by sender
- the current `Tipset`
- exposes the `MpoolRpcProvider`
- signature caches
- republished messages
- tracks the local addresses (messages added locally)
- TODO: What is Calico height?
- subscribes to __`HeadChange`__ events:
- asks the `MpoolRpcProvider` for all Messages in reverted blocks and re-adds them as selected
- asks the `MpoolRpcProvider` for all Messages in applied blocks and removes them from selected
- adds selected messages to pending, if they pass validation (sig, gas limit)
- also puts them via `MpoolRpcProvider`
- at every republish intervals:
- asks the `MpoolRpcProvider` for the base fee to be used as a lower bound
- adds messages from the local addresses to the pending; if empty quit (what about non-local addresses? probably others will republish)
- creates a chain of messages where it has contiguous nonces
- calculates gas and rewards, checks if within block limit
- selects most profitable messages (I think, looks complicated)
- each selected message as a pubsub message to the `Libp2pService
- can be used to:
- add a message locally and send a pubsub message, if it passes validation (sig, size, fee limits)
- add a message if passes basic validation _and_ matches the state nonce expectations of the sender, and the sender has enough balance
- project a nonce/sequence of an address using the state as well as pending messages
- get pending messages of an address
- get signed messages in blocks by looking up cached signatures for unsigned messages as well
- estimate the gas premium based on the number of blocks included in something
- clear local and/or pending messages
- setup __`ChainMuxer`__:
- uses the `StateManager`
- uses the `MessagePool`
- uses `Libp2pService` `NetworkEvent` receiver
- uses `Libp2pService` `NetworkMessage` sender
- uses a `Tipset` sender
- uses a `Tipset` receiver
- builds a __``SyncNetworkContext``__:
- from `NetworkMessage` sender and `Blockstore` of the `StateManager`
- turns async request/response into single (async) methods that can be awaited
- can be used to:
- request chain exchange from the network (headers and/or messages)
- requests can go to a specific peer, or all top peers until succeeds
- tracks the peers it talked to (bad, failed, successful)
- request data through Bitswap, if it can't find it in the `Blockstore` (stores the result in the `Blockstore` when the request succeeds)
- send a Hello request and await the response
- has its own state:
* Idle: will evaluate network heads
* `Connect`: if Behind, starts to bootstrap; if `InRange`|`InSync`, starts to follow
* `Bootstrap`: when ready, go Idle
* `Follow`: if ready, means it failed; go Idle
- has the state of the Worker; TODO: what is this?
- can be used to:
- get bad blocks it's caching
- get the worker state
- get the full tipset (with messages) either from the Blockstore or from the network via `SyncNetworkContext`
- handles `GossipsubEvents (`NetworkEvent`; maintains metrics for them too):
- `Hello` request: get full tipset from store/network; waits/blocks; validate, persist, update peer head
- peer connected: looks up heaviest chain and sends Hello request; logs result in the peer manager of `SyncNetworkContext`
- peer disconnected: removes peer from `SyncNetworkContext`
- pubsub `SignedMessage`: adds it to the mempool
- pubsub data block: enrich a gossipsub block to full tipset using Bitswap via `SyncNetworkContext`; waits until ready, seems to stop processing anything else; validate, persist, update peer head
- `ChainExchangeRequest`: ignored, not supported; handled by `Lib2p2Service`
- `BitswapBlock`: ignored, not supported; handled by `Lib2p2Service`
- evaluate network heads:
- receive a couple of gossipsub events to build a sample of peers' chain heads
- find the heaviest tipset in the sample
- decide if it's `InSync`|`InRange`|`Behind` ; `InRange` means just 1 behind
- bootstrap:
- gets the local head of the chain and the one sampled from the network
- uses the network as _proposed_, the local as _current_
- creates a __`TipsetRangeSyncer`__:
- uses the `StateManager`
- uses the `Beacon`
- uses the `SyncNetworkContext`
- uses the `ChainStore`
- uses the `BadBlockCache`
- uses the `Genesis`
- gets a `current` and `proposed` head `Tipset`
- it rejects any proposal that is older than the current epoch
- starts syncing between the proposal and the current head
- it can add further sync tasks if the epoch and parents match
- during a `sync_tipset_range`:
- maintains the status of this task in the common `WorkerState` tracker
- maintains a `ProgressBar` about which epoch it's at
- walks backwards from proposed to current in `sync_headers_in_reverse`:
- if it knows that a parent it hasn't visited yet is in the `BadBlockCache`, it marks all descendants as bad as well
- asks the best peers for the oldest parents in the loop using the `SyncNetworkContext`, if they are not available already from the block store; asks for max 100 blocks to be fetched
- once it reaches the _current epoch_ it looks at whether it's on a fork
- if it's on a fork:
- requests `SyncNetworkContext` for a further 500 ancestors of the oldest ancestor which is at the same height as the _current_ but different
- tries to find the common ancestor
- stores all synced block headers to the `Blockstore`
- for each header it syncs the messages in it using `sync_messages_check_state`:
- if it's fully available in the `ChainStore` then it calls `validate_tipset` which calls __`validate_block`__ for each `Block` in the `Tipset`
- `block_sanity_checks` requires that a block has non-empty election proof, signature, BLS aggregate and a ticket
- loads the parents from the `ChainStore`
- loads some kind of "lookback" tipset from the `StateManager
- looks up the `Beacon` at the parents
- requires a specific timestamp based on the number of epochs passed (some levels can be empty, nulls)
- compares the timestamp to the System clock`
- gets the address of the miner at "lookback" from the `StateManager`
- checks the messages in the `Block` with `check_block_messages`:
- gets the BLS public key of each unsigned message sender
- verifies the aggregate BLS signature
- gets the price list at the epoch
- sums up the gas of all messages, checks block limit
- checks account nonces
- resolves senders to SECP public keys and checks the sigs
- checks the header root hash against he message contents
- call `validate_miner`:
- retrieve the power from the Power Actor using the `StateManager`in the parent state, but it discards the result
- calculates the basefee at the parents and compares against the header
- calculates the weigth at the parents and compares against the header
- validates the state root at the parents and compares against the header; receipts as as well; this is where it calls `tipset_state` of the `StateManager` which executes the messages in the parent blocks
- validate Winner Election Proof-of-SpaceTime: checks the election proof in the header and uses the `Statemanager` to check if the miner was eligible; uses the `Beacon` for the VRF; checks slashing`
- checks that the miner signed the block
- validates DRAND of the block
- validates Ticket Election Proofs: again `Beacon`, VRF
- validates Winning Proof-of-SpaceTime
- marks the block as validated in the `ChainStore`
- if the messages are not stored locally:
- fetch them from thet network with `SyncNetworkContext`
- call `validate_tipset`
- persist in the `Blockstore
- finally sets the proposed parent as the heaviest tipset in `ChainStore` `
- it consumes `NetworkEvents` which are passed to the above gossipsub handler handler, but the result (the resolved Tipsets) are dropped while in the bootstrap state
- waits for the missing tipsets to be synced
- follow:
- creates a __`TipsetProcessor`__:
- uses the `StateManager`
- uses the `Beacon`
- uses the `SyncNetworkContext`
- uses the `ChainStore`
- uses the `BadBlockCache`
- uses the `Genesis`
- mostly for passing them on as dependencies to the ``Tipset`RangeSyncer`
- can create a `TipsetRangeSyncer`:
- get the _current_ heaviest `Tipset` from the `ChainStore`
- pass the _proposed_ heaiest `Tipset` from the input group
- adds all the others too
- has `Idle`, `FindRange` and `SyncRange` states
- polls the `Tipset` Stream fed by the ChainMuxer
- assumes these have been validated by the __`TipsetValidator`__:
- checks that the epoch number is no higher than a reasonable maximum given the block production interval; uses the current system timestamp as a baseline
- checks that the message CIDs in the header match the content
- I assume at this point we don't necessarily have the parent tipsets at hand to really validate the weights
- pulls as many as it can at a time
- groups them by epoch and parent; if the group exists it _tries_ to add it, but drops it if it can't be added
- depending on state:
- when `Idle`:
- select the heaviest tipset; the weight is determined by the parents' weight, excluding the blocks in the tipset (I think, based on the BlockHeader comments)
- goes into `FindRange` mode with that tipset
- when `FindRange`:
- if currently syncing something it tries to merge it with the new tipsets by taking the entries from the same epoch and parents
- then finds the next heaviest group and earmarks it for the _next_ sync, merging with already earmarked next sync group if possible, replacing if heavier
- when `SyncRange`:
- tries to add tipsets to any currently ongoing range sync, if it receied new tipsets for the same epoch and parents
- again tries to extend or replace the _next_ sync with the next heaviest group
- then it polls the range syncers and moves along its own states, initiating the next sync if the current one finished
- the `Tipset`RangeSyncer is side effecting, there's no return value
- receives `NetworkEvents`
- passes them to the gossipsub handler above, which returns a resolved tipset
- if it's heavier than the one it has in the `ChainStore`, send it to the `Tipset`Processor
- runs the `ChainMuxer`
- runs the `Libp2pService
- runs the JSON-RPC service:
- uses the `StateManager`
- uses the `KeyStore`
- uses the `MessagePool`
- uses the `SyncState` of the `ChainMuxer`
- uses the `BadBlockCache` of the `ChainMuxer`
- uses the `Beacon` of the `StateManager`
- uses the `ChainStore`
- uses the `Tipset` sender of the ChainMux for new mined blocks:
- `GossipBlock` can be submitted via JSON-RPC
- it resolves the included CIDs from the Blockstore (have to be already known),
- uses a `Tipset`Validator to check that the message root CID is correct
- reinserts(?) messages into the `Blockstore`
- sends the block to the `ChainMux`
- sends the block to the network via pubsub
- waits for Ctrl+C

## Observations

* Forest doesn't seem to have mining functionality yet, ie. it never proposes a block, except ones received via JSON-RPC.
* It's main functionality is to follow consensus, and to keep publishing transaction messages to the network until they are finally included in a block.
* It follows the structure of the Go implementation.
* It's a bit loose sometimes in what kind of data it admits, e.g. invalid CIDs received through Bitswap are stored and emitted.
* It seems to be okay with blocking event processing to resolve CIDs from the network.
* It makes lots of mutable fields public, which makes it difficult to reason about the invariants of a component, since at least in theory anyone can modify their internal data from anywhere.

## Further Steps

To implement HC, we seem to have a blank slate, because Forest doesn't have consensus yet, it's what Tendermint would call a Full Node, but not a Validator Node.

We need to add a `Consensus` interface that can:
* Mine blocks, either in a loop, or by scheduling callbacks from an external scheduler. For this it needs to access the Mempool.
* Publish blocks to the network.
* Validate blocks from the network to check that they conform to the consensus we want to achieve.

By validation rejecting any block that is not according to our consensus, we make sure that honest nodes never follow a fork in the Filecoin chain which aren't legal.

To integrate `Consensus`, we need the following touch points:
* the validation will be called from `tipset_syncer.rs`, in particular the ``Tipset`Processor` and probably ``Tipset`RangeSyncer`

To handle subnets:
* we need ``Libp2pService` to subscribe to cross-message topics
* we need to extend ``PubsubMessage`` with new types for cross messages
* we need the `ChainMuxer` to handle the new ``PubsubMessage`` types
* we need a new resolution mechanism to ask the CID from the subnets.
* the cross message pool
* the ability to spawn new processes