IPC Stack Implementation Analysis
===
###### tags: `HC`, `B`, `B3`
_@adlrocha - August 2022_

The goal of IPC is to implement a framework to horizontally scale IPLD-based blockchain networks able to reach the performance of current centralized systems (Gtps throughputs and fast finality). The two things to prioritize as we chose the stack for the final implementation are: __performance__, for obvious reasons, and __modularity__, so users are able to fine-tune the system to their needs as they scale.

This document explores the different alternatives available for the implementation of a product based on the concept of IPC.

## Revisions
- `2022-08-26`: Advantages and disadvantages of the different design approaches.
- `2022-09-30`: Updated stripped Eudico design.
- `2022-09-01`: Added [new section with modular architecture design](#-New-modular-blockchain-from-existing-modules).

## :tulip: Stripped Eudico
> _Time to ship:_ :+1:
> _Battle-tested:_ :+1:
> _Modularity:_ :hand_with_index_and_middle_fingers_crossed:
> _Long-term aligned:_ :hand_with_index_and_middle_fingers_crossed:
> _Probability of reaching performance target:_ :hand_with_index_and_middle_fingers_crossed:

The first alternative is to use Eudico (fork of Lotus) as a base for the implementation of the production-ready IPC implementation.

The approach would be the following:
- Strip-out all the Filecoin-specific modules from our current implementation of Eudico (i.e. sector storage, sealing, etc.).
- Modularize all common stack modules to enable developers with the flexibility of implementing their own custom modules, and deploy custom subnets with their preferred components (the libp2p for blockchains).
- Clean and productionize all the MVP code for IPC.
- Enable a way to run peers that sync with different subnets in different processes and peers (instead of go routines) to allow operators to choose their desired deployments strategy (the subnet manage abstraction may need to coordinate peers deployed in several hosts).

Eventually, this stripped Eudico would become the basic blockchain core used to implement other blockchain networks and subnets. For instance, Filecoin could choose to deploy itself over Eudico by attaching all the Filecoin-specific modules into Eudico. This would give them modularity and the ability to leverage all the functionailities of IPC by-design.

This codebase would depart from Lotus and would become less a fork and more its own project.

### Advantages
- We already have an MVP implementation of IPC.
- We can benefit from all the work and innovations done by the Lotus team (and we could eventually cherry-pick from their new features even if we become our own project, as the codebase would be "close-enough").
- The stack is "battle-tested". It's been deployed in production without any major outages for almost two years.
- We could ship faster in the short-term (we know the codebase and we have an MVP of IPC).
- Mir is written in Go and is already integrated with Eudico.
- It already ships with a WASM runtime, and the FVM.

### Drawbacks
- While we may be more productive in the short-term, we may have additional overheads as we fulfill our long-term vision of a modularized blockchain stack based on IPVM (a generic WASM runtime).
- As we inherit a battle-tested codebase we may also be inheriting some technical debt.
- Go is more prone to "nasty-bugs" (we have already experienced hard-to-debug data races, and other issues that have blocked us for several days).
- We may start facing the performance bottlenecks that other teams re-writing their stack to Rust faced (although honestly I don't it'll be the case).

### Design
The design of a _"stripped Eudico"_ will be comprised by two parts:
- _Removing Filecoin-specific packages_, keeping only the basic modules required for the core operation of the blockchain.
- `build/proof/params` can be removed.
- `node/builder_miner` can be removed.
- `itests` can be cleaned to include the integration tests focused on the core blockchain operation.
- `lotuspond` can be removed.
- `paychmgr` can be removed. The implementation of payment channels will be application-specific. In the future, we may introduce a standard implementation for IPC, but in the meantime we can bypass this.
- `miner` can be removed, as the miner of each consensus algorithm in IPC will be provided as part of the consensus implementation.
- `markets` can be removed as it implements the logic for the storage and retrieval markets.
- `cli/{paych, client_retr, filplus, disputer, etc.}`: removing cli commands that are not linked to the core operation of the blockchain.
- `chain/builtin/policy` can be removed.
- `market`, `paych`, `power`, `verifreg` and `miner` actors won't be used in the core implementation (they can be included afterwards if needed).
- `chain/market` will no longer be needed.
- `gen/slashfiler` can be removed.

- _Abstracting basic modules of the stack_. A few of the modules that we should aim to abstract to enable custom implementation of them for different consensus algorithsm are:
- `chain/consensus`: Already decoupled in Eudico, but we can probably come up with a better interface that fits more consensus algorithms than just longest-chain protocols. A better consensus interface would prevent from requiring a lot of ad-hoc code for BFT-based consensus (as has been the case for Mir).
- `chain/messagepool` so we can support the implementation of different message pools coupled with the consensus protocols.
- `chain/sync` to implement the syncing logic according to the consensus algorithm (or this should be common for all consensus protocols?).
- `chain/exchange` to determine the different ways that nodes can use to fetch data from other peers.
- `message` interface supported by the `vm` so different consensus algorithms are allowed to use different message formats according to their needs.

The goal up till here would be to have a system with all the core modules required to deploy a blockchain network with support for IPC and the ability to configure many of its pieces. Eventually Filecion should be "implementable" over this system by just coupling the speific Filecoin modules.

With this, we'll be ready to focus on the implementation of all IPC-specific processes. The key processes to improve from the MVP implementation are:
- The subnet manager: For simplicity, we use a monolithic architecture for the whole IPC implementation. When we spawn or sync with a new subnet, we start new routines for the processes required by that subnet. This architecture is prone to errors in a production environments. We should abstract the subnet manager such that subnet-specific processes may be hosted in different hosted and handled by different daemons but can be handled by a single subnet manager (similar to how mining or sealing is handled in Lotus).
- The content resolution protocol simply works, but being a key component for the operation of IPC we should test it exhaustively and identify potential attack-vectors.
- The current implementation of the checkpoint protocol may be a bit "racy" and it sometimes gets out-of-sync without a clear reasons. We need to exhaustively test the protocol and fix the bugs. We should also consider improving it to use other verification schemes like the use of threshold signatures.
- UX of the CLI for subnets can be improved.

### Milestones
> WIP

For now, we are focusing on [this](https://hackmd.io/XgBUBIKNSS21RYglDID1MA).

### References
- [Eudico](https://github.com/filecoin-project/eudico/)

## :evergreen_tree: IPC Forest
> _Time to ship:_ :-1:
> _Battle-tested:_ :-1:
> _Modularity:_ :+1:
> _Long-term aligned:_ :+1:
> _Probability of reaching performance target:_ :hand_with_index_and_middle_fingers_crossed:

This approach is similar to the one for Eudico, but using Forest as a base. Forest already misses many of the storage-specific features of Filecoin, and it only implements the basic modules required to sync as a full-node with the Filecoin network (i.e. the basic blockchain stack).

In this case, there is a bit more of preliminary work to be done. We'd have to:
- Decouple the consensus interface of Forest so it can be implemented by different consensus implementations (this has already been integrated in Forest by @aakoshh).
- Have (at least one) fast consensus implementation for subnets (@aakoshh already working in Narwhal).
- Implement all the IPC-specific processes (deploying new subnet stacks, listening to events of other subnets, checkpointing process, etc.).
- CLI utilities to interact with the different subnets.

In the process, we will try to modularize everything in the same way we are targeting for Eudico but, ideally, with the ability to compile all of this modules to WASM. As it would happen with Eudico, we would build a base that other projects are able to use for their own implementations. In this case we could choose to: (i) depart from Forest and build our own project; (ii) or talk to the Forest team to transform Forest into the core IPC client (the only thing that we would need to respect is the ability to fully sync with Filecoin mainnet -- and it's testnets -- ).

### Advantages
- Rust is a better language for modularity, and has better support for WASM (eventually, we'll be able to compile the different modules of the system to WASM.).
- Smaller codebase and more comprehensible code structure.
- We can benefit from many of the innovations being done in Rust in the space (e.g. Narwhal, the Move programming language, WASM runtimes, etc.).
- For the reasons mentioned above, it is more aligned with our vision of a modularized blockchain over IPVM/WASM.
- We have the opportunity to re-write IPC from scratch with all that we've learned through the MVP, removing any technical debt we may have in our first implementation.
- We may be able to achieve better performance than with the Go implementation.
- It already ships with a WASM runtime and the FVM.

### Drawbacks
- We would have to implement IPC from scratch. We would be less productive in the short-term (and it would take, a-priori, more time until we can ship something).
- The Forest stack hasn't been battle-tested in production as much a Lotus (it may not even be feature-complete in some cases). Also, there is not a strong/dedicated team behind it as is the case in Lotus.

### Design
Forest implementation is way more modular than Eudico. We may have more flexibility extracting a modular implementation of a blockchain node from Forest than from Eudico. However, if we choose to start our IPC implementation from Forest we will have to implement all IPC processes from scratch. If we choose to follow this direction we'll need to implement the following features:

1. Join and spawn a subnet. Validate messages in parallel from the two subnets.
- Spawn the stack and processes to sync with a new subnet.
- Run two parallel chains running different consensus algorithms.
3. End-to-end lifecycle of a subnet (joining, sending messages within the subnet, leaving, syncing).
- Implement the basic commands required to interact with the node.
- Handle the processes to join, leave, etc.
4. Cross-net messages between different subnets.
- Checkpointing protocol
- Execution of cross-net messages.
6. A high-performance subnet consensus protocol.
- Narwhal?
8. Support for the atomic execution protocol.
- Commands and off-chain execution of the protocol.

> Excerpt from a conversation with Stebalien about Forest regarding his opinion about using Forest as a baseline for IPC
>
> _"I only know about the forest actors, we didn't use much else.
In terms of sync, in know that lotus is careful to not add blocks to the data store until they've been validated. We often buffer things internally while validating (and that should generally be safe...
In terms of the team..., their goal is to stay up to date with filecoin mainnet. Beyond that, you'll probably be on your own."_

### Milestones
> WIP

### Resources
- [Forest](https://github.com/ChainSafe/forest)
- [Forest School](https://hackmd.io/psTcu8tyTQqBwkJsqH8p1Q) by @aakoshh
- [Implementation notes of IPC](https://hackmd.io/@consensuslab/rkhV38g9q?type=view).

## :hotsprings: IPC over Substrate
> Probably not an option

Implementing a blockchain is hard. We could have written a new IPLD-based blockchain for IPC from scratch, but that would have taken us a lot of time. Forking from Eudico and Forest gives us a base with all the low-level modules to implement a blockchain. With this in mind, why not considering Substrate that already gives us a modular framework to implement our IPC blockchain?

After going through Substrate's architecture and documentation, there are several reasons why we think that Substrate may not be an option for the implementation of IPC:
- Substrate's storage is abstracted as a k-v store over a Patricia trie. Our networks have to be IPLD-based, and we will need to do some work to make Substrate "IPLD-powered".
- It ships with its own WASM runtime used to execute the blockchain state transitions as business logic. In our case, we want to use FVM as the main runtime for our networks, this means that we would have to modify Substrate to integrate the FVM as a runtime. Also, FVM is designed to persist its state over an IPLD blockstore (so it may not be straightforward to integrate directly). IPVM will also target an IPLD data store, so even if we shipped with IPVM there would be some adaptation and integration work to be done.

That being said, Substrate has already tackled the problem of "modularizing a blockchain" and we can get a lot of ideas from them as we build or solution (even if we don't base it on them), such as:
- Abstract the implementation of [consensus algorithms](https://docs.substrate.io/fundamentals/consensus/)
- [Architecture](https://docs.substrate.io/fundamentals/architecture/) based on an outter-node (handling networking and consensus), and an inner-node (for business logic, state transition, and upgrades).
- [Runtime architecture](https://docs.substrate.io/fundamentals/runtime-development/)
- [On-chain runtime upgrades](https://docs.substrate.io/build/upgrade/)

In conclsion, and according to the documentation, substrate is for _"blockchains [...] tailored to a very specific use case"_ which is not our case.

### Advantages
- Battle-tested and well-documented.
- It would allow us to build IPC from scratch over an already modular blockchain framework.
- Ships with a WASM runtime.

### Drawbacks
- Not IPLD-friendly.
- Oriented to implementing use-case specific blockchains.

## :rocket: New modular blockchain from existing modules
> _Time to ship:_ :-1:
> _Battle-tested:_ :-1:
> _Modularity:_ :+1:
> _Long-term aligned:_ :+1:
> _Probability of reaching performance target:_ :hand_with_index_and_middle_fingers_crossed:

The reason for leveraging an existing codebase to implement IPC is because _"writing a blockchain from scratch is hard"_. But what if we could leverage and adapt existing modules from existing codebase and combine them to write a core implementation of a blockchain that fits our goals? Being our goals _"the implementation of the IPC framework on top of a lightweight IPLD-based blockchain to achieve horizontal scaling and high performance."_

We have seen while tackling the integration of high-performant BFT-based consensus algorithms into the existing codebase (mainly Lotus and Forest), that we find ourselves making unnecessary trade-offs to fit the architecture of these codebase (initially written for a longest-chain protocol). By writing our own implementation based on existing modules we have the opportunity to build a new architecture able to accommodate any kind of consensus algorithm without sacrificing performance (or requiring ad-hoc glue code for the integration). Having our own implementation will also help us reason about what we are doing, and make more informed design decisions (something that is not always the case for Lotus and Forest).

### Design
#### Subnet Chain Stack
We propose the following architecture for each IPC chain:
- The key component of the network is the `syncer`. The `syncer` interacts with the rest of modules of the system to ensure that the peer has an up-to-date view of the chain. It can be implemented as an event-loop that listens to events from the `p2p` network and the `consensus` to collect information about new blocks.
The syncer handles all the synchronization logic for the node, from reverting potential forks, to filling gaps in the chain, or syncing the chain from scratch. Each consensus implementation may provide the syncer with different methods to tackle this.
When the `syncer` finds a new block, it notifies with a `HeadChange` event to the `mempool`and `StateManager` so they can update their state accordingly.
- The `syncer` is notified about new blocks with a `NewBlock` event from the `consensus` or the `p2p` network. Either the `consensus` finds new block itself (by mining a block, or agreeing through a BFT protocol), or other peers broadcasted the new block to the network.
- The `StateManager` is responsible for handling the state of the chain. It stores the chain of blocks and messages, and keeps and up-to-date view of the state. This modules receives new `HeadChange` events from the `syncer` with new blocks, and it apply the blocks and trigger the state changes in the `ChainStore` (backed by an IPLD datastore). The execution runtime used by the `StateManager` is the FVM.
- The `mempool` is responsible for keeping an up-to-date view of unverified messages in a peer. It listents to `NewMessage` events from the network to collect new unverified messages, and `HeadChanges` from the syncer, so it can garbage collect from its store messages that have already been verified. Implementations of the `mempool` should expose two different interfaces so it can be equally leveraged by any implementaiton of a consensus algorithm: a pull interface (where the consensus algorithm is responsible for polling new unverified messages); and a push interface, where the mempool streams batches of new unverified messages to the `consensus` module as they come.
- The `consensus` module is responsible for the ordering of messages and potentially assembling them in blocks or batches. It can interact directly with the network and is responsible for notifying the `syncer` when new blocks have been agreed upon. It also listens to `HeadChange` events from the `syncer` so its aware of the progress the chain is making off-band (if needed by the consensus implementation).
- Finally, for the block and message format, the proposal is to keep the `Message` structure currently used in Filecoin (which is general enough to accommodate any case of use case, and can be used executed in the FVM runtime without major changes), but to build a new `Block` trait/interface that enables all modules to use different block formats seamlessly, so different consensus protocols are able to implement the block format they need to batch messages.

> Basic blockchain modules
>

We think that this architecture is general enough to accommodate any kind of consensus implementation, from BFT-like to longest-chain protocols (which wasn´t the case with the current codebases). _For the sake of brevity, we leave as an exercise to the reader to map `<consensus algorithm of choice>` into the current architecture to find flaws or potential improvements to the design_.

#### IPC architecture
Our implementation of IPC, would start a new subnet chain stack for each chain as shown in the next figure (example with two subnets). Along with the chain stack, three new modules are spawned for their operation and compatibility with IPC:
- The `cross-net mempool` which is listening to `HeadChange` events in its own chain and that of the parents to collect unverified cross-net messages that need to be fed to its `mempool` for validation.
- The `checkpoint protocol` that listens to new `HeadChange` events of its own syncer to determine if a new checkpoint needs to be created, singed and propagated to the parent.
- The `content resolver` that upon request, resolves content stored in other subnets.

> IPC architecture
>

With that, we have all of the modules required for a full IPC implementation (ideally with the ability to configure it and extend it as needed).

### Milestones
> WIP

### Advantages
- We remove any technical debt and baggage imposed by existing codebases. Instead of making Mir or Narwhal work with Filecoin as a "workaraound", we can build an architecture that seamlessly accommodate any kind of consensus algorithm for high performance.
- Having lightweight and well-defined architecture for the network can help us reach better performance.
- Easier to reason about and to make informed design decisions.

### Drawbacks
- Not a battle-tested implementation.
- It may require more time to be implemented.
- We may realize that is not simple and we end up with a worse approach that those of the existing codebases.
- The codes needs to be audited.
- Thorough testing.