# Tendermint Core as a Consensus Protocol in Eudico

## Introduction
At the present time, Eudico doesn't use a BFT-type consensus.

This document provides a design of applying [Tendermint Core](https://docs.tendermint.com/master/) in Filecoin and the description of its experimental implementation in Eudico client.

Tendermint Core is used in the service-based model, rather than in the traditional [ABCI-based model](https://docs.tendermint.com/master/app-dev/app-architecture.html).

The main goal of the document is to provide a full understanding of the proposed design, Tendermint Core interactions, and how the entire system works.

### Goals
1. Explore and test the expressiveness and power of Eudico’s design and interfaces (e.g., Consensus interface) by adding support for the [Tendermint consensus protocol](https://tendermint.com/static/docs/tendermint.pdf).
2. Improve Eudico characteristics by adding a BFT-type consensus protocol with better characteristics and justified properties in addition to already i plemented and used consensus protocols (e.g., PoW).
3. Research Eudico's “consensus upgradeability” and improve it if necessary.
4. Explore Filecoin design and Eudico implementation pitfalls for future integration with MirBFT protocols.

### Background
Design models that can be used to implement a BFT-type consensus protocol are arranged from the highest level of abstraction to the lowest one:
1. Service-based or “Consensus as a Service” (e.g., applying Tendermint Core in this work).
2. Application-based: traditional use of Tendermint Core via its ABCI interface (e.g., [Ethermin-ABCI](https://github.com/cosmos/ethermint_abci)).
3. Framework-based (e.g., [MirBFT](https://github.com/hyperledger-labs/mirbft)).
4. Library-based: Adoptation a consensus protocol implementation (e.g., [Quorum](https://github.com/ConsenSys/quorum/tree/master/consensus/istanbul), or the original [Istanbul BFT](https://github.com/ethereum/EIPs/issues/650).
5. Add a consensus algorithm from scratch: implement all parts of the mechanism including a consensus algorithm.

In application-based approach we need to implement an application interface only.

In framework-based approach we deal with internal communications between software packages instead of network communications between external services as in the service-based implementations.
For example, MirBFT logically follows the same approach as Tendermint. They both are application-centric. In both cases, we need to implement an interface for the replicated application and we don't have access to consensus' internals or even control under consenus execution. See a MirBFT [example](https://github.com/hyperledger-labs/mirbft/blob/main/samples/chat-demo/app.go#L49). Here we also need to implement a transport layer somehow.

In library-based approach we deal with adaptation: we need to implement transport and application interfaces, but we don’t need to implement a consensus algorithm. Examples: [Istanbul BFT](https://github.com/ethereum/EIPs/issues/650), [etcd-raft](https://github.com/etcd-io/etcd/tree/main/contrib/raftexample) application.

## System Overview

### Assumptions
1. Permissioned setting: The set of Tendermint validators is [permissioned](https://docs.tendermint.com/master/nodes/validators.html).
2. 1:1 relationship between Eudico nodes and Tendermint nodes.
4. Sidecar-based architecture: Eudico and Tendermint nodes are co-located.
5. Each Eudico node trusts the corresponding Tendermint node.
6. A Eudico node and a Tendermint node use the same `secp256k1` key.

### Applying Tendermint
The [standard approach](https://docs.tendermint.com/master/tutorials/go.html) of using Tendermint framework to provide consensus in Filecoin is to implement a special interface between Tendermint Core and Eudico client. This interface called [Application BlockChain Interface (ABCI)](https://docs.tendermint.com/master/spec/abci/) and it allows you to replicate your application (state machine) with the Tendermint consensus protocol. In this case, Tendermint nodes would use Eudico client as a backend and call its functions to validate and apply transactions through ABCI methods (e.g.,`CheckTx`, `DeliveryTx`). [`Ethermint-ABCI`](https://github.com/cosmos/ethermint_abci/blob/c0f67298e7a01b3d91a09e8804e7943790b4b66e/ethereum/eth_state.go) and [CAS-Demo](https://github.com/6thc/tendermint-cas-demo) are examples of how Tendermint Core is typically used.
Using of this approach within Eudico, probably, would require a lot of refactoring and the ability to perform transactions sequentially.

Instead of that approach, we suggest using Tendermint Core as a message system providing BFT total order broadcast of input messages without any modifications of the Filecoin state.

In this design, each Eudico node interacts with a trusted Tendermint node located remotely or on the same host (as a sidecar) using its [RPC](https://docs.tendermint.com/master/rpc/). 1:1 relationship between a (Eudico node, subnet) pair and a Tendermint node is assumed.
A Eudico node sends input messages to the corresponding Tendermint node. Tendermint validates them, and ensures that the messages are recorded on every Tendermint node in the same order in blocks (batches of the messages).
Then the Eudico node retrieves Tendermint blocks via Tendermint RPC, performs [static and semantic validation](https://spec.filecoin.io/#section-systems.filecoin_blockchain.struct.block.block-syntax-validation), removes corrupted messages, generates a Filecoin block with the verified messages, resolves and translates the miner address, signs the block, sends the block, and then, finally applies the messages directly.

The figure below depicts a high-level arhitecture of the system.


The following sections consider in detail the main mechanisms introduced to employ Tendermint consensus.

The figure below depicts sending messages through Tendermint consenus engine.


### Message Pool
When sending input messages from Eudico's Mempool to Tendermint we shouldn't send the same messages multiple times. The reasons for that are as follows:
1. [Bug](https://github.com/tendermint/tendermint/issues/7185) in Tendermint that can cause `broadcast_tx_sync` to hang.
2. The throughput of pulling from Mempool is higher than the throughput of Tendermint consensus requested via network, so we don't need to send the same messages multiple times to decrease network load, Tendermint node load, etc.
3. That doesn't make any sense because Tendermint mempool filters the same messages.
4. Tendermint client doesn't guarantee that a message sent to Tendermint RPC will be received. Especially, if a Eudico node and Tendermint node are not co-located.

At the same time, we must be sure that the sent messages will be included in a Tendermint block.

So, we should send each message from mempool to Tendermint once and resend it only if the message has not been added into a Filecoin block after a few epochs and do this until the message is added into a block.

Message pool is a mechanism that addresses those aspects.

### Message Validation
If an attacker pushes into a mempool an invalid message (e.g., a message with an incorrect or wrong signature), a Tendermint node will be able to commit the next block containing that incorrect message and submit it to the other nodes. The nodes will detect that the message is incorrect using validation functions (e.g.,[`CheckMsgs`](https://github.com/filecoin-project/eudico/blob/4a3ccfed70a6a920a10fe43ca57dcf7f4785b31c/chain/consensus/common/cns_validations.go#L67)) and discard the block. But this could lead to potential problems with liveness because Tendermint has commited the block and the nodes will get the same block from Tendermint blockchain with invalid transactions and try to apply it again and again. So, all Eudico nodes must reach a consensus on which the messages in the block are invalid and filter them out.

Since Tendermint nodes, by design, don't perform message validation Tendermint blocks may contain corrupted (malicious, invalid) messages. At the same time, Eudico mempool performs all necessary checks for input validation. So, we can suppose that corrupted messages can be injected into Tendermint blocks only by malicious clients or malicious Tendermint nodes.

Those statements lead us to the following:
1. We must implement message filtering for all Tendermint blocks.
2. We don't need to remove found corrupted messages from our local mempool and message pool.
3. If the Tendermint block contains corrupted messages, we just need to filter them out and then apply the block.

So, the mempool validation mechanism should guarantee that a honest clients send only valid messages to its Tendermint node, and the filtering mechanism should guarantee that the honest Eudico client extracts only valid messages after they have been totally ordered by Tendermint.

Note, in this design, a Tendermint block may contain messages that are invalid in the Filecoin domain and must be filtered out by all honest nodes to guarantee safety and liveness.

An honest node must employ the following rules to create a valid Filecoin block:
1. `BLSAggregate` must be computed after validation and filtering of all messages.
2. `Timestamp` must be set to the Tendermint block `Time` field value.
3. `Miner` must be set to the address of the Tendermint block proposer in Filecoin address format.
4. `BlockSig` must be set to `nil`.

### Filecoin Block Syncing

Syncing is performed through Eudico P2P reusing the [BlockSync](https://spec.filecoin.io/#section-algorithms.block_sync) protocol which synchronises specific parts of the chain, that is from and to a specific height.

That said, a new node joining the Eudico subnet syncs the state over Eudico BlockSync protocol.

### Blockchains Binding

A block signature calculated on the Eudico client’s account private key cannot be used since in this case each Eudico node would mine and submit the same Filecoin block, but with different signatures.

It is suggested to insert Tendermint block's hash value into Filecoin block to cryptographycally bind Tendrmint and Filecoin blocks:
`filecoin.BlockHeader.Ticket = tendermint.Block.Header.Hash())`

To check the ticket of a block a Eudico node should request the corresponding block via Tendermint RPC.

Eudico node should also check that all messages from a Filecoin block are contained in the corresponding Tendermint block.

Messages and hashes of all synced blocks must be validated using Tendermint RPC. Blocks received from the Tendermint node may not be validated.

### Block Mining

Let `tb` be a Tendermint block proposed by a Tendermint node `t_node_i` and `fb` be a Filecoinb block computed based on the block `tb`. Then the miner of the block `fb` is a Filecoin node corresponding to `t_node_i` (connected to this node).

Miner address resolution is the process of translating Tendermint address to Filecoin address. It is not straightforward because Filecoin and Tendermint address formats and public key formats are different and use hash function for address derivation.

If a node is going to submit the block proposed by the corresponding Tendermint node then the node just sets its own Filecoin address to `Miner` field. If a node submits the block proposed by another Tendermint node the the node does the following:
1. Submit request to [Validators](https://docs.tendermint.com/master/rpc/#/Info/validators) method and retrieves the proposer publick key `pubKeyComp` by the known proposer address, contained in the Tendermint block.
2. Convert `pubKeyComp` in compressed format to `pubKey` in uncompressed format.
3. Compute `addr = Filecoin-address(pubKey)`.
4. Sets `Miner` field to `addr`.

## Message Flow

Eudico node:
Input: messages from mempool, the current epoch.

1. A Eudico node gets the head of the chain and is going to mine the next block at `h`.
2. All messages from mempool are fetched into `msgs`.
3. Each message from `msgs` that has not been sent to Tendermint yet or was sent `finalityWait` epoch ago is submitted to the Tendermint RPC using `BroadcastTxSync` method.
4. `BroadcastTxSync` broadcasts transaction bytes to a Tendermint node synchronously and returns a response after `CheckTx` execution.
5. If the message is accepted by Tendermint then the hash of the message and the current epoch is added to the Message pool.
6. Requests the Tendermint block at height `h`.
7. Creates a block template with all messages from the Tendermint block.
8. Calls `PrepareForSignature` that filters all cryptographically invalid messages and validate the block: The function checks all BLS and secp256k1 signatures for all messages sequentially, aggregates multiple BLS signatures into one `BLSAggregate` signature.
9. Calls `FilterBlockMessages` function that filter all messages that are not semantically correct.
10. Sets the `Ticket` to the hash the Tendermint block, sets `BlockSig` to nil.
11. Submits the block over P2P.

The result is a Filecoin block containing valid messages, iff the corresponding Tendermint block contains valid messages.

Tendermint node:
1. The Tendermint node receives a message from the Eudico node and calls `CheckTx`.
2. `CheckTx` decodes the input raw bytes and validates that the decoded message matches to`SignedMessage` format.
3. If there are no input messages from Eudico nodes then empty blocks will be committed.
4. Tendermint Core proposes, confirms, and commits the next block containing the ordered messages from Eudico nodes.

### Basic flow
Example of the flow with 4 miners all running Eudico and Tendermint node each, submitting 1 transaction each (1 per miner) concurrently and showing what happens in the system.

TBD

## Agenda

| Item | Description |
| -------- | -------- |
| ❗🔴 | Something the team must be aware of|
|🔥NEW🔥 | New content |
| ❓️🔴 | Question or decision we need to discuss|

## References
1. [Byzantine Fault Tolerance: Tendermint.](https://docs.google.com/presentation/d/1k0CbJQBHoMJkuOXdAKHFw9dcaXES_3dFYnCp1I1bm2M/edit)