# WIP: 0L Incident Report Dec 16-18

INCIDENT NOT FULLY RESOLVED 7pm ET Dec 18

# TL;DR

0L Network temporarily halted due to DOS seemingly caused by end-user transactions with incorrect sequence numbers. The hypothesis is that some amplification of errors caused Validator nodes to rate-limit connections to each other, and stop propagating votes.

# Definitions
Diem-node: the validator and fullnode software. 0L Network runs on a fork of Diem core 1.3.1.

Epoch Boundary: 0L changes epochs every 24h at approx 10:30 AM ET.

Carpe: a desktop miner and wallet app for 0L. It sends transactions via 4 public fullnodes designated for those users.

Hot Upgrade: An upgrade procedure for 0L which issues a writeset at a coordinated time (round 2) of an epoch.

### Dec 16

An automated "Hot Upgrade" took place at 10:35 ET. There were some issues with this upgrade which affected individual validators (10/51), but the state on chain was correct and the upgrade succeeded at implementing bytcode and state migration.

### Dec 17

By morning it appeared that other nodes experienced unrelated instability, and the network had a liveness failure (f+1), making blocks slowly.

Typically at any time we will see many DiemAccount::1003, sequence number errors in validator logs. However a waterfall of these logs began to appear. And then network connections began to fail between peers.

Blocks stopped being produced, and the occasional log would show the same block being voted. CPUs on validator nodes were reported to being used at 100%.

After a number of validators async and uncoordinatedly restarted diem-node, eventually got network moving again, with very flow blocks and eventually crossed the 24h epoch boundary. Then a number of nodes dropped off in the new epoch (now 26 validators)

Initially block production was fast, but then again a waterfall of DiemAccount::1003 erorrs appeared, and the network slowly went to a halt.

It required a coordinated stop by all 26 validators, plust shutting down fullnodes serving Carpe app, to regain consensus and block production.

### Dec 18
After a day of normal block production with clean logs, we conducted a test with the Fullnode devices that were serving Carpe users (which had been off since the coordinated stop).

A new epoch begins with 46 validators.

Attempting to open up for transactions Saturday 4:30 PM ET

- Got all the network running smoothly, all validators online

- Got fullnodes up to sync with network, firewall blocking 8080 connections

- Open port 8080 on 1 fullnode: (ODL), no issues for 20 minutes.

- Tested an account creation transaction successfully. No error logs.

- Unclear if any Carpe users were connected to this node. Unlikely.

- Start second fullnode (MB) and within 2 minutes the network halts

- Roughly 10 different IP addresses (Carpe users) connect to this fullnode

- Start seeing some error logs on Validator nodes: DiemAccount::1002 sequence number too old. 10-20 of these errors.

- A minute later see a deluge of DiemAccount::1003 sequence number too new
- Block production slows down.
- Start seeing `Outbound connection failed for peer`
- No blocks produced, just network connection attempts to all peers.

- Shut down the two Fullnodes which had been online.

- The network had halted at this point.
- Approx 22/46 of validators were available to synchornoussly shut down. After a 5 minute shutdown, the nodes were brought back online, and block production resumed, and the 1003 errors were no longer present (the fullnodes remained off).

The fullnodes serving Carpe users remain off.

# Hypothesis
A validator node that is not synced will start displaying 1003 errors from incoming end-user transactions. The node can get overwhelmed and CPU starts thrashing. Mempool resubmission makes this worse. Eventual the Validators deny service to other Validators, and blocks stop being produced. We think this is the case because of the "Test starting an unsynced validator" below.

Besides this, there seems to be caching of user transactions that is happening external to mempool, possibly client software. Additionally if an end user is using a Json-rpc endpoint whose sync is off, they may be sending txs with invalid sequence numbers. We think this is the case because of the "Test opening 8080 of a synced Fullnoder" below.

What's unclear is why any amount of tx prologue rejections would cause the Validator nodes from not being able to reach one another. There's some thermal runaway effect where errors are multiplying. Another possibility is that diem-nodes become faulty and attempt to reconnect to each other maxing-out the rate-limit which is intended to prevent DDOS, but effectively contributing to the DOS.

## Observations

The 1003 appear because a tx uses a sequence number more advanced than one validator knows about, it then is getting stuck in mempool of a validator. What we also notice is that the same validator is rebroadcasting that stuck tx to other validators repeatedly, and those will see 1006 errors.

### The same TXS in mempool seem to be repeatedly attempted


### Bandwidth Usage


### Carpe fullnodes are the culprit


## Test starting an unsynced validator

Users are sending transactions to network, and there are many validators already in sync and processing transasasa
0. Start syncing in validator mode
1. Start seeing prologue errors
2. Hangs with CPU at 100%
3. More prologue errors
4. Hangs on kill

## Test opening 8080 of a synced Fullnode

Users are sending transactions to network, and there are many validators already in sync and processing transactions.

0. Fullnode is synced, but JSON_RPC on 8080 is closed
1. Open port 8080
2. Validator node (that are in sync) begin to see 1003 errors
3. block production slows
4. Cascade of prologue errors
5. Hangs with CPU at 100%

## Contributing factors

- State sync is slow. Fullnodes seem to fall out of sync and have a difficult time catching up

- Validators which need to recover have a long time syncing, can't advance to the current network state, and must use a snapshot restore point.

- Validators on 0L are running as public fullnodes (without the VFN intermediary layer). This is possibly causing self-dial and thermal runaway.

# Error code Reference

## DiemAccount::1002

PROLOGUE_ESEQUENCE_NUMBER_TOO_OLD

The user account's transaction sequence number submitted is too OLD (low number) compared to what is on the chain for that user.

Move:

https://github.com/0o-de-lally/libra/blob/d3c474a8752cb869a57fb7c7f98422d290a3a56c/language/diem-framework/modules/DiemAccount.move#L2679-L2682

Rust:
https://github.com/OLSF/libra/blob/9b9e5eb0ef1f57ce8470995c037ab4574a909dfb/language/diem-vm/src/errors.rs#L68

## DiemAccount::1003

PROLOGUE_ESEQUENCE_NUMBER_TOO_NEW

The user account's transaction sequence number submitted is too NEW (high number) compared to what is on the chain for that user.

Move:

https://github.com/0o-de-lally/libra/blob/d3c474a8752cb869a57fb7c7f98422d290a3a56c/language/diem-framework/modules/DiemAccount.move#L2790-L2799

Rust:

https://github.com/OLSF/libra/blob/9b9e5eb0ef1f57ce8470995c037ab4574a909dfb/language/diem-vm/src/errors.rs#L17

## DiemAccount::1006

PROLOGUE_ETRANSACTION_EXPIRED

transaction expiration time exceeds block time.

Move:

https://github.com/OLSF/libra/blob/d3c474a8752cb869a57fb7c7f98422d290a3a56c/language/diem-framework/modules/DiemAccount.move#L217

Rust:
https://github.com/OLSF/libra/blob/9b9e5eb0ef1f57ce8470995c037ab4574a909dfb/language/diem-vm/src/errors.rs#L20

# Post Mortem

The network could get into a split-brain mode easily because of certain validator.node.yaml configurations, that state was near impossible to recover from giving compounding effects of sync delays, and incorrect networking parameters.

## Root Causes
A combination of factors contributed to the outage, including runaway CPU usage and networking failures:

1. State sync is slow in Diem, with no mechanism to skip to an advanced waypoint. A validator can spend all day behind sync. The further back a validator node gets the more susceptible it is to synchronization bugs, as seen below.
1. The Diem mempool management for all node types does not wait for the node to be in sync. So mempool related work continues to occur while the node is behind sso making it harder to catch up. Admitting txs to the mempool is expensive work, especially if it includes Tower proof verification.
1. 0L's network topology contributed to the errors since serving on the fullnodes and validator networks on the same instance can create a panoply of issues in failing networks. See writeup below.
1. Parameters: The mempool was accepting up to 100 transactions per user concurrently, there's no reasonable use case for that with 0L. Possibly backlogged Miner tranasactions were staying in mempool for extended periods.
1. Parameters: Max failed ping requests was set very low. When nodes became overwhelmed they would quickly shut down new connections from a requesting node. This meant that fewer nodes could connect to each other, and pockets of nodes (split brains) would be proposing different blocks.
1. Parameters: Max out-going connections. If nodes were failing due to CPU runaway and ping failure requests, they needed to connect to other nodes, however one setting was limiting each node from initiating more than 10 oubound connections. In a split brain case this exacerbates the inability to recover.
1. Sending expensive transactions to the network (verify VDF is extremely expensive at 150ms per verification) while in this situation may lead to the Validator processing that transaction in the a) json rpc admission control (if the port was open), b) mempool sync as a fullnode c) validator node voting/proposing the block.
1. When the network is in split brain mode, it will ordinarily consume more resources to compare forks. To do this while the above failures are happening will cause the CPU to reach maximum usage, which make networking and other tasks impossible.

## Network Topology Effects
Having the network topology diverge from Diem's design worked for happy cases, but made recovery in slow and stalled networks worse or impossible without a coordinated restart.

Like many BFT networks there is an intermediary proxy node that isolates the Validator from transactions. In diem this is the VFN (Validator Fullnode). 0L experimenting with removing this requirement from validators but it causes a number of issues.


0L configs instead had the validator share duties, connecting to the Validator network and to the Public Fullnode network simultaneously. This was to simplify the architecture and cost, to make 0L more accessible to non-professional validators.

By design the Validator should only connect to a Private fullnode, which in turn is connected to the public fullnodes.

Here's the list of things we think goes wrong with this one instance of `diem-node` serving on public fullnode network, and validator network:

1. By design the Validator node does not trust information from the Public fullnode network. Even if they are running the same instance, the validator's DB is only incremented with transactions from other validators. The fullnode functions however will be using the same database (without incrementing). There could be resource contention issues related to this.

1. If the validator falls out of sync, fullnode network will be further ahead. But the validator cannot benefit from that new data. So the node may be seeing transactions in the future from mempool synchronization, but can't make blocks from it. So the `diem-node` is working on two forks at the same time.

1. These issues compound when the network has split brain, and leaders are failing and there are competing proposals. The validator network and fullnodes may be syncing and verifying different trees.

## Mempool and Sync issues

There are three variants of state: 1. validator-produced block state (true state) -- this is consistent. All validators see the same state. 2. FN chain state. This is possibly consistent with true state, but likely older. It's never some random state, always represents some older version of true state, depending on how far behind the node is. 3. proposed txn state. This is like a ort cloud of state that originates with clients but is propagated around by FNs. txns get into the ort cloud if and only if they pass through a FN that believes (at the time) that the txn is valid. That validation is performed in the context of that node's current chain state.

Now, set up like this, we have a machine prone to thermal runaway whenever FNs fall significantly behind true state. Because the FNs will allow though more and more invalid txn the further behind they get.

Having a dedicated private fullnode for a validator improves this situation because it ensures that no txn can reach a validator without passing through an up-to-date FN. For this situation it's not the existence of the peer FN per se that fixes the problem, but rather that the FN is properly managed and hence synced up. That's not to minimize the otheer issues that occur when a `diem-node` instance is serving on both public and validator node

There's a secondary mechanism necessary fro the meltdown : the more the FNs get behind the more bogus txn they allow through, which in turn burns more CPU in the validators, and eventually the weak validators drop out of quorum. (Also exacerbated if the nodes cannot reconnect to each other).