Postmortem of incident on 2022-10-15

What has happened?

At 22:34 GMT we have received a message from Lido asking if something has changed on our side.
They have been receiving a lot of blocks in the last couple of ours.
After checking our bundle-events I've seen that we haven't been delivering any blocks in the last 50 slots (which is unusual, we have at least one that gets accepted).
Our relay has declared that we've submitted in total 2000 blocks, which is higher that I've expected.
Looking at one of the blocks coming from our relay has revealed that the block's reward was not matching the declared reward in the blinded block.
Looking at the last block that got delivered, we have seen that the coinbase of the block was set to https://etherscan.io/address/0x5caf7c1b096cf684b09ece3d3a142db0d46fc58e .

What has gone wrong?

With hindsight two things have gone wrong.

If only one of them has gone wrong, this would've not happened.

1. We have not protected the block submission endpoint of our relay, which enabled an unknown party to connect their own block builder to our relay.
2. I've disabled checking of the block rewards for the block submitted to the relay.
   Whoever has connected their builders has figured out that the check has been disabled and has started submitting blocks with wrongly declared block rewards.
   The submitted blocks have been accepted by the block proposers, since mev-boost trusts mev-boost-relay to return the exact profit for the block proposer.

Immediate mitigation

After realising what was going on the first action was to stop submission of the blocks.
I've added basic authentication for the k8s ingress of the relay, which has effectively stopped all HTTP requests from succeeding.
This has given me time to refine the approach by creating different k8s ingresses for block building paths, which require HTTP authentication and all other paths for the mev-boost-relay service.
I've also configured our builders to use the new basic authentication when communicating with the relay.
All of the above has effectively stopped everyone else from connecting builders to our relay.

Long term mitigation

To fix the issue in the long run, we have enabled following checks for the incoming blocks into our relay:

1. If the coinbase of the block is not the same as the proposed reward address, then the last transaction in the block must be for the reward address and the amount in ETH must match the declared block profit. (this is what Flashbots is using and we'll switch to this soon)

2. If the coinbase of the block is the same as the proposed reward address, then the sum of transaction tips under consideration of the current base fee must match the declared block profit (this is what we're using right now)

What were the consequences?

In total, 183 blocks have been submitted with the wrong reward.
Total amount of block rewards that have not been delivered to the block proposers is 7.472492260431932751 ETH.
3 different block proposers have been used to submit those blocks:
0x8cc02635…a346263a1f: 109
0xa095ee16…c23309e8c4: 58
0x973377ac…de4c19574a: 16

Apart from the monetary damage, this is a huge reputation loss for us.
I do wonder if the ETH from the address 0x5caf7c1b096cf684b09ece3d3a142db0d46fc58e will ever be moved. If not, the aim of the unknown block builder was definitely only the reputation damage for Manifold.

Lessons Learned

• I should never remove block reward checks from the relay
• We should very carefully decide who can connect to our relay as a block builder