# Beacon Node DB

PR: https://github.com/sigp/lighthouse/pull/4718

During the EPF, I put together some code that abstracted away some of the levelDB specific database functionality within the beacon node backend. The existing `KeyValueStore` trait was already doing some of the abstraction, however the code always "assumed" the KV store was levelDB.

TLDR I copied the slasher db architecture! I created a `BeaconNodeBackend` type that implements the `KeyValueStore` and `ItemStore` traits. I then replaced all references of levelDB w/ the new `BeaconNodeBackend` type. Within the BeaconNodeBackend type a `cfg!` macro was used to activate levelDB/Redb specific code paths based on config values.

Recently I was able to add Redb as an alternative database implementation. Redb is an ACID embedded key value store implemented in pure rust. Data is stored in copy-on-write B-trees. See the [design doc](https://github.com/cberner/redb/blob/master/docs/design.md) for more info

I've ran both the levelDB and Redb db implementations on mainnet and here are the results

## Metrics Round 1

In this round of metrics gathering I have also included block processing metrics AND disk read/writes from the process itself. I used MacOS built in activity monitor to track disk read/writes for the lighthouse process.

### Results

I made two separate runs for each db implementation. The data for each run can be found below. To summarize

- Redb bytes written to disk was much less than levelDB. We're talking about roughly 12Gb writes per hour for Redb vs 32 Gb writes per hour for levelDb.
- Redb bytes read from disk results were inconsistent between the two runs, I'll probably need to make a third run to confirm values but it seems that Redb required much less reads from disk vs levelDB.
- Block Processing DB Write for redb fluctuated between 25-40ms vs levelDB at 20-25ms
- Block processing DB reads seemed slightly faster for Redb on average, especially during read spikes.

### Redb

#### Run #1

Note, my computer fell asleep during Run #1, some of the data here may be skewed because of this.

**Block Processing Metrics**: https://snapshots.raintank.io/dashboard/snapshot/pQ0M8E9EOV2w7V3mSsxPSH1auzGXonOL

**Database Metrics**: https://snapshots.raintank.io/dashboard/snapshot/t4xr2Xia6alpL46P9cW9w3UtzZ8qayx6

| Start Time | End Time | Bytes Written To Disk | Bytes Read From Disk |
| -------- | -------- | -------- | ------- |
| 2024-08-02 16:10 GMT +2 | 2024-08-02 23:39 GMT +2 | 90.55 GB | 4.90 GB |

**Bytes written per hour**: 11.82 GB/hr

#### Run #2

**DB metrics**:
https://lhm.sigp.io/dashboard/snapshot/iapHRvhuL5H9ba7FQNiZ4Sr92ZiJXad7

**Block processing metrics**:
https://snapshots.raintank.io/dashboard/snapshot/RKXtZTa8uKVrYxaE9QJ3GjllevRGdUkI

| Start Time | End Time | Bytes Written To Disk | Bytes Read From Disk |
| -------- | -------- | -------- | -------- |
| 2024-09-03 18:12 GMT +2 | 2024-09-03 23:30 GMT+2 | 64.37GB | 491.0MB |

**Bytes written per hour**: 12.12 GB/hr

### levelDB

#### Run #1

**Block processing metrics**:
https://snapshots.raintank.io/dashboard/snapshot/XN3zWddfAvoMn0R52WnYZ4u5ZxH9UPQi

**Database Metrics**: https://snapshots.raintank.io/dashboard/snapshot/6w5E73M0C47zCfeZsrYStYoBAYqFnTkE

| Start Time | End Time | Bytes Written To Disk | Bytes Read From Disk |
| -------- | -------- | -------- | -------- |
| 2024-09-03 00:00 GMT +2 | 2024-09-03 9:03 GMT + 2 | 296.37GB | 20.25GB |

**Bytes written per hour**: 32.4 Gb/hr

#### Run #2

**DB Metrics**: https://snapshots.raintank.io/dashboard/snapshot/4olVlEPmopbZ7vqQMAIg46m5Miqwirpf

**Block processing metrics**:
https://snapshots.raintank.io/dashboard/snapshot/TyIl4iLERJRKohI43xT8g3xPZzubvlxR

| Start Time | End Time | Bytes Written To Disk | Bytes Read From Disk |
| -------- | -------- | -------- | ------- |
| 2024-09-03 09:10 GMT +2 | 2024-09-03 18:01 GMT +2 | 284.79GB | 27.56 GB|

**Bytes written per hour**: 31.7GB/hr

## Metrics Round 2

In this round I've disabled backfill-sync

### Redb

**DB Metrics**: https://snapshots.raintank.io/dashboard/snapshot/hMwTYy5rQbQcoOVbNjh4cM5qpPNuh28w

**Block processing metrics**:
https://snapshots.raintank.io/dashboard/snapshot/pAL8KxKXv6Dv4D94CMXjE59Zl2GkGb4C

| Start Time | End Time | Bytes Written To Disk | Bytes Read From Disk |
| -------- | -------- | -------- | ------- |
| 2024-10-02 00:46 GMT+2 | 2024-10-02 09:22 GMT+2 | 105.33GB | 899.2MB|

**Bytes written per hour**: 12.24GB/hr

## Next steps

I think we should "stress" test the redb implementation. The goal here would be to understand at what scenarios the database could become corrupted. We'll also want to trigger a db compaction.