# DevOps Engineer Trial Project - K8S cluster for running bitcoind

###### tags: `Engineering` `Recruiting`

## Problem statement

> Bitcoin is a decentralized digital currency, without a central bank or single administrator, that can be sent from user to user on the peer-to-peer bitcoin network without the need for intermediaries. - https://en.wikipedia.org/wiki/Bitcoin

We want to offer our users to participate in the bitcoin network by running full nodes, and plan to run these nodes on a Kubernetes cluster. Here are the [minimum requirements](https://bitcoin.org/en/full-node#minimum-requirements) to run a full node on [mainnet](https://academy.binance.com/en/glossary/mainnet).

- 350 GB of free disk space, accessible at a minimum read/write speed of 100 MB/s.
- 2 GB of memory

It will take several days to make a fresh full node ready because it has to download all the history data. The easiest way to accelerate the synchronization process is to share the blockchain data from an up-to-date node with new nodes. The summary of shared data is listed in the following table.

| Filename | Disk usage | I/O Frequency |
| -------- | ---------- | ------------- |
| blocks/* | high | low |
| blocks/index/* | low | high |
| chainstate/* | low | high |

:::warning
Each node has a unique block database, and all of the files are highly connected. So if you copy just a few files from one installation's "blocks" or "chainstate" directories into another installation, this will almost certainly cause the second node to crash or get stuck at some random point in the future.
:::

You can see more details in [this article](https://bitcoindev.network/understanding-the-data/).

https://en.bitcoin.it/wiki/Splitting_the_data_directory

Each full node is a pod on a Kubernetes cluster. We leverage OverlayFS to share blockchain data between these pods.

| File systems | PV type | Shared | Size |
| ------------ | ------- | ------ | ---- |
| Upper | EBS | F (Each pod will have its own EBS volume) | 6G |
| Lower | Self-hosting NFS on a AWS EC2 instance | T | 350G |

There is only one dedicated full node which can write to the lower file system.

Here are the AWS resources we use for this project:

- EKS
- EC2 instances
- Kubernetes nodes (c5.xlarge)
- self-hosing NFS (m6g.8xlarge)

## Problems

### High disk I/O
The main bottleneck we encounter is the high I/O on *chainstate*. The *chainstate* folder is the location of the blockchain state database (LevelDB). When a full node receives a new block, it has to update every spendable coin, who owns it, and how much it's worth. The synchronization process will create read pressure to the lower file system, and write pressure to the upper file system.

We've tried AWS EFS and soon we ran out of credits so later we switched to self-hosting NFS. The NFS we set up can support up to 200 nodes now. We're searching for the scaling solution for the file system.

### High cost

A detailed cost analysis is available here: https://hackmd.io/jmvoL6liR9WmwSlV3JQ9SA

We run the cluster using AWS EKS on EC2 and the cost is quite high due to the usage of on-demand EC2 instances. According to our experiment, a c5.xlarge instance can accommodate about 11 nodes. It seems that reserved instances in the only way to lower the cost here. The use of spot instances might hinder user experience so currently we don't consider it. We're wondering if it's possible to lower the cost to $5 per full node per month.