# BSC Node Setup Story
I had an AWS `c6g.2xlarge` instance even set IOPS to 16k doesn't seem to help after one day. I've started a new AWS EC2 i3 instance with 900G ephemeral NVME SSD to verify whether an even higher IOPS can help sync eventually. Will update this thread if I got some progress. Date when the comment was posted: 2021-05-28.
## Update 1
Some raw numbers to check during syncing:
`i3.xlarge` instance with ephemeral native NVME, IOPS 70K, iostat MB_wrtn/s was averaged to be about 10M/s
<details>
<summary>expand to see details</summary>
```
# /usr/local/bin/geth attach http://localhost:8545 --exec "eth.syncing"
{
currentBlock: 5045035,
highestBlock: 7796190,
knownStates: 96385243,
pulledStates: 95904052,
startingBlock: 0
}
# iostat -m -d 10 10
Linux 4.14.231-173.361.amzn2.x86_64 (ip-172-31-66-40.ec2.internal) 05/28/2021 _x86_64_ (4 CPU)
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
xvda 9.06 0.04 0.10 236 587
nvme0n1 125.74 0.00 164.96 6 982969
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
xvda 0.30 0.00 0.00 0 0
nvme0n1 69.80 0.00 4.22 0 42
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
xvda 0.00 0.00 0.00 0 0
nvme0n1 202.20 0.00 18.09 0 180
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
xvda 39.20 0.00 0.34 0 3
nvme0n1 117.30 0.00 6.66 0 66
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
xvda 0.20 0.00 0.00 0 0
nvme0n1 84.00 0.00 4.57 0 45
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
xvda 35.60 0.00 0.26 0 2
nvme0n1 192.90 0.00 18.79 0 187
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
xvda 31.20 0.10 0.18 1 1
nvme0n1 87.20 0.00 4.41 0 44
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
xvda 0.40 0.00 0.00 0 0
nvme0n1 55.70 0.00 3.03 0 30
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
xvda 36.50 0.00 0.28 0 2
nvme0n1 185.50 0.00 18.31 0 183
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
xvda 0.00 0.00 0.00 0 0
nvme0n1 97.10 0.00 6.20 0 62
```
</details>
`c5.2xlarge` instance with `gp3` volume, IOPS 16K, MB_wrtn/s was almost 0
<details>
<summary>expand to see details</summary>
```
# /usr/local/bin/geth attach http://localhost:8545 --exec "eth.syncing"
{
currentBlock: 7797374,
highestBlock: 7797452,
knownStates: 687576703,
pulledStates: 687420045,
startingBlock: 7794295
}
# iostat -m -d 10 10
Linux 4.14.231-173.361.amzn2.aarch64 (ip-172-31-3-204.ec2.internal) 05/28/2021 _aarch64_ (8 CPU)
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
nvme0n1 1631.77 15.09 1.67 10541477 1166468
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
nvme0n1 1503.30 12.04 0.66 120 6
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
nvme0n1 759.20 5.85 0.00 58 0
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
nvme0n1 1555.24 12.45 0.00 124 0
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
nvme0n1 1356.30 10.76 0.00 107 0
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
nvme0n1 750.30 6.12 1.07 61 10
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
nvme0n1 837.80 6.93 0.00 69 0
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
nvme0n1 927.60 7.53 0.00 75 0
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
nvme0n1 1528.40 12.61 0.06 126 0
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
nvme0n1 1484.80 12.07 0.00 120 0
```
</details>
## Update 2
Finally synced in 10 hours!
Based on geth process log, those are some final signals before the node became synced.
- Indexing transactions, it kept printing out percentage until it reaches 100, took about 20mins for this may be, during this period `eth.syncing` result was frozen, but we can notice that pulledStates was the same as knownStates.
- Generating state snapshot, this was a new feature in 1.10 for a faster snapshot sync mode, it kept saying resuming/aborting. Just let it running for a while and don't worry about it.
<details>
<summary>expand to see details</summary>
```
# /usr/local/bin/geth attach http://localhost:8545 --exec "eth.syncing"
{
currentBlock: 7808427,
highestBlock: 7808533,
knownStates: 340871401,
pulledStates: 340871401,
startingBlock: 7803641
}
# /usr/local/bin/geth attach http://localhost:8545 --exec "eth.syncing"
false
# date
Fri May 28 17:48:12 UTC 2021
# journalctl -f -u geth
May 28 17:50:38 ip-172-31-66-40.ec2.internal geth[3882]: INFO [05-28|17:50:38.371] Imported new chain segment blocks=1 txs=104 mgas=14.633 elapsed=271.241ms mgasps=53.948 number=7,808,617 hash=d33527..30585e dirty=1019.97MiB
May 28 17:50:38 ip-172-31-66-40.ec2.internal geth[3882]: INFO [05-28|17:50:38.373] Unindexed transactions blocks=1 txs=77 tail=5,458,618 elapsed=1.835ms
May 28 17:50:41 ip-172-31-66-40.ec2.internal geth[3882]: INFO [05-28|17:50:41.471] Imported new chain segment blocks=1 txs=212 mgas=22.106 elapsed=392.647ms mgasps=56.300 number=7,808,618 hash=a543ea..e0b103 dirty=1020.60MiB
May 28 17:50:41 ip-172-31-66-40.ec2.internal geth[3882]: INFO [05-28|17:50:41.472] Unindexed transactions blocks=1 txs=59 tail=5,458,619 elapsed=1.510ms
```
</details>
## Update 3
I'll share my AWS EC2 based configuration here hope that can help others struggling with building a fully synced node.
<details>
<summary>1. EC2 Instance Config</summary>
I used a `i3.xlarge` EC2 instance with 4 CPUs, 32GB Memory, 900G native NVME and 8G gp3 volume for hosting the operating system. The AMI was Amazon Linux 2 AMI (HVM), SSD Volume Type, in 64-bit (x86) arch.
When the instance first started you have to create file system on the 900G blk device by `mkfs -t xfs /dev/nvme0n1` (check device id by running `lsblk`). And then `mkdir /opt/bsc` and `mount /dev/nvme0n1 /opt/bsc`
</details>
<details>
<summary>2. /opt/bsc/node/config.toml</summary>
```
[Eth]
NetworkId = 56
SyncMode = "fast"
NoPruning = false
NoPrefetch = false
LightPeers = 100
DatabaseCache = 24000
DatabaseFreezer = ""
TrieCleanCache = 256
TrieDirtyCache = 256
UltraLightFraction = 75
TrieTimeout = 5000000000000
EnablePreimageRecording = false
EWASMInterpreter = ""
EVMInterpreter = ""
[Eth.Miner]
DelayLeftOver = 50000000
GasFloor = 8000000
GasCeil = 8000000
GasPrice = 1000000000
Recommit = 3000000000
Noverify = false
[Eth.Ethash]
CacheDir = "ethash"
CachesInMem = 2
CachesOnDisk = 3
CachesLockMmap = false
DatasetDir = "/opt/bsc/node/ethash"
DatasetsInMem = 1
DatasetsOnDisk = 2
DatasetsLockMmap = false
PowMode = 0
[Eth.TxPool]
Locals = []
NoLocals = false
Journal = "transactions.rlp"
Rejournal = 3600000000000
PriceLimit = 1000000000
PriceBump = 10
AccountSlots = 512
GlobalSlots = 10000
AccountQueue = 256
GlobalQueue = 5000
Lifetime = 10800000000000
[Eth.GPO]
Blocks = 20
Percentile = 60
OracleThreshold = 1000
[Node]
DataDir = "/opt/bsc/node/data"
HTTPHost = "0.0.0.0"
NoUSB = true
InsecureUnlockAllowed = false
IPCPath = "geth.ipc"
HTTPPort = 8545
HTTPVirtualHosts = ["localhost"]
HTTPModules = ["net", "web3", "eth", "debug"]
WSPort = 8546
WSModules = ["net", "web3", "eth", "debug"]
[Node.P2P]
MaxPeers = 1000
NoDiscovery = false
BootstrapNodes = ["enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311","enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311","enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311"]
StaticNodes = ["enode://f3cfd69f2808ef64838abd8786342c0b22fdd28268703c8d6812e26e109f9a7cb2b37bd49724ebb46c233289f22da82991c87345eb9a2dadeddb8f37eeb259ac@18.180.28.21:30311","enode://ae74385270d4afeb953561603fcedc4a0e755a241ffdea31c3f751dc8be5bf29c03bf46e3051d1c8d997c45479a92632020c9a84b96dcb63b2259ec09b4fde38@54.178.30.104:30311","enode://d1cabe083d5fc1da9b510889188f06dab891935294e4569df759fc2c4d684b3b4982051b84a9a078512202ad947f9240adc5b6abea5320fb9a736d2f6751c52e@54.238.28.14:30311","enode://f420209bac5324326c116d38d83edfa2256c4101a27cd3e7f9b8287dc8526900f4137e915df6806986b28bc79b1e66679b544a1c515a95ede86f4d809bd65dab@54.178.62.117:30311","enode://c0e8d1abd27c3c13ca879e16f34c12ffee936a7e5d7b7fb6f1af5cc75c6fad704e5667c7bbf7826fcb200d22b9bf86395271b0f76c21e63ad9a388ed548d4c90@54.65.247.12:30311","enode://f1b49b1cf536e36f9a56730f7a0ece899e5efb344eec2fdca3a335465bc4f619b98121f4a5032a1218fa8b69a5488d1ec48afe2abda073280beec296b104db31@13.114.199.41:30311","enode://4924583cfb262b6e333969c86eab8da009b3f7d165cc9ad326914f576c575741e71dc6e64a830e833c25e8c45b906364e58e70cdf043651fd583082ea7db5e3b@18.180.17.171:30311","enode://4d041250eb4f05ab55af184a01aed1a71d241a94a03a5b86f4e32659e1ab1e144be919890682d4afb5e7afd837146ce584d61a38837553d95a7de1f28ea4513a@54.178.99.222:30311","enode://b5772a14fdaeebf4c1924e73c923bdf11c35240a6da7b9e5ec0e6cbb95e78327690b90e8ab0ea5270debc8834454b98eca34cc2a19817f5972498648a6959a3a@54.170.158.102:30311","enode://f329176b187cec87b327f82e78b6ece3102a0f7c89b92a5312e1674062c6e89f785f55fb1b167e369d71c66b0548994c6035c6d85849eccb434d4d9e0c489cdd@34.253.94.130:30311","enode://cbfd1219940d4e312ad94108e7fa3bc34c4c22081d6f334a2e7b36bb28928b56879924cf0353ad85fa5b2f3d5033bbe8ad5371feae9c2088214184be301ed658@54.75.11.3:30311","enode://c64b0a0c619c03c220ea0d7cac754931f967665f9e148b92d2e46761ad9180f5eb5aaef48dfc230d8db8f8c16d2265a3d5407b06bedcd5f0f5a22c2f51c2e69f@54.216.208.163:30311","enode://352a361a9240d4d23bb6fab19cc6dc5a5fc6921abf19de65afe13f1802780aecd67c8c09d8c89043ff86947f171d98ab06906ef616d58e718067e02abea0dda9@79.125.105.65:30311","enode://bb683ef5d03db7d945d6f84b88e5b98920b70aecc22abed8c00d6db621f784e4280e5813d12694c7a091543064456ad9789980766f3f1feb38906cf7255c33d6@54.195.127.237:30311","enode://11dc6fea50630b68a9289055d6b0fb0e22fb5048a3f4e4efd741a7ab09dd79e78d383efc052089e516f0a0f3eacdd5d3ffbe5279b36ecc42ad7cd1f2767fdbdb@46.137.182.25:30311","enode://21530e423b42aed17d7eef67882ebb23357db4f8b10c94d4c71191f52955d97dc13eec03cfeff0fe3a1c89c955e81a6970c09689d21ecbec2142b26b7e759c45@54.216.119.18:30311","enode://d61a31410c365e7fcd50e24d56a77d2d9741d4a57b295cc5070189ad90d0ec749d113b4b0432c6d795eb36597efce88d12ca45e645ec51b3a2144e1c1c41b66a@34.204.129.242:30311","enode://bb91215b1d77c892897048dd58f709f02aacb5355aa8f50f00b67c879c3dffd7eef5b5a152ac46cdfb255295bec4d06701a8032456703c6b604a4686d388ea8f@75.101.197.198:30311","enode://786acbdf5a3cf91b99047a0fd8305e11e54d96ea3a72b1527050d3d6f8c9fc0278ff9ef56f3e56b3b70a283d97c309065506ea2fc3eb9b62477fd014a3ec1a96@107.23.90.162:30311","enode://4653bc7c235c3480968e5e81d91123bc67626f35c207ae4acab89347db675a627784c5982431300c02f547a7d33558718f7795e848d547a327abb111eac73636@54.144.170.236:30311","enode://c6ffd994c4ef130f90f8ee2fc08c1b0f02a6e9b12152092bf5a03dd7af9fd33597d4b2e2000a271cc0648d5e55242aeadd6d5061bb2e596372655ba0722cc704@54.147.151.108:30311","enode://99b07e9dc5f204263b87243146743399b2bd60c98f68d1239a3461d09087e6c417e40f1106fa606ccf54159feabdddb4e7f367559b349a6511e66e525de4906e@54.81.225.170:30311","enode://1479af5ea7bda822e8747d0b967309bced22cad5083b93bc6f4e1d7da7be067cd8495dc4c5a71579f2da8d9068f0c43ad6933d2b335a545b4ae49a846122b261@52.7.247.132:30311"]
ListenAddr = ":30311"
EnableMsgEvents = false
[Node.HTTPTimeouts]
ReadTimeout = 30000000000
WriteTimeout = 30000000000
IdleTimeout = 120000000000
```
</details>
<details>
<summary>3. /etc/systemd/system/geth.service</summary>
```
[Unit]
Description=BSC geth go client
After=syslog.target network.target
[Service]
Type=simple
ExecStart=/usr/local/bin/geth --config /opt/bsc/node/config.toml --metrics --metrics.addr 0.0.0.0
KillMode=process
KillSignal=SIGINT
TimeoutStopSec=90
Restart=on-failure
RestartSec=10s
[Install]
WantedBy=multi-user.target
```
</details>
<details>
<summary>4. chain initialization steps</summary>
```
# follow the guide to setup AWS EC2 time sync
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html
# use /tmp as the working directory
cd /tmp
# download bsc 1.1.0-beta
wget https://github.com/binance-chain/bsc/releases/download/v1.1.0-beta/geth_linux
chmod +x geth_linux
mv geth_linux /usr/local/bin/geth
# initialize genesis
wget https://github.com/binance-chain/bsc/releases/download/v1.1.0-beta/mainnet.zip
unzip mainnet.zip
/usr/local/bin/geth --datadir /opt/bsc/node/data init genesis.json
# start service
systemctl enable geth
systemctl start geth
# check syncing progress
/usr/local/bin/geth attach http://localhost:8545 --exec "eth.syncing"
# check peer count
curl -s 127.0.0.1:6060/debug/metrics | grep peer
# check geth daemon logs
journalctl -f -u geth
```
</details>
## Update 4
### Cost
According to [AWS Pricing Calculator](https://calculator.aws/#/createCalculator/EC2), the cost for a `i3.xlarge` instance is about $228.40 per month.
### Caveats
`i3.xlarge` instance NVME disk was ephemeral which means the data would be gone if you ever stopped the EC2 instance. Since EC2 SLA is not particularly high (99.99% which translates into 52m 35s downtime per year), it is recommended to run at least 2 of the above instances behind a load balancer to ensure continuous service.
## Update 5
Instance type should be changed to `i3.2xlarge` now due to disk space usage is reaching `i3.xlarge` instance type's SSD upper limits.
Sign in with Wallet
Connect another wallet