###### tags: `operational-dashboard` `data-pipeline` `trueblocks`

# Determining Operational Health of TrueBlocks

## Are VMs running?
Use cloud provider's native VM monitoring tools

## Are services running?
### Erigon and RPCDaemon
I run erigon and rpcdaemon under `supervisor` as specified in https://chasewright.com/getting-started-with-turbo-geth-on-ubuntu/. So,
> `supervisorctl status erigon & supervisorctl status rpcdaemon`

returns
> `erigon RUNNING pid 1162, uptime 4 days, 6:38:34`
`rpcdaemon RUNNING pid 1163, uptime 4 days, 6:38:34`

#### Logging output
`Supervisor` is configured to redirect stardard out and standard err to log files. So, I can examine these outputs with
> `tail -f /var/log/supervisor/erigon.err.log`

If you are not using `supervisor`, you should redirect erigon output to files to make it easier to analyse them.

A sample output looks like

[INFO] [02-11|15:19:55.378] [1/16 Headers] Waiting for headers...    from=10151039
[INFO] [02-11|15:20:09.222] [1/16 Headers] Processed                 highest inserted=10151040 age=0
[INFO] [02-11|15:20:09.245] [5/16 Execution] Completed on            block=10151040
[INFO] [02-11|15:20:09.516] Commit cycle                             in=252.25844ms
[INFO] [02-11|15:20:09.517] RPC Daemon notified of new headers       from=10151039 to=10151040
[INFO] [02-11|15:20:09.517] [1/16 Headers] Waiting for headers...    from=10151040
[INFO] [02-11|15:20:24.205] [1/16 Headers] Processed                 highest inserted=10151041 age=0
[INFO] [02-11|15:20:24.229] [5/16 Execution] Completed on            block=10151041
[INFO] [02-11|15:20:24.298] Commit cycle                             in=40.345841ms
[INFO] [02-11|15:20:24.298] RPC Daemon notified of new headers       from=10151040 to=10151041
[INFO] [02-11|15:20:24.298] [1/16 Headers] Waiting for headers...    from=10151041
[INFO] [02-11|15:20:39.082] [1/16 Headers] Processed                 highest inserted=10151042 age=0
[INFO] [02-11|15:20:39.096] [5/16 Execution] Completed on            block=10151042
[INFO] [02-11|15:20:39.147] Commit cycle                             in=32.331716ms
[INFO] [02-11|15:20:39.147] RPC Daemon notified of new headers       from=10151041 to=10151042
[INFO] [02-11|15:20:39.147] [1/16 Headers] Waiting for headers...    from=10151042
[INFO] [02-11|15:20:41.348] [p2p] GoodPeers                          eth66=33
[INFO] [02-11|15:20:45.098] [txpool] stat                            block=10151042 pending=9979 baseFee=12 queued=29997 alloc_mb=576 sys_mb=1194
[INFO] [02-11|15:20:54.113] [1/16 Headers] Processed                 highest inserted=10151043 age=0
[INFO] [02-11|15:20:54.130] [5/16 Execution] Completed on            block=10151043
[INFO] [02-11|15:20:54.419] Commit cycle                             in=268.621149ms
[INFO] [02-11|15:20:54.420] RPC Daemon notified of new headers       from=10151042 to=10151043
[INFO] [02-11|15:20:54.420] [1/16 Headers] Waiting for headers...    from=10151043
[INFO] [02-11|15:21:09.099] [1/16 Headers] Processed                 highest inserted=10151044 age=0
[INFO] [02-11|15:21:09.128] [5/16 Execution] Completed on            block=10151044
[INFO] [02-11|15:21:09.696] Commit cycle                             in=512.95123ms
[INFO] [02-11|15:21:09.696] RPC Daemon notified of new headers       from=10151043 to=10151044

Things to note from this output are:
* The first 5 log lines that repeat approximately every 25 sec. (See Lines 1-15, above.) This repetition is effectively the node's heartbeat. My node is running against Rinkeby. I expect other networks, in particular mainnet, to have different dynamics.
* This heartbeat is occasionally overlayed with logs from additional activity. Examples are seen in Line 17 and Line 18.
* Line 17 queries the number of peers with which this node is networked.
* Line 18 queries transaction pool (`txpool`) status.
* This heartbeat can be interrupted by error recovery. In the log shown below, a `bad_block_hash` is detected at Line 3. Lines 4-14 log recovery. Lines 15-19 log the normal next beat.
* The number of networked peers, as shown in Line 17. You must have at least one peer. More are better up to a point, after which performance can degrade. Erigon has a command line parameter to limit the number of peers, but I do not know what it is, nor do I know how to determine a good setting.
*

[INFO] [02-11|15:37:39.438] [1/16 Headers] Waiting for headers...    from=10151110
[INFO] [02-11|15:37:45.091] [txpool] stat                            block=10151110 pending=9933 baseFee=12 queued=30000 alloc_mb=515 sys_mb=1194
[INFO] [02-11|15:37:54.072] UnwindTo                                 block=10151109 bad_block_hash=0x0000000000000000000000000000000000000000000000000000000000000000
[INFO] [02-11|15:37:54.072] [1/16 Headers] Processed                 highest inserted=10151111 age=0
[INFO] [02-11|15:37:54.085] [7/16 HashState] Unwinding started       from=10151110 to=10151109 storage=false codes=true
[INFO] [02-11|15:37:54.086] [7/16 HashState] Unwinding started       from=10151110 to=10151109 storage=false codes=false
[INFO] [02-11|15:37:54.086] [7/16 HashState] Unwinding started       from=10151110 to=10151109 storage=true codes=false
[INFO] [02-11|15:37:54.087] [8/16 IntermediateHashes] Unwinding of trie hashes from=10151110 to=10151109 csbucket=AccountChangeSet
[INFO] [02-11|15:37:54.087] [8/16 IntermediateHashes] Unwinding of trie hashes from=10151110 to=10151109 csbucket=StorageChangeSet
[INFO] [02-11|15:37:54.092] [8/16 IntermediateHashes] Trie root      hash=0xbf8f2416ed1f6343055d0ffd8fd5276b007ff7f6bb3a1fbea1a7af7aeee463c8
[INFO] [02-11|15:37:54.093] [5/16 Execution] Unwind Execution        from=10151110 to=10151109
[INFO] [02-11|15:37:54.116] [5/16 Execution] Completed on            block=10151111
[INFO] [02-11|15:37:54.160] Commit cycle                             in=32.028001ms
[INFO] [02-11|15:37:54.161] RPC Daemon notified of new headers       from=10151109 to=10151111
[INFO] [02-11|15:37:54.161] [1/16 Headers] Waiting for headers...    from=10151111
[INFO] [02-11|15:38:09.167] [1/16 Headers] Processed                 highest inserted=10151112 age=0
[INFO] [02-11|15:38:09.253] [5/16 Execution] Completed on            block=10151112
[INFO] [02-11|15:38:10.176] Commit cycle                             in=897.780358ms
[INFO] [02-11|15:38:10.176] RPC Daemon notified of new headers       from=10151111 to=10151112

#### Useful curl commands

curl -s --data '{"method":"web3_clientVersion","params":[],"id":"1","jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST http://localhost:8545/
curl -s --data '{"method":"eth_chainId","params":[],"id":"1","jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST http://localhost:8545/
curl -s --data '{"method":"eth_getBalance","params":["0x31b98d14007bdee637298086988a0bbd31184523","0x0"],"id":"1","jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST http://localhost:8545/
curl -s --data '{"method":"eth_blockNumber","params":[],"id":"1","jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST http://localhost:8545/

### TrueBlocks Status

TrueBlocks Core status is queried by the CLI command:

chifra status --terse

which returns status information in the following format:

2022/02/13 17:35:37 Client:       erigon/2022.01.3/linux-amd64/go1.17.2 (archive, tracing)
2022/02/13 17:35:37 TrueBlocks:   GHC-TrueBlocks//0.25.0-alpha-28cf2d7e8-20220212 (eskey, no pinkey)
2022/02/13 17:35:37 Config Path:  /home/david/.local/share/trueblocks/
2022/02/13 17:35:37 Chain (ids):  rinkeby (4,4)
2022/02/13 17:35:37 Cache Path:   /home/david/.local/share/trueblocks/cache/rinkeby/
2022/02/13 17:35:37 Index Path:   /home/david/.local/share/trueblocks/unchained/rinkeby/
2022/02/13 17:35:37 RPC Provider: http://localhost:8545/
2022/02/13 17:35:37 Progress:     10163095, 10156754, 10162813, 10162840

This information is largely self-explanatory with the exception of the "Progress" field. The four numbers in the "Progress" field correspond to:
1. Block number at the head of the network's chain
2. Block number of TrueBlock's most recent finalized block
3. Block number of TrueBlock's most recent staged block
4. Block number of TrueBlock's most recent scraped block

### TrueBlocks Index
TrueBlocks stores the index that it builds in subdirectoies of the path listed after "Index Path:" in the status information shown above. This path will be referred to as `<IndexPath>`.

#### Unripe Index
TrueBlocks calls account references in the 27 most recently scraped blocks the "unripe" part of the index. TrueBlocks store this part of the index in plain text in the `<IndexPatah>/unripe` directory.

#### Staging Index

#### Finalized Index

#### Useful CLI commands

ls <IndexPath>/unripe/* | wc --words                      # Number of unripe blocks
cat <IndexPath>/unripe/* | wc --lines                     # Number of unripe records
cut -f 2 <IndexPath>/staging/* | sort | uniq | wc --lines # Number of staged blocks
cat <IndexPath>/staging/* | wc --lines                    # Number of staged records