# Troubleshooting Aug 19th 2022 Halt

## Cheatsheet

### Update validator node yaml

To make sure you are on the same commit version please do this:

cd libra
git fetch && git checkout release-v5.2.0 -f && git pull

# run the init tool from debug build. No need to compile all bins
cargo r -p ol -- init --val

## Reset Keystore file

This will reset the safety rules in the keystore file, and update with the recent waypoint

ol init --key-store --waypoint 63520776:60136ccc87e7c0a6d7cc5cfd05a3808e0cc994d3e2d570b78700616b29d4d666

### Check the state of your DB

First your node needs to be running, then execute this command.

db-backup one-shot query node-state

## Errors

### Unable to initialize safety rules.

2022-08-20T18:04:39.020262Z [consensus] ERROR consensus/src/epoch_manager.rs:309 Unable to initialize safety rules. YOUR VALIDATOR WILL NOT BE ABLE TO SIGN BLOCKS but it will be able to sync. This is likely a problem with the restore point in your DB. {"epoch":290,"error":{"ValidatorNotInSet":"7EC16859C24200D8E074809D252AC740"}}

This can be caused by a number of things.

First try to reset the key_store.json file with a correct waypoint.

ol init --key-store --waypoint 63520776:60136ccc87e7c0a6d7cc5cfd05a3808e0cc994d3e2d570b78700616b29d4d666

If that doesn't work, it may be bad db state.
If a snapshot is available on epoch-archive you can try to restore:

ol restore

### Sequence number not continuous


2022-08-20T17:12:14.518164Z [consensus] ERROR consensus/src/liveness/leader_reputation.rs:79 [leader reputation] Fail to refresh window {"error":"DB corrupt: Sequence number not continuous, expected: 45779214, actual: 45779229."}


### Liveness storage error.


2022-08-20T17:17:02.254039Z [consensus] ERROR consensus/src/persistent_liveness_storage.rs:392 Failed to construct recovery data {"error":"\nRoot id: 494f1b52\nBlocks in db: \n\t[id: bc3f6536, epoch: 290, round: 21, parent_id: 5e5bb830]\n\t[id: bce344eb (NIL), epoch: 290, round: 22, parent_id: bc3f6536]\n\t[id: 5c815252 (NIL), epoch: 290, round: 23, parent_id: bce344eb]\n\t[id: 103aea9b, epoch: 290, round: 24, parent_id: 5c815252]\n\t[id: 20e5713f (NIL), epoch: 290, round: 25, parent_id: 5c815252]\n\t[id: eb58406c (NIL), epoch: 290, round: 26, parent_id: 20e5713f]\n\t[id: 5366d816, epoch: 290, round: 27, parent_id: 20e5713f]\n\t[id: 03229a57 (NIL), epoch: 290, round: 28, parent_id: 20e5713f]\n\t[id: 827b70c8 (NIL), epoch: 290, round: 29, parent_id: 20e5713f]\n\t[id: 739abced (NIL), epoch: 290, round: 30, parent_id: 827b70c8]\n\t[id: 09e84d0f (NIL), epoch: 290, round: 31, parent_id: 739abced]\n\t[id: 2f6e3730 (NIL), epoch: 290, round: 32, parent_id: 739abced]\n\t[id: c3943f4a (NIL), epoch: 290, round: 33, parent_id: 2f6e3730]\n\t[id: d561fdcd (NIL), epoch: 290, round: 34, parent_id: c3943f4a]\nQuorum Certs in db: \n\tQuorumCert: [VoteData: [block id: 9317ea52, epoch: 286, round: 00, parent_block_id: 9317ea52, parent_block_round: 00], LedgerInfo: [commit_info: BlockInfo: [epoch: 286, round: 0, id: 9317ea52, executed_state_id: 19185856, version: 63244108, timestamp (us): 1660693832380707, next_epoch_state: None]]]\n\tQuorumCert: [VoteData: [block id: bc3f6536, epoch: 290, round: 21, parent_block_id: 5e5bb830, parent_block_round: 20], LedgerInfo: [commit_info: BlockInfo: [epoch: 290, round: 19, id: 494f1b52, executed_state_id: b6c6415d, version: 63520927, timestamp (us): 1660964830862171, next_epoch_state: None]]]\n\tQuorumCert: [VoteData: [block id: bce344eb, epoch: 290, round: 22, parent_block_id: bc3f6536, parent_block_round: 21], LedgerInfo: [commit_info: BlockInfo: [epoch: 290, round: 20, id: 5e5bb830, executed_state_id: 6698f864, version: 63520952, timestamp (us): 1660964851885128, next_epoch_state: None]]]\n\tQuorumCert: [VoteData: [block id: 5c815252, epoch: 290, round: 23, parent_block_id: bce344eb, parent_block_round: 22], LedgerInfo: [commit_info: BlockInfo: [epoch: 290, round: 21, id: bc3f6536, executed_state_id: cebf4872, version: 63520953, timestamp (us): 1660964856755297, next_epoch_state: None]]]\n\tQuorumCert: [VoteData: [block id: 20e5713f, epoch: 290, round: 25, parent_block_id: 5c815252, parent_block_round: 23], LedgerInfo: [commit_info: BlockInfo: [epoch: 0, round: 0, id: 00000000, executed_state_id: 00000000, version: 0, timestamp (us): 0, next_epoch_state: None]]]\n\tQuorumCert: [VoteData: [block id: 827b70c8, epoch: 290, round: 29, parent_block_id: 20e5713f, parent_block_round: 25], LedgerInfo: [commit_info: BlockInfo: [epoch: 0, round: 0, id: 00000000, executed_state_id: 00000000, version: 0, timestamp (us): 0, next_epoch_state: None]]]\n\tQuorumCert: [VoteData: [block id: 739abced, epoch: 290, round: 30, parent_block_id: 827b70c8, parent_block_round: 29], LedgerInfo: [commit_info: BlockInfo: [epoch: 0, round: 0, id: 00000000, executed_state_id: 00000000, version: 0, timestamp (us): 0, next_epoch_state: None]]]\n\tQuorumCert: [VoteData: [block id: 2f6e3730, epoch: 290, round: 32, parent_block_id: 739abced, parent_block_round: 30], LedgerInfo: [commit_info: BlockInfo: [epoch: 0, round: 0, id: 00000000, executed_state_id: 00000000, version: 0, timestamp (us): 0, next_epoch_state: None]]]\n\tQuorumCert: [VoteData: [block id: c3943f4a, epoch: 290, round: 33, parent_block_id: 2f6e3730, parent_block_round: 32], LedgerInfo: [commit_info: BlockInfo: [epoch: 0, round: 0, id: 00000000, executed_state_id: 00000000, version: 0, timestamp (us): 0, next_epoch_state: None]]]\n\n\nCaused by:\n    unable to find root: 494f1b52"}


### State Sync Commit or Callback Error

2022-08-20T20:24:47.694250Z [shared-mem] ERROR mempool/src/shared_mempool/tasks.rs:380 {"event":"callback_fail","name":"state_sync_commit"}

2022-08-20T20:24:49.760124Z [state-sync] ERROR state-sync/src/coordinator.rs:475 {"error":"CallbackSendFailed(\"Did not receive ACK for commit notification from mempool! Error: Elapsed(())\")","name":"commit_flow"}

2022-08-20T20:24:52.112291Z [consensus] ERROR consensus/src/state_computer.rs:84 Failed to notify state synchronizer {"error":"UnexpectedError(\"State sync client timeout: failed to receive commit() ack in time!\")"}

2022-08-20T20:24:54.767933Z [state-sync] ERROR state-sync/src/coordinator.rs:475 {"error":"CallbackSendFailed(\"Did not receive ACK for commit notification from mempool! Error: Elapsed(())\")","name":"commit_flow"}

2022-08-20T20:24:54.768104Z [state-sync] ERROR state-sync/src/coordinator.rs:484 {"error":"CallbackSendFailed(\"Failed to send commit ACK to consensus!: Ok(CommitResponse { success: false, error_message: Some(\\\"Failed to send callback: Did not receive ACK for commit notification from mempool! Error: Elapsed(())\\\") })\")","name":"commit_flow"}

## Benign Errors

### Epochs not consecutive

2022-08-20T18:16:49.028899Z [json-rpc] ERROR json-rpc/src/runtime.rs:317 {"is_batch":false,"response":{"diem_chain_id":1,"diem_ledger_version":63523321,"diem_ledger_timestampusec":1661018348133637,"jsonrpc":"2.0","id":3637994,"error":{"code":-32000,"message":"Server error: Epochs are not consecutive.","data":null}},"response_error":true,"trace_id":"c0ea936e210ea347"}

This appears benign, related to json-rpc

### Outbound connection failed on port 6179

2022-08-20T19:14:53.719264Z [network-Public] ERROR network/src/peer_manager/mod.rs:1227 [validator,Public,8b03a0f4] Outbound connection failed for peer ecaf65ad at /ip4/34.145.88.77/tcp/6179/