Logs after disabling too_new seq numb, and mempool resubmit for disconnected.

# Logs after disabling too_new seq numb, and mempool resubmit for disconnected. We're testing two pull requests that could help mitigate the issue. https://github.com/OLSF/libra/pull/906 https://github.com/OLSF/libra/pull/905 ## Test starting an unsynced validator Users are sending transactions to network, and there are many validators already in sync and processing transasasa 0. Start syncing in validator mode 1. Start seeing prologue errors 2. Hangs with CPU at 100% 3. More prologue errors 4. Hangs on kill ## Logs ========================================== 152990 ========================================== ========================================== 152991 ========================================== ========================================== 152992 ========================================== 2021-12-19T18:30:40.900779Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:30:40.932067Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:30:40.975541Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:30:41.010716Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:30:41.053491Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:30:41.090399Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:30:41.127970Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:30:41.167705Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:30:51.389192Z [network-Validator] ERROR network/src/peer_manager/mod.rs:1227 [validator,Validator,7756310e] Outbound connection failed for peer 048ccb33 at /ip4/147 .182.193.24/tcp/6180/ln-noise-ik/ca0869dc7c7217f6867f5e10b16fd91cda9d1ee55f5d80cdb89da36f44934a30/ln-handshake/0: Transport error: Connection refused (os error 111) {"error":"Tra nsport error: Connection refused (os error 111)","network_address":"/ip4/147.182.193.24/tcp/6180/ln-noise-ik/ca0869dc7c7217f6867f5e10b16fd91cda9d1ee55f5d80cdb89da36f44934a30/ln-h andshake/0","network_context":{"role":"validator","network_id":"Validator","peer_id":"7756310e875636763d16f080bb236cab"},"remote_peer":"048CCB33BF48A0F333023E0E8ECCE568"} 2021-12-19T18:30:51.421606Z [network-Validator] ERROR network/src/peer_manager/mod.rs:1227 [validator,Validator,7756310e] Outbound connection failed for peer 93a91b21 at /ip4/204 .130.133.205/tcp/6180/ln-noise-ik/d796448f3fe45481f9f060aa58dfb6674adac8e6397ad019836a1780b2faf055/ln-handshake/0: Transport error: Connection refused (os error 111) {"error":"Tr ansport error: Connection refused (os error 111)","network_address":"/ip4/204.130.133.205/tcp/6180/ln-noise-ik/d796448f3fe45481f9f060aa58dfb6674adac8e6397ad019836a1780b2faf055/ln -handshake/0","network_context":{"role":"validator","network_id":"Validator","peer_id":"7756310e875636763d16f080bb236cab"},"remote_peer":"93A91B21218AFF3CEB211EB02B27CBE0"} 2021-12-19T18:30:55.325827Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:30:55.460559Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount[867/1910$ tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:30:55.468160Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:30:55.541222Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:30:55.593544Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:04.989617Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:04.998208Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:05.025860Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:05.077421Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:05.082938Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:05.113754Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:05.414168Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:11.672974Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:11.674741Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:11.720025Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:11.720390Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:11.760514Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:11.762334Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:11.811896Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:11.813556Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:11.853402Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:11.855202Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:12.364665Z [network-Validator] ERROR network/src/peer_manager/mod.rs:1227 [validator,Validator,7756310e] Outbound connection failed for peer b1471dc5 at /ip4/3.1 41.11.134/tcp/6180/ln-noise-ik/a386d359f7c0c8f53c1ec75853e16d73e17c5b12fb660692ad0ac84b71a3c215/ln-handshake/0: Transport error: timeout elapsed {"error":"Transport error: timeou t elapsed","network_address":"/ip4/3.141.11.134/tcp/6180/ln-noise-ik/a386d359f7c0c8f53c1ec75853e16d73e17c5b12fb660692ad0ac84b71a3c215/ln-handshake/0","network_context":{"role":"v alidator","network_id":"Validator","peer_id":"7756310e875636763d16f080bb236cab"},"remote_peer":"B1471DC5764695ABB4CADF16E26BD4C7"} 2021-12-19T18:31:13.389075Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:13.442156Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:13.451365Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:13.485438Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:13.496960Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:13.528304Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:13.539558Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:13.646051Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:13.965287Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:30.819672Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:30.849187Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:30.871134Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:30.889694Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:30.911040Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:30.980631Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:31.027697Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:31.035268Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:31.162782Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:31.187805Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:31.295054Z [shared-mem] ERROR language/diem-vm/src/errors.rs:95 [diem_vm] Unexpected prologue Move abort: 00000000000000000000000000000001::DiemAccount::1003 (Ca tegory: 235 Reason: 3) {"base_version":12618640,"name":"validation","txn_id":0} 2021-12-19T18:31:32.027044Z [state-sync] ERROR state-sync/src/coordinator.rs:206 {"error":"ReceivedNonSequentialChunk(\"PeerId:355ae7b5, NodeNetworkId:(Validator)\", \"12619640\" , \"12617641\")","name":"process_chunk_message"} 2021-12-19T18:31:32.027265Z [state-sync] ERROR state-sync/src/coordinator.rs:196 {"error":"InvalidStateSyncPeer(\"PeerId:50040b55, NodeNetworkId:(Public)\", \"inbound\")","name": "new_peer"}