v8-Rho: stateful upgrade

--- tags: v8-rho --- # v8-Rho: stateful upgrade ## Start from `v7.1.0` and funded exploited address - Result: **FAILING** - Starting from `v7.1.0` - Funded account `cosmos13dqvh4qtg4gzczuktgnw8gc2ewnwmhdwnctekxctyr4azz4dcyysecgq7e` with `3460036989uatom` - Upgrading to `v8.0.0-rc1` - Upgrade at block height 12010180 - Single validator - Genesis file: https://files.polypore.xyz/genesis/mainnet-genesis-tinkered/tinkered-genesis_2022-09-11T07%3A43%3A05.6452382Z_v7.0.3_12010083.json.gz - Error message: `panic: cannot delete latest saved version (1)` ` ``` Jan 09 06:20:41 dante-rho-testing cosmovisor[825]: 6:20AM INF received complete proposal block hash=4C1DF0A3AC845DD56397BA4FD3B23D188CD173628362CDDE3EB7765A34049E44 height=12010177 module=consensus Jan 09 06:20:41 dante-rho-testing cosmovisor[825]: 6:20AM INF finalizing commit of block hash={} height=12010177 module=consensus num_txs=0 root=1DD27E7BB941B0C2CC39FA476328698530C5028455B7ABAD147229E587F38C67 Jan 09 06:20:41 dante-rho-testing cosmovisor[825]: 6:20AM INF minted coins from module account amount=30304209uatom from=mint module=x/bank Jan 09 06:20:41 dante-rho-testing cosmovisor[825]: 6:20AM INF executed block height=12010177 module=state num_invalid_txs=0 num_valid_txs=0 Jan 09 06:20:41 dante-rho-testing cosmovisor[825]: 6:20AM INF commit synced commit=436F6D6D697449447B5B333620353720362036302032303120352032343120313620323134203137302032343720313839203931203230362032303020322031373320323234203234342037322032313720313631203133352031353520393620323533203130362031313720313737203132392036362033345D3A4237343243317D Jan 09 06:20:41 dante-rho-testing cosmovisor[825]: 6:20AM INF committed state app_hash=2439063CC905F110D6AAF7BD5BCEC802ADE0F448D9A1879B60FD6A75B1814222 height=12010177 module=state num_txs=0 Jan 09 06:20:41 dante-rho-testing cosmovisor[825]: 6:20AM INF indexed block height=12010177 module=txindex Jan 09 06:20:46 dante-rho-testing cosmovisor[825]: 6:20AM INF Timed out dur=4902.1848 height=12010178 module=consensus round=0 step=1 Jan 09 06:20:49 dante-rho-testing cosmovisor[825]: 6:20AM INF Timed out dur=3000 height=12010178 module=consensus round=0 step=3 Jan 09 06:20:50 dante-rho-testing cosmovisor[825]: 6:20AM INF Timed out dur=1000 height=12010178 module=consensus round=0 step=7 Jan 09 06:20:50 dante-rho-testing cosmovisor[825]: 6:20AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"4236329B4C82E1A607C066138063F95C0467120D2E0B3F9E525FE0C2B93D458D","parts":{"hash":"D9589CEBDD6D4885FE004C1DFC690DBED2F1E2D13359548AFD1B0EBAD70BB79A","total":1}},"height":12010178,"pol_round":-1,"round":1,"signature":"Z+1yR9IqaNyQGrs8nHfnuOkAscZoEYFM16SdXoCxkxDhtUxDf9QrrgUlqj3ajR0tnMCph6uwkC3jrCWwQuxvDw==","timestamp":"2023-01-09T11:20:50.165255172Z"} Jan 09 06:20:50 dante-rho-testing cosmovisor[825]: 6:20AM INF received complete proposal block hash=4236329B4C82E1A607C066138063F95C0467120D2E0B3F9E525FE0C2B93D458D height=12010178 module=consensus Jan 09 06:20:50 dante-rho-testing cosmovisor[825]: 6:20AM INF finalizing commit of block hash={} height=12010178 module=consensus num_txs=0 root=2439063CC905F110D6AAF7BD5BCEC802ADE0F448D9A1879B60FD6A75B1814222 Jan 09 06:20:50 dante-rho-testing cosmovisor[825]: 6:20AM INF minted coins from module account amount=30304205uatom from=mint module=x/bank Jan 09 06:20:50 dante-rho-testing cosmovisor[825]: 6:20AM INF executed block height=12010178 module=state num_invalid_txs=0 num_valid_txs=0 Jan 09 06:20:50 dante-rho-testing cosmovisor[825]: 6:20AM INF commit synced commit=436F6D6D697449447B5B3234382031313520363420323533203139312032343120313831203539203135332038302031383120323720313032203134362031323120313120323620343420313830203232362032313520313631203431203130362032323620323533203333203133322032343820323136203130352034305D3A4237343243327D Jan 09 06:20:50 dante-rho-testing cosmovisor[825]: 6:20AM INF committed state app_hash=F87340FDBFF1B53B9950B51B6692790B1A2CB4E2D7A1296AE2FD2184F8D86928 height=12010178 module=state num_txs=0 Jan 09 06:20:50 dante-rho-testing cosmovisor[825]: 6:20AM INF indexed block height=12010178 module=txindex Jan 09 06:20:50 dante-rho-testing cosmovisor[825]: 6:20AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10 Jan 09 06:20:50 dante-rho-testing cosmovisor[825]: 6:20AM INF No addresses to dial. Falling back to seeds module=pex Jan 09 06:20:55 dante-rho-testing cosmovisor[825]: 6:20AM INF Timed out dur=4888.529685 height=12010179 module=consensus round=0 step=1 Jan 09 06:20:55 dante-rho-testing cosmovisor[825]: 6:20AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"BAF95A2A58F9003C7D29DDC78703828F5CFBEC362467B35AB77F8BA9886D16E6","parts":{"hash":"A203A2D6EAB592BC957A89991F256F9CEDD0F2036497BE6252611C7DE7D7EB60","total":1}},"height":12010179,"pol_round":-1,"round":0,"signature":"4NpdPC7V2iJlXdxI8GgWK/c0gv3eq5jxObI6+PXHQn3tfwt0RPxGGoyiRXG/M/wycvwOVf1Lt/Hogea2jLlGCw==","timestamp":"2023-01-09T11:20:55.179187182Z"} Jan 09 06:20:55 dante-rho-testing cosmovisor[825]: 6:20AM INF received complete proposal block hash=BAF95A2A58F9003C7D29DDC78703828F5CFBEC362467B35AB77F8BA9886D16E6 height=12010179 module=consensus Jan 09 06:20:55 dante-rho-testing cosmovisor[825]: 6:20AM INF finalizing commit of block hash={} height=12010179 module=consensus num_txs=0 root=F87340FDBFF1B53B9950B51B6692790B1A2CB4E2D7A1296AE2FD2184F8D86928 Jan 09 06:20:55 dante-rho-testing cosmovisor[825]: 6:20AM INF minted coins from module account amount=30304201uatom from=mint module=x/bank Jan 09 06:20:55 dante-rho-testing cosmovisor[825]: 6:20AM INF executed block height=12010179 module=state num_invalid_txs=0 num_valid_txs=0 Jan 09 06:20:55 dante-rho-testing cosmovisor[825]: 6:20AM INF commit synced commit=436F6D6D697449447B5B3134302032372031353220313034203233322031333720323135203132352033382032343120323036203232352036352031383720313420323332203230352032353120323131203534203133362032343820323820323434203136362031323220313833203335203833203839203233392032315D3A4237343243337D Jan 09 06:20:55 dante-rho-testing cosmovisor[825]: 6:20AM INF committed state app_hash=8C1B9868E889D77D26F1CEE141BB0EE8CDFBD33688F81CF4A67AB7235359EF15 height=12010179 module=state num_txs=0 Jan 09 06:20:55 dante-rho-testing cosmovisor[825]: 6:20AM INF indexed block height=12010179 module=txindex Jan 09 06:21:00 dante-rho-testing cosmovisor[825]: 6:21AM INF Timed out dur=4858.580359 height=12010180 module=consensus round=0 step=1 Jan 09 06:21:00 dante-rho-testing cosmovisor[825]: 6:21AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"E9190EB945FDA65BFB6C073FBE28DCFDE63A0622B8870E5D9D593AB6DCCFB27E","parts":{"hash":"5A2B1504CB60F6B60F5783F94F6F98BEF3FB14B386E1E537B6DABE2774DE0218","total":1}},"height":12010180,"pol_round":-1,"round":0,"signature":"klSeQlHpQ/83FzebofleqTaFaXB55KN59vBQhuEzCOYhQOWG0ecs4lFS4VeMOVCie+rmRB2Hg7oLVMs0qwJtAQ==","timestamp":"2023-01-09T11:21:00.197033786Z"} Jan 09 06:21:00 dante-rho-testing cosmovisor[825]: 6:21AM INF received complete proposal block hash=E9190EB945FDA65BFB6C073FBE28DCFDE63A0622B8870E5D9D593AB6DCCFB27E height=12010180 module=consensus Jan 09 06:21:00 dante-rho-testing cosmovisor[825]: 6:21AM INF finalizing commit of block hash={} height=12010180 module=consensus num_txs=0 root=8C1B9868E889D77D26F1CEE141BB0EE8CDFBD33688F81CF4A67AB7235359EF15 Jan 09 06:21:00 dante-rho-testing cosmovisor[825]: 6:21AM ERR UPGRADE "v8-Rho" NEEDED at height: 12010180: {"binaries":{"linux/amd64":"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-linux-amd64?checksum=sha256:db106075fe866601727dc67c81011fe0e05aa5aef5fbb15c7986ddb7a2bb4326","linux/arm64":"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-linux-arm64?checksum=sha256:a56923a14c6970c8a4978d8e256e1e50277738541c80dd3cfba8d3a8f64636ad","darwin/amd64":"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-darwin-amd64?checksum=sha256:0a56c72ba8e3fe27088cde309e1a821355f98ac6cf8b6e36b664ce1879aabb3b","darwin/arm64":"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-darwin-arm64?checksum=sha256:f11462f585d66cfd2061bf719b33142b9ec3324c17f478fb627299030a3c3c9e","windows/amd64":"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-windows-amd64.exe?checksum=sha256:75c8eb84c893a7ce02bb6e54722d9fe53dc912dceb34a43f0e4536cd000827b2"}} Jan 09 06:21:00 dante-rho-testing cosmovisor[825]: 6:21AM ERR CONSENSUS FAILURE!!! err="UPGRADE \"v8-Rho\" NEEDED at height: 12010180: {\"binaries\":{\"linux/amd64\":\"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-linux-amd64?checksum=sha256:db106075fe866601727dc67c81011fe0e05aa5aef5fbb15c7986ddb7a2bb4326\",\"linux/arm64\":\"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-linux-arm64?checksum=sha256:a56923a14c6970c8a4978d8e256e1e50277738541c80dd3cfba8d3a8f64636ad\",\"darwin/amd64\":\"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-darwin-amd64?checksum=sha256:0a56c72ba8e3fe27088cde309e1a821355f98ac6cf8b6e36b664ce1879aabb3b\",\"darwin/arm64\":\"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-darwin-arm64?checksum=sha256:f11462f585d66cfd2061bf719b33142b9ec3324c17f478fb627299030a3c3c9e\",\"windows/amd64\":\"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-windows-amd64.exe?checksum=sha256:75c8eb84c893a7ce02bb6e54722d9fe53dc912dceb34a43f0e4536cd000827b2\"}}" module=consensus stack="goroutine 101 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:727 +0x4c\npanic({0x1932180, 0xc130546170})\n\truntime/panic.go:838 +0x207\ngithub.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0x7fffd5b94e26, 0x10}, 0xc000d4bda0, {0x250e6f0, 0xc000e6d390}, {0x252c5b8, 0xc00103fa00}, 0xc00109e540, {0x2508740, 0xc0004addc0}, ...}, ...)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.9/x/upgrade/abci.go:71 +0xf9f\ngithub.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.9/x/upgrade/module.go:130\ngithub.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x2521318, 0xc000052060}, {0x252dec0, 0xc02279b480}, {{0xb, 0x0}, {0xc01c9c2290, 0xd}, 0xb742c4, ...}, ...}, ...)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.9/types/module/module.go:491 +0x3bb\ngithub.com/cosmos/gaia/v7/app.(*GaiaApp).BeginBlocker(_, {{0x2521318, 0xc000052060}, {0x252dec0, 0xc02279b480}, {{0xb, 0x0}, {0xc01c9c2290, 0xd}, 0xb742c4, ...}, ...}, ...)\n\tgithub.com/cosmos/gaia/v7/app/app.go:707 +0x85\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc091d683e0, 0x20, 0x20}, {{0xb, 0x0}, {0xc01c9c2290, 0xd}, 0xb742c4, {0xb3d876f, ...}, ...}, ...})\n\tgithub.com/cosmos/cosmos-sdk@v0.45.9/baseapp/abci.go:177 +0x97c\ngithub.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc091d683e0, 0x20, 0x20}, {{0xb, 0x0}, {0xc01c9c2290, 0xd}, 0xb742c4, {0xb3d876f, ...}, ...}, ...})\n\tgithub.com/tendermint/tendermint@v0.34.21/abci/client/local_client.go:280 +0x118\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc091d683e0, 0x20, 0x20}, {{0xb, 0x0}, {0xc01c9c2290, 0xd}, 0xb742c4, {0xb3d876f, ...}, ...}, ...})\n\tgithub.com/tendermint/tendermint@v0.34.21/proxy/app_conn.go:81 +0x55\ngithub.com/tendermint/tendermint/state.execBlockOnProxyApp({0x25221c0?, 0xc14a4c11a0}, {0x25275a0, 0xc001163350}, 0xc01ab90f00, {0x252d748, 0xc000e9a258}, 0xb742c3?)\n\tgithub.com/tendermint/tendermint@v0.34.21/state/execution.go:307 +0x3dd\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc1e80b8370, 0x8}}, {0xc1e80b8380, 0xd}, 0xb74264, 0xb742c3, {{0xc08edeaee0, ...}, ...}, ...}, ...)\n\tgithub.com/tendermint/tendermint@v0.34.21/state/execution.go:140 +0x171\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc001010a80, 0xb742c4)\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1659 +0xafd\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc001010a80, 0xb742c4)\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1568 +0x2ff\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1()\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1503 +0x87\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit(0xc001010a80, 0xb742c4, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1541 +0xcb7\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc001010a80, 0xc035155f40, {0x0, 0x0})\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:2155 +0xb7c\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc001010a80, 0xc035155f40, {0x0?, 0x47e566?})\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1953 +0x2c\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc001010a80, {{0x2509900?, 0xc2196d3318?}, {0x0?, 0x0?}})\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:856 +0x44b\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc001010a80, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:783 +0x512\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:379 +0x12d\n" Jan 09 06:21:00 dante-rho-testing cosmovisor[825]: 6:21AM INF service stop impl={"Logger":{}} module=consensus msg={} wal=/home/gaia/.gaia/data/cs.wal/wal Jan 09 06:21:00 dante-rho-testing cosmovisor[825]: 6:21AM INF service stop impl={"Dir":"/home/gaia/.gaia/data/cs.wal","Head":{"ID":"9ty8qCLUqXhG:/home/gaia/.gaia/data/cs.wal/wal","Path":"/home/gaia/.gaia/data/cs.wal/wal"},"ID":"group:9ty8qCLUqXhG:/home/gaia/.gaia/data/cs.wal/wal","Logger":{}} module=consensus msg={} wal=/home/gaia/.gaia/data/cs.wal/wal Jan 09 06:21:00 dante-rho-testing cosmovisor[680]: 6:21AM INF daemon shutting down in an attempt to restart module=cosmovisor Jan 09 06:21:00 dante-rho-testing cosmovisor[680]: 6:21AM INF no upgrade binary found, beginning to download it module=cosmovisor Jan 09 06:21:01 dante-rho-testing cosmovisor[680]: 6:21AM INF downloading binary complete module=cosmovisor Jan 09 06:21:01 dante-rho-testing cosmovisor[680]: 6:21AM INF pre-upgrade command does not exist. continuing the upgrade. module=cosmovisor Jan 09 06:21:01 dante-rho-testing cosmovisor[680]: 6:21AM INF upgrade detected, relaunching app=gaiad module=cosmovisor Jan 09 06:21:01 dante-rho-testing cosmovisor[680]: 6:21AM INF running app args=["start","--x-crisis-skip-assert-invariants","--home","/home/gaia/.gaia"] module=cosmovisor path=/home/gaia/.gaia/cosmovisor/upgrades/v8-rho/bin/gaiad Jan 09 06:21:01 dante-rho-testing cosmovisor[1286]: 6:21AM INF starting node with ABCI Tendermint in-process Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF service start impl=multiAppConn module=proxy msg={} Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF service start connection=query impl=localClient module=abci-client msg={} Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF service start connection=snapshot impl=localClient module=abci-client msg={} Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF service start connection=mempool impl=localClient module=abci-client msg={} Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF service start connection=consensus impl=localClient module=abci-client msg={} Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF service start impl=EventBus module=events msg={} Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF service start impl=PubSub module=pubsub msg={} Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF service start impl=IndexerService module=txindex msg={} Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF ABCI Handshake App Info hash="�\x1b�h��}&��A�\x0e��6��\x1c��z�#SY�\x15" height=12010179 module=consensus protocol-version=0 software-version=v8.0.0-rc1 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF ABCI Replay Blocks appHeight=12010179 module=consensus stateHeight=12010179 storeHeight=12010180 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF Replay last block using real app module=consensus Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF applying upgrade "v8-Rho" at height: 12010180 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF start to run module migrations... Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF adding a new module: globalfee Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF adding a new module: packetforwardmiddleware Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF migrating module transfer from version 1 to version 2 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF running the rest of the upgrade handler... Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF upgrade complete Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF minted coins from module account amount=30304197uatom from=mint module=x/bank Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: 6:21AM INF executed block height=12010180 module=consensus num_invalid_txs=0 num_valid_txs=0 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: panic: cannot delete latest saved version (1) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: goroutine 1 [running]: Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).PruneStores(0xc000466dd0, 0x1, {0x0?, 0x0?, 0xc000f24800?}) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/cosmos-sdk@v0.45.11/store/rootmulti/store.go:460 +0x285 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc000466dd0) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/cosmos-sdk@v0.45.11/store/rootmulti/store.go:429 +0x18b Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc0000eee00) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/cosmos-sdk@v0.45.11/baseapp/abci.go:314 +0x1e9 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc000e1e8a0) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint@v0.34.24/abci/client/local_client.go:264 +0xb6 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc0012906b0?) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint@v0.34.24/proxy/app_conn.go:93 +0x22 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint/state.(*BlockExecutor).Commit(_, {{{0xb, 0x0}, {0xc0000538c8, 0x8}}, {0xc000053920, 0xd}, 0xb74264, 0xb742c4, {{0xc000d7cbe0, ...}, ...}, ...}, ...) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint@v0.34.24/state/execution.go:228 +0x269 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0000538c8, 0x8}}, {0xc000053920, 0xd}, 0xb74264, 0xb742c4, {{0xc000d7cbe0, ...}, ...}, ...}, ...) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint@v0.34.24/state/execution.go:180 +0x6ee Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc0000538c8, 0x8}}, {0xc000053920, 0xd}, 0xb74264, 0xb742c3, {{0xc000f2e560, ...}, ...}, ...}, ...) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:503 +0x23c Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc0000538c8, 0x8}}, {0xc000053920, 0xd}, 0xb74264, 0xb742c3, {{0xc000f2e560, ...}, ...}, ...}, ...) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:416 +0x7ae Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc001291d90, {0x2563c28, 0xc00044c820}) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:268 +0x3c8 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc0000538c8, 0x8}}, {0xc000053920, 0xd}, 0xb74264, 0xb742c3, ...}, ...) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint@v0.34.24/node/node.go:329 +0x1b8 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint/node.NewNode(0xc000c9a8c0, {0x2553620, 0xc000cf1540}, 0xc0004b4c80, {0x253e9e0, 0xc000f601e0}, 0x0?, 0x0?, 0xc0004b5000, {0x2557600, ...}, ...) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/tendermint/tendermint@v0.34.24/node/node.go:777 +0x597 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x256a558, 0xc000cb29c0}, {0xc00011e8c0, 0xe}, {0x255a888, 0xc0004b4f60}, ...}, ...) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/cosmos-sdk@v0.45.11/server/start.go:280 +0x7db Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc00017c300?, {0xc000cb2360?, 0x0?, 0x3?}) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/cosmos-sdk@v0.45.11/server/start.go:128 +0x169 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/spf13/cobra.(*Command).execute(0xc00017c300, {0xc000cb22d0, 0x3, 0x3}) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/spf13/cobra@v1.6.1/command.go:916 +0x862 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/spf13/cobra.(*Command).ExecuteC(0xc001084c00) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/spf13/cobra.(*Command).Execute(...) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/spf13/cobra@v1.6.1/command.go:968 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/spf13/cobra.(*Command).ExecuteContext(...) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/spf13/cobra@v1.6.1/command.go:961 Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc00011b440, 0x10}) Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/cosmos-sdk@v0.45.11/server/cmd/execute.go:36 +0x1eb Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: main.main() Jan 09 06:21:16 dante-rho-testing cosmovisor[1286]: github.com/cosmos/gaia/v8/cmd/gaiad/main.go:16 +0x2c Jan 09 06:21:16 dante-rho-testing cosmovisor[680]: Error: exit status 2 Jan 09 06:21:16 dante-rho-testing cosmovisor[680]: 6:21AM ERR error="exit status 2" module=cosmovisor Jan 09 06:21:16 dante-rho-testing systemd[1]: cosmovisor.service: Main process exited, code=exited, status=1/FAILURE Jan 09 06:21:16 dante-rho-testing systemd[1]: cosmovisor.service: Failed with result 'exit-code'. Jan 09 06:21:16 dante-rho-testing systemd[1]: cosmovisor.service: Consumed 17min 52.018s CPU time. ``` ## Start from `v7.1.0` with Jan 6 export - Result: **FAILING** - Starting from `v7.1.0` - Account `cosmos13dqvh4qtg4gzczuktgnw8gc2ewnwmhdwnctekxctyr4azz4dcyysecgq7e` has a balance of `3460036989uatom` - Upgrading to `v8.0.0-rc1` - Upgrade at block height 13562550 - Single validator - Genesis file: https://files.polypore.xyz/genesis/mainnet-genesis-tinkered/tinkered-genesis_2023-01-06T20%3A22%3A50.460851274Z_v7.1.0_13562370.json.gz - Error message: `panic: cannot delete latest saved version (1)` ``` Jan 09 07:33:12 dante-rho-testing cosmovisor[815]: 7:33AM INF indexed block height=13562549 module=txindex Jan 09 07:33:17 dante-rho-testing cosmovisor[815]: 7:33AM INF Timed out dur=4864.271451 height=13562550 module=consensus round=0 step=1 Jan 09 07:33:17 dante-rho-testing cosmovisor[815]: 7:33AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"3CE4149360D3025FBA6BAE655D334E6D2982E810EC8A0D2B115743E9D52E69C8","parts":{"hash":"A77DB15AB01F2B7E7B6AEE308F011D59432FBFF104E0DF1B40C534EA64F2C835","total":1}},"height":13562550,"pol_round":-1,"round":0,"signature":"bqCm8lvRITrM9h0kQZQrtK3wvUfvnA6mfLoGUqJSPPVtotkFWfTJ6gBiAVKADgma//UswNDuh/rY1oEr7r8VDQ==","timestamp":"2023-01-09T12:33:17.856026446Z"} Jan 09 07:33:17 dante-rho-testing cosmovisor[815]: 7:33AM INF received complete proposal block hash=3CE4149360D3025FBA6BAE655D334E6D2982E810EC8A0D2B115743E9D52E69C8 height=13562550 module=consensus Jan 09 07:33:17 dante-rho-testing cosmovisor[815]: 7:33AM INF finalizing commit of block hash={} height=13562550 module=consensus num_txs=0 root=41B53AD1F6C969341C09D383586461ACE32EF3E41C76FA903A08AB75118EB8BA Jan 09 07:33:17 dante-rho-testing cosmovisor[815]: 7:33AM ERR UPGRADE "v8-Rho" NEEDED at height: 13562550: {"binaries":{"linux/amd64":"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-linux-amd64?checksum=sha256:db106075fe866601727dc67c81011fe0e05aa5aef5fbb15c7986ddb7a2bb4326","linux/arm64":"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-linux-arm64?checksum=sha256:a56923a14c6970c8a4978d8e256e1e50277738541c80dd3cfba8d3a8f64636ad","darwin/amd64":"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-darwin-amd64?checksum=sha256:0a56c72ba8e3fe27088cde309e1a821355f98ac6cf8b6e36b664ce1879aabb3b","darwin/arm64":"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-darwin-arm64?checksum=sha256:f11462f585d66cfd2061bf719b33142b9ec3324c17f478fb627299030a3c3c9e","windows/amd64":"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-windows-amd64.exe?checksum=sha256:75c8eb84c893a7ce02bb6e54722d9fe53dc912dceb34a43f0e4536cd000827b2"}} Jan 09 07:33:17 dante-rho-testing cosmovisor[815]: 7:33AM ERR CONSENSUS FAILURE!!! err="UPGRADE \"v8-Rho\" NEEDED at height: 13562550: {\"binaries\":{\"linux/amd64\":\"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-linux-amd64?checksum=sha256:db106075fe866601727dc67c81011fe0e05aa5aef5fbb15c7986ddb7a2bb4326\",\"linux/arm64\":\"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-linux-arm64?checksum=sha256:a56923a14c6970c8a4978d8e256e1e50277738541c80dd3cfba8d3a8f64636ad\",\"darwin/amd64\":\"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-darwin-amd64?checksum=sha256:0a56c72ba8e3fe27088cde309e1a821355f98ac6cf8b6e36b664ce1879aabb3b\",\"darwin/arm64\":\"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-darwin-arm64?checksum=sha256:f11462f585d66cfd2061bf719b33142b9ec3324c17f478fb627299030a3c3c9e\",\"windows/amd64\":\"https://github.com/cosmos/gaia/releases/download/v8.0.0-rc1/gaiad-v8.0.0-rc1-windows-amd64.exe?checksum=sha256:75c8eb84c893a7ce02bb6e54722d9fe53dc912dceb34a43f0e4536cd000827b2\"}}" module=consensus stack="goroutine 78 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:727 +0x4c\npanic({0x1932180, 0xc0b8522730})\n\truntime/panic.go:838 +0x207\ngithub.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0x7ffff73bfe26, 0x10}, 0xc000b80c60, {0x250e6f0, 0xc000bd44c0}, {0x252c5b8, 0xc0010f3780}, 0xc0011b24b0, {0x2508740, 0xc000e0c000}, ...}, ...)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.9/x/upgrade/abci.go:71 +0xf9f\ngithub.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.9/x/upgrade/module.go:130\ngithub.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x2521318, 0xc000136000}, {0x252dec0, 0xc0302ef840}, {{0xb, 0x0}, {0xc3c1c23060, 0xd}, 0xcef2b6, ...}, ...}, ...)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.9/types/module/module.go:491 +0x3bb\ngithub.com/cosmos/gaia/v7/app.(*GaiaApp).BeginBlocker(_, {{0x2521318, 0xc000136000}, {0x252dec0, 0xc0302ef840}, {{0xb, 0x0}, {0xc3c1c23060, 0xd}, 0xcef2b6, ...}, ...}, ...)\n\tgithub.com/cosmos/gaia/v7/app/app.go:707 +0x85\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc02efc9f60, 0x20, 0x20}, {{0xb, 0x0}, {0xc3c1c23060, 0xd}, 0xcef2b6, {0x32b9473f, ...}, ...}, ...})\n\tgithub.com/cosmos/cosmos-sdk@v0.45.9/baseapp/abci.go:177 +0x97c\ngithub.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc02efc9f60, 0x20, 0x20}, {{0xb, 0x0}, {0xc3c1c23060, 0xd}, 0xcef2b6, {0x32b9473f, ...}, ...}, ...})\n\tgithub.com/tendermint/tendermint@v0.34.21/abci/client/local_client.go:280 +0x118\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc02efc9f60, 0x20, 0x20}, {{0xb, 0x0}, {0xc3c1c23060, 0xd}, 0xcef2b6, {0x32b9473f, ...}, ...}, ...})\n\tgithub.com/tendermint/tendermint@v0.34.21/proxy/app_conn.go:81 +0x55\ngithub.com/tendermint/tendermint/state.execBlockOnProxyApp({0x25221c0?, 0xc30b3aefc0}, {0x25275a0, 0xc000ef0090}, 0xc0301e01e0, {0x252d748, 0xc00000ff38}, 0xcef2b5?)\n\tgithub.com/tendermint/tendermint@v0.34.21/state/execution.go:307 +0x3dd\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc30b3a8140, 0x8}}, {0xc30b3a8150, 0xd}, 0xcef203, 0xcef2b5, {{0xc02dccaa00, ...}, ...}, ...}, ...)\n\tgithub.com/tendermint/tendermint@v0.34.21/state/execution.go:140 +0x171\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc0d59ec000, 0xcef2b6)\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1659 +0xafd\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc0d59ec000, 0xcef2b6)\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1568 +0x2ff\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1()\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1503 +0x87\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit(0xc0d59ec000, 0xcef2b6, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1541 +0xcb7\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc0d59ec000, 0xc3fe8cdd60, {0x0, 0x0})\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:2155 +0xb7c\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc0d59ec000, 0xc3fe8cdd60, {0x0?, 0x47e566?})\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1953 +0x2c\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc0d59ec000, {{0x2509900?, 0xc02d6be7b8?}, {0x0?, 0x0?}})\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:856 +0x44b\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc0d59ec000, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:783 +0x512\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:379 +0x12d\n" Jan 09 07:33:17 dante-rho-testing cosmovisor[815]: 7:33AM INF service stop impl={"Logger":{}} module=consensus msg={} wal=/home/gaia/.gaia/data/cs.wal/wal Jan 09 07:33:17 dante-rho-testing cosmovisor[815]: 7:33AM INF service stop impl={"Dir":"/home/gaia/.gaia/data/cs.wal","Head":{"ID":"523wJp0kKQRK:/home/gaia/.gaia/data/cs.wal/wal","Path":"/home/gaia/.gaia/data/cs.wal/wal"},"ID":"group:523wJp0kKQRK:/home/gaia/.gaia/data/cs.wal/wal","Logger":{}} module=consensus msg={} wal=/home/gaia/.gaia/data/cs.wal/wal Jan 09 07:33:17 dante-rho-testing cosmovisor[675]: 7:33AM INF daemon shutting down in an attempt to restart module=cosmovisor Jan 09 07:33:17 dante-rho-testing cosmovisor[675]: 7:33AM INF no upgrade binary found, beginning to download it module=cosmovisor Jan 09 07:33:18 dante-rho-testing cosmovisor[675]: 7:33AM INF downloading binary complete module=cosmovisor Jan 09 07:33:18 dante-rho-testing cosmovisor[675]: 7:33AM INF pre-upgrade command does not exist. continuing the upgrade. module=cosmovisor Jan 09 07:33:18 dante-rho-testing cosmovisor[675]: 7:33AM INF upgrade detected, relaunching app=gaiad module=cosmovisor Jan 09 07:33:18 dante-rho-testing cosmovisor[675]: 7:33AM INF running app args=["start","--x-crisis-skip-assert-invariants","--home","/home/gaia/.gaia"] module=cosmovisor path=/home/gaia/.gaia/cosmovisor/upgrades/v8-rho/bin/gaiad Jan 09 07:33:19 dante-rho-testing cosmovisor[1725]: 7:33AM INF starting node with ABCI Tendermint in-process Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF service start impl=multiAppConn module=proxy msg={} Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF service start connection=query impl=localClient module=abci-client msg={} Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF service start connection=snapshot impl=localClient module=abci-client msg={} Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF service start connection=mempool impl=localClient module=abci-client msg={} Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF service start connection=consensus impl=localClient module=abci-client msg={} Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF service start impl=EventBus module=events msg={} Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF service start impl=PubSub module=pubsub msg={} Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF service start impl=IndexerService module=txindex msg={} Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF ABCI Handshake App Info hash="A�:��i4\x1c\tӃXda��.��\x1cv��:\b�u\x11��" height=13562549 module=consensus protocol-version=0 software-version=v8.0.0-rc1 Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF ABCI Replay Blocks appHeight=13562549 module=consensus stateHeight=13562549 storeHeight=13562550 Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF Replay last block using real app module=consensus Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF applying upgrade "v8-Rho" at height: 13562550 Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF start to run module migrations... Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF adding a new module: globalfee Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF adding a new module: packetforwardmiddleware Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF migrating module transfer from version 1 to version 2 Jan 09 07:33:35 dante-rho-testing cosmovisor[1725]: 7:33AM INF running the rest of the upgrade handler... Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: 7:33AM INF upgrade complete Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: 7:33AM INF minted coins from module account amount=34318633uatom from=mint module=x/bank Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: 7:33AM INF executed block height=13562550 module=consensus num_invalid_txs=0 num_valid_txs=0 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: panic: cannot delete latest saved version (1) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: goroutine 1 [running]: Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).PruneStores(0xc00103d110, 0x1, {0x0?, 0x0?, 0xc000f2cb40?}) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/cosmos-sdk@v0.45.11/store/rootmulti/store.go:460 +0x285 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc00103d110) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/cosmos-sdk@v0.45.11/store/rootmulti/store.go:429 +0x18b Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc000ddd880) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/cosmos-sdk@v0.45.11/baseapp/abci.go:314 +0x1e9 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc000cb1800) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint@v0.34.24/abci/client/local_client.go:264 +0xb6 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc00125e6b0?) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint@v0.34.24/proxy/app_conn.go:93 +0x22 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint/state.(*BlockExecutor).Commit(_, {{{0xb, 0x0}, {0xc00107f010, 0x8}}, {0xc00107f020, 0xd}, 0xcef203, 0xcef2b6, {{0xc000d80b20, ...}, ...}, ...}, ...) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint@v0.34.24/state/execution.go:228 +0x269 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc00107f010, 0x8}}, {0xc00107f020, 0xd}, 0xcef203, 0xcef2b6, {{0xc000d80b20, ...}, ...}, ...}, ...) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint@v0.34.24/state/execution.go:180 +0x6ee Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc00107f010, 0x8}}, {0xc00107f020, 0xd}, 0xcef203, 0xcef2b5, {{0xc00104a3a0, ...}, ...}, ...}, ...) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:503 +0x23c Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc00107f010, 0x8}}, {0xc00107f020, 0xd}, 0xcef203, 0xcef2b5, {{0xc00104a3a0, ...}, ...}, ...}, ...) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:416 +0x7ae Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc00125fd90, {0x2563c28, 0xc0001bc000}) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:268 +0x3c8 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc00107f010, 0x8}}, {0xc00107f020, 0xd}, 0xcef203, 0xcef2b5, ...}, ...) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint@v0.34.24/node/node.go:329 +0x1b8 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint/node.NewNode(0xc0005ea640, {0x2553620, 0xc0005994a0}, 0xc0005968b0, {0x253e9e0, 0xc000fd4c90}, 0x0?, 0x0?, 0xc000596d00, {0x2557600, ...}, ...) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/tendermint/tendermint@v0.34.24/node/node.go:777 +0x597 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x256a558, 0xc0001db320}, {0xc000122b10, 0xe}, {0x255a888, 0xc001047a10}, ...}, ...) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/cosmos-sdk@v0.45.11/server/start.go:280 +0x7db Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc0010c6900?, {0xc000509f50?, 0x0?, 0x3?}) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/cosmos-sdk@v0.45.11/server/start.go:128 +0x169 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/spf13/cobra.(*Command).execute(0xc0010c6900, {0xc000509ec0, 0x3, 0x3}) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/spf13/cobra@v1.6.1/command.go:916 +0x862 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/spf13/cobra.(*Command).ExecuteC(0xc000ffcc00) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/spf13/cobra.(*Command).Execute(...) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/spf13/cobra@v1.6.1/command.go:968 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/spf13/cobra.(*Command).ExecuteContext(...) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/spf13/cobra@v1.6.1/command.go:961 Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc00107e5d0, 0x10}) Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/cosmos-sdk@v0.45.11/server/cmd/execute.go:36 +0x1eb Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: main.main() Jan 09 07:33:36 dante-rho-testing cosmovisor[1725]: github.com/cosmos/gaia/v8/cmd/gaiad/main.go:16 +0x2c Jan 09 07:33:36 dante-rho-testing cosmovisor[675]: Error: exit status 2 Jan 09 07:33:36 dante-rho-testing cosmovisor[675]: 7:33AM ERR error="exit status 2" module=cosmovisor Jan 09 07:33:36 dante-rho-testing systemd[1]: cosmovisor.service: Main process exited, code=exited, status=1/FAILURE Jan 09 07:33:36 dante-rho-testing systemd[1]: cosmovisor.service: Failed with result 'exit-code'. Jan 09 07:33:36 dante-rho-testing systemd[1]: cosmovisor.service: Consumed 24min 28.365s CPU time. ``` ## Start from `v7.1.0 with Jan 6 export, upgrade to upgrade-debug` - Result: **FAILING** - Run on: 2023-01-09 09:37 - Starting from `v7.1.0` - Account `cosmos13dqvh4qtg4gzczuktgnw8gc2ewnwmhdwnctekxctyr4azz4dcyysecgq7e` has a balance of `3460036989uatom` - Upgrading to commit `7df84e24108293fe50d5465850d55fa57e300626` (upgrade-debug branch) - Upgrade at block height 13563390 - Single validator - Genesis file: https://files.polypore.xyz/genesis/mainnet-genesis-tinkered/tinkered-genesis_2023-01-06T20%3A22%3A50.460851274Z_v7.1.0_13562370.json.gz - Error message: `panic: cannot delete latest saved version (1)` ``` Jan 09 09:37:07 dante-rho-testing systemd[1]: Started cosmovisor. Jan 09 09:37:07 dante-rho-testing cosmovisor[2922]: 9:37AM INF running app args=["start","--x-crisis-skip-assert-invariants","--home","/home/gaia/.gaia"] module=cosmovisor path=/home/gaia/.gaia/cosmovisor/genesis/bin/gaiad Jan 09 09:37:07 dante-rho-testing cosmovisor[2931]: 9:37AM INF starting node with ABCI Tendermint in-process Jan 09 09:37:08 dante-rho-testing cosmovisor[2922]: 9:37AM INF daemon shutting down in an attempt to restart module=cosmovisor Jan 09 09:37:08 dante-rho-testing cosmovisor[2922]: 9:37AM INF pre-upgrade command does not exist. continuing the upgrade. module=cosmovisor Jan 09 09:37:08 dante-rho-testing cosmovisor[2922]: 9:37AM INF upgrade detected, relaunching app=gaiad module=cosmovisor Jan 09 09:37:08 dante-rho-testing cosmovisor[2922]: 9:37AM INF running app args=["start","--x-crisis-skip-assert-invariants","--home","/home/gaia/.gaia"] module=cosmovisor path=/home/gaia/.gaia/cosmovisor/upgrades/v8-rho/bin/gaiad Jan 09 09:37:08 dante-rho-testing cosmovisor[2947]: 9:37AM INF starting node with ABCI Tendermint in-process Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF service start impl=multiAppConn module=proxy msg={} Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF service start connection=query impl=localClient module=abci-client msg={} Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF service start connection=snapshot impl=localClient module=abci-client msg={} Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF service start connection=mempool impl=localClient module=abci-client msg={} Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF service start connection=consensus impl=localClient module=abci-client msg={} Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF service start impl=EventBus module=events msg={} Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF service start impl=PubSub module=pubsub msg={} Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF service start impl=IndexerService module=txindex msg={} Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF ABCI Handshake App Info hash="��\r.z$\x19\a�ӵ��Ԙ\vʎ.��Vn��\b\x1a��" height=13563389 module=consensus protocol-version=0 software-version=HEAD-7df84e24108293fe50d5465850d55fa57e300626 Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF ABCI Replay Blocks appHeight=13563389 module=consensus stateHeight=13563389 storeHeight=13563390 Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF Replay last block using real app module=consensus Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF applying upgrade "v8-Rho" at height: 13563390 Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF start to run module migrations... Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF running fix-bank-metadata Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF Found correct: true Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF Atom metadata: description:"The native staking token of the Cosmos Hub." denom_units:<denom:"uatom" aliases:"microatom" > denom_units:<denom:"matom" exponent:3 aliases:"milliatom" > denom_units:<denom:"atom" exponent:6 > base:"uatom" display:"atom" Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF running fix-quicksilver Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF update ICA params Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF running the rest of the upgrade handler... Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF adding a new module: globalfee Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF adding a new module: packetforwardmiddleware Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF migrating module transfer from version 1 to version 2 Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF upgrade complete Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF minted coins from module account amount=34316249uatom from=mint module=x/bank Jan 09 09:37:24 dante-rho-testing cosmovisor[2947]: 9:37AM INF executed block height=13563390 module=consensus num_invalid_txs=0 num_valid_txs=0 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: panic: cannot delete latest saved version (1) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: goroutine 1 [running]: Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).PruneStores(0xc000d7aea0, 0x1, {0x0?, 0x0?, 0xc000efb400?}) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/cosmos-sdk@v0.45.11/store/rootmulti/store.go:460 +0x285 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc000d7aea0) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/cosmos-sdk@v0.45.11/store/rootmulti/store.go:429 +0x18b Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc000d6cfc0) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/cosmos-sdk@v0.45.11/baseapp/abci.go:314 +0x1e9 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc001668720) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint@v0.34.24/abci/client/local_client.go:264 +0xb6 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc0011b46b0?) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint@v0.34.24/proxy/app_conn.go:93 +0x22 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint/state.(*BlockExecutor).Commit(_, {{{0xb, 0x0}, {0xc0010d4030, 0x8}}, {0xc0010d46b0, 0xd}, 0xcef203, 0xcef5fe, {{0xc000fd2960, ...}, ...}, ...}, ...) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint@v0.34.24/state/execution.go:228 +0x269 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0010d4030, 0x8}}, {0xc0010d46b0, 0xd}, 0xcef203, 0xcef5fe, {{0xc000fd2960, ...}, ...}, ...}, ...) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint@v0.34.24/state/execution.go:180 +0x6ee Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc0010d4030, 0x8}}, {0xc0010d46b0, 0xd}, 0xcef203, 0xcef5fd, {{0xc001de4000, ...}, ...}, ...}, ...) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:503 +0x23c Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc0010d4030, 0x8}}, {0xc0010d46b0, 0xd}, 0xcef203, 0xcef5fd, {{0xc001de4000, ...}, ...}, ...}, ...) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:416 +0x7ae Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc0011b5d90, {0x2566348, 0xc000666340}) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:268 +0x3c8 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc0010d4030, 0x8}}, {0xc0010d46b0, 0xd}, 0xcef203, 0xcef5fd, ...}, ...) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint@v0.34.24/node/node.go:329 +0x1b8 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint/node.NewNode(0xc0010a3400, {0x2555d40, 0xc000cef360}, 0xc000d29160, {0x25410c0, 0xc001de8a68}, 0x0?, 0x0?, 0xc000d293b0, {0x2559d20, ...}, ...) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/tendermint/tendermint@v0.34.24/node/node.go:777 +0x597 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x256cc78, 0xc000ccc9c0}, {0xc0000524c0, 0xe}, {0x255cfa8, 0xc000d29730}, ...}, ...) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/cosmos-sdk@v0.45.11/server/start.go:280 +0x7db Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc0000c4300?, {0xc000ccc360?, 0x0?, 0x3?}) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/cosmos-sdk@v0.45.11/server/start.go:128 +0x169 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/spf13/cobra.(*Command).execute(0xc0000c4300, {0xc000ccc2d0, 0x3, 0x3}) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/spf13/cobra@v1.6.1/command.go:916 +0x862 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/spf13/cobra.(*Command).ExecuteC(0xc000ffcc00) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/spf13/cobra.(*Command).Execute(...) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/spf13/cobra@v1.6.1/command.go:968 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/spf13/cobra.(*Command).ExecuteContext(...) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/spf13/cobra@v1.6.1/command.go:961 Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc000cfcbc0, 0x10}) Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/cosmos-sdk@v0.45.11/server/cmd/execute.go:36 +0x1eb Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: main.main() Jan 09 09:37:25 dante-rho-testing cosmovisor[2947]: github.com/cosmos/gaia/v8/cmd/gaiad/main.go:16 +0x2c Jan 09 09:37:25 dante-rho-testing cosmovisor[2922]: Error: exit status 2 Jan 09 09:37:25 dante-rho-testing cosmovisor[2922]: 9:37AM ERR error="exit status 2" module=cosmovisor Jan 09 09:37:25 dante-rho-testing systemd[1]: cosmovisor.service: Main process exited, code=exited, status=1/FAILURE Jan 09 09:37:25 dante-rho-testing systemd[1]: cosmovisor.service: Failed with result 'exit-code'. Jan 09 09:37:25 dante-rho-testing systemd[1]: cosmovisor.service: Consumed 20.234s CPU time. ``` ## Start from `v7.1.0 with Jan 6 export, upgrade to upgrade-only` - Result: **FAILING** - Run on: 2023-01-09 11:37 - Starting from `v7.1.0` - Account `cosmos13dqvh4qtg4gzczuktgnw8gc2ewnwmhdwnctekxctyr4azz4dcyysecgq7e` has a balance of `3460036989uatom` - Upgrading to commit `b1bc14f2dc1fa0dcf2b3abb995b1b0a1faa92a5d` (upgrade-only branch) - Upgrade at block height 13562450 - Single validator - Genesis file: https://files.polypore.xyz/genesis/mainnet-genesis-tinkered/tinkered-genesis_2023-01-06T20%3A22%3A50.460851274Z_v7.1.0_13562370.json.gz - Error message: `panic: cannot delete latest saved version (1)` ``` ration":4859793230,"height":13562450,"round":0,"step":1} old_ti={"duration":3000000000,"height":13562449,"round":0,"step":3} Jan 09 11:37:18 dante-rho-testing cosmovisor[771]: 11:37AM DBG Attempt to update stats for non-existent peer module=consensus peer= Jan 09 11:37:18 dante-rho-testing cosmovisor[771]: 11:37AM DBG Scheduled timeout dur=4859.79323 height=13562450 module=consensus round=0 step=1 Jan 09 11:37:18 dante-rho-testing cosmovisor[771]: 11:37AM INF indexed block height=13562449 module=txindex Jan 09 11:37:18 dante-rho-testing cosmovisor[771]: 11:37AM DBG indexed block txs height=13562449 module=txindex num_txs=0 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM INF Timed out dur=4859.79323 height=13562450 module=consensus round=0 step=1 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG received tock height=13562450 module=consensus round=0 step=1 timeout=4859.79323 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG entering new round current={} height=13562450 module=consensus round=0 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG entering propose step current={} height=13562450 module=consensus round=0 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG node is a validator height=13562450 module=consensus round=0 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG propose step; our turn to propose height=13562450 module=consensus proposer=973C48DF8B3356C45E44494723A6E0D45DEB8131 round=0 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG Received tick module=consensus new_ti={"duration":3000000000,"height":13562450,"round":0,"step":3} old_ti={"duration":4859793230,"height":13562450,"round":0,"step":1} Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG Timer already stopped module=consensus Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG Scheduled timeout dur=3000 height=13562450 module=consensus round=0 step=3 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG signed proposal height=13562450 module=consensus proposal={"Type":32,"block_id":{"hash":"873FA8C8FA40D13FC7AF75993CE6C9BE1513329278254E36654F0CDB6AAF59EC","parts":{"hash":"91536E2A59B24585068FA408E72C9DCE2395798AE68BE78640A9190AD35E493F","total":1}},"height":13562450,"pol_round":-1,"round":0,"signature":"MunIN/CqrDC1Ijnta1K9mqnn8RuiZEnVtLVic+Kn1OUWcP2mejl+Gkm0tlivLl2OBC8xAxPJoOpPNe9+FIL/Dg==","timestamp":"2023-01-09T16:37:22.945240317Z"} round=0 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG Broadcast channel=32 module=p2p msgBytes={} Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"873FA8C8FA40D13FC7AF75993CE6C9BE1513329278254E36654F0CDB6AAF59EC","parts":{"hash":"91536E2A59B24585068FA408E72C9DCE2395798AE68BE78640A9190AD35E493F","total":1}},"height":13562450,"pol_round":-1,"round":0,"signature":"MunIN/CqrDC1Ijnta1K9mqnn8RuiZEnVtLVic+Kn1OUWcP2mejl+Gkm0tlivLl2OBC8xAxPJoOpPNe9+FIL/Dg==","timestamp":"2023-01-09T16:37:22.945240317Z"} Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM INF received complete proposal block hash=873FA8C8FA40D13FC7AF75993CE6C9BE1513329278254E36654F0CDB6AAF59EC height=13562450 module=consensus Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG entering prevote step current={} height=13562450 module=consensus round=0 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG prevote step: ProposalBlock is valid height=13562450 module=consensus round=0 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG signed and pushed vote height=13562450 module=consensus round=0 vote={"block_id":{"hash":"873FA8C8FA40D13FC7AF75993CE6C9BE1513329278254E36654F0CDB6AAF59EC","parts":{"hash":"91536E2A59B24585068FA408E72C9DCE2395798AE68BE78640A9190AD35E493F","total":1}},"height":13562450,"round":0,"signature":"ehtWq4s1+2N9UDm+Etqe3U7UdeuqaqpNwOa/ctl2P29CDAsVbusklBX7SVpcTnpLajC5EuZT+9oQf11g09FCBw==","timestamp":"2023-01-09T16:37:22.951945219Z","type":1,"validator_address":"973C48DF8B3356C45E44494723A6E0D45DEB8131","validator_index":0} Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG Broadcast channel=32 module=p2p msgBytes={} Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG Attempt to update stats for non-existent peer module=consensus peer= Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG adding vote cs_height=13562450 module=consensus val_index=0 vote_height=13562450 vote_type=1 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG Broadcast channel=32 module=p2p msgBytes={} Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG added vote to prevote module=consensus prevotes="VoteSet{H:13562450 R:0 T:SIGNED_MSG_TYPE_PREVOTE +2/3:873FA8C8FA40D13FC7AF75993CE6C9BE1513329278254E36654F0CDB6AAF59EC:1:91536E2A59B2(0.744420371038354) BA{175:x______________________________________________________________________________________________________________________________________________________________________________} map[]}" vote={"block_id":{"hash":"873FA8C8FA40D13FC7AF75993CE6C9BE1513329278254E36654F0CDB6AAF59EC","parts":{"hash":"91536E2A59B24585068FA408E72C9DCE2395798AE68BE78640A9190AD35E493F","total":1}},"height":13562450,"round":0,"signature":"ehtWq4s1+2N9UDm+Etqe3U7UdeuqaqpNwOa/ctl2P29CDAsVbusklBX7SVpcTnpLajC5EuZT+9oQf11g09FCBw==","timestamp":"2023-01-09T16:37:22.951945219Z","type":1,"validator_address":"973C48DF8B3356C45E44494723A6E0D45DEB8131","validator_index":0} Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG updating valid block because of POL module=consensus pol_round=0 valid_round=-1 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG Broadcast channel=32 module=p2p msgBytes={} Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG entering precommit step current={} height=13562450 module=consensus round=0 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG precommit step; +2/3 prevoted proposal block; locking hash=873FA8C8FA40D13FC7AF75993CE6C9BE1513329278254E36654F0CDB6AAF59EC height=13562450 module=consensus round=0 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG signed and pushed vote height=13562450 module=consensus round=0 vote={"block_id":{"hash":"873FA8C8FA40D13FC7AF75993CE6C9BE1513329278254E36654F0CDB6AAF59EC","parts":{"hash":"91536E2A59B24585068FA408E72C9DCE2395798AE68BE78640A9190AD35E493F","total":1}},"height":13562450,"round":0,"signature":"8sClMG7P9Z1Izo7Y9L9u9GAmLIZoUatVvqtF9XhWhAMimJz5ktQ1w71A6nBeOVNaw7YmTYdu5hMzzwTHXeimCg==","timestamp":"2023-01-09T16:37:22.956902551Z","type":2,"validator_address":"973C48DF8B3356C45E44494723A6E0D45DEB8131","validator_index":0} Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG Broadcast channel=32 module=p2p msgBytes={} Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG Attempt to update stats for non-existent peer module=consensus peer= Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG adding vote cs_height=13562450 module=consensus val_index=0 vote_height=13562450 vote_type=2 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG Broadcast channel=32 module=p2p msgBytes={} Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG added vote to precommit data=Votes:560365843/752754579(0.744) height=13562450 module=consensus round=0 validator=973C48DF8B3356C45E44494723A6E0D45DEB8131 vote_timestamp=2023-01-09T16:37:22Z Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG entering new round with invalid args current={} height=13562450 module=consensus round=0 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG entering precommit step with invalid args current={} height=13562450 module=consensus round=0 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG entering commit step commit_round=0 current={} height=13562450 module=consensus Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG commit is for a locked block; set ProposalBlock=LockedBlock block_hash=873FA8C8FA40D13FC7AF75993CE6C9BE1513329278254E36654F0CDB6AAF59EC commit_round=0 height=13562450 module=consensus Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG Broadcast channel=32 module=p2p msgBytes={} Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM INF finalizing commit of block hash={} height=13562450 module=consensus num_txs=0 root=68B21D206D12F9E7B0F49EA265B147B7F7C1EEB994DAADE275E3E882C2BCA312 Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM DBG committed block block={} height=13562450 module=consensus Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM ERR UPGRADE "v8-Rho" NEEDED at height: 13562450: Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM ERR CONSENSUS FAILURE!!! err="UPGRADE \"v8-Rho\" NEEDED at height: 13562450: " module=consensus stack="goroutine 116 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:727 +0x4c\npanic({0x1932180, 0xc159346670})\n\truntime/panic.go:838 +0x207\ngithub.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0x7ffeae89de25, 0x10}, 0xc000f820f0, {0x250e6f0, 0xc000f8b740}, {0x252c5b8, 0xc000511bf0}, 0xc000542120, {0x2508740, 0xc000da1500}, ...}, ...)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.9/x/upgrade/abci.go:71 +0xf9f\ngithub.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.9/x/upgrade/module.go:130\ngithub.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x2521318, 0xc00012e000}, {0x252dec0, 0xc01e132f80}, {{0xb, 0x0}, {0xc0533571d0, 0xd}, 0xcef252, ...}, ...}, ...)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.9/types/module/module.go:491 +0x3bb\ngithub.com/cosmos/gaia/v7/app.(*GaiaApp).BeginBlocker(_, {{0x2521318, 0xc00012e000}, {0x252dec0, 0xc01e132f80}, {{0xb, 0x0}, {0xc0533571d0, 0xd}, 0xcef252, ...}, ...}, ...)\n\tgithub.com/cosmos/gaia/v7/app/app.go:707 +0x85\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc043afe640, 0x20, 0x20}, {{0xb, 0x0}, {0xc0533571d0, 0xd}, 0xcef252, {0x37e8c0a6, ...}, ...}, ...})\n\tgithub.com/cosmos/cosmos-sdk@v0.45.9/baseapp/abci.go:177 +0x97c\ngithub.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc043afe640, 0x20, 0x20}, {{0xb, 0x0}, {0xc0533571d0, 0xd}, 0xcef252, {0x37e8c0a6, ...}, ...}, ...})\n\tgithub.com/tendermint/tendermint@v0.34.21/abci/client/local_client.go:280 +0x118\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc043afe640, 0x20, 0x20}, {{0xb, 0x0}, {0xc0533571d0, 0xd}, 0xcef252, {0x37e8c0a6, ...}, ...}, ...})\n\tgithub.com/tendermint/tendermint@v0.34.21/proxy/app_conn.go:81 +0x55\ngithub.com/tendermint/tendermint/state.execBlockOnProxyApp({0x25221c0?, 0xc056692480}, {0x25275a0, 0xc0010611c0}, 0xc01e090000, {0x252d748, 0xc000fd25a0}, 0xcef251?)\n\tgithub.com/tendermint/tendermint@v0.34.21/state/execution.go:307 +0x3dd\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc175a42e70, 0x8}}, {0xc175a42e90, 0xd}, 0xcef203, 0xcef251, {{0xc01ff0d420, ...}, ...}, ...}, ...)\n\tgithub.com/tendermint/tendermint@v0.34.21/state/execution.go:140 +0x171\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc0ac728000, 0xcef252)\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1659 +0xafd\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc0ac728000, 0xcef252)\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1568 +0x2ff\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1()\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1503 +0x87\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit(0xc0ac728000, 0xcef252, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1541 +0xcb7\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc0ac728000, 0xc4ef220280, {0x0, 0x0})\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:2155 +0xb7c\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc0ac728000, 0xc4ef220280, {0x0?, 0x47e566?})\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:1953 +0x2c\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc0ac728000, {{0x2509900?, 0xc01ae33718?}, {0x0?, 0x0?}})\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:856 +0x44b\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc0ac728000, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:783 +0x512\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\tgithub.com/tendermint/tendermint@v0.34.21/consensus/state.go:379 +0x12d\n" Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM INF service stop impl={"Logger":{}} module=consensus msg={} wal=/home/gaia/.gaia/data/cs.wal/wal Jan 09 11:37:22 dante-rho-testing cosmovisor[771]: 11:37AM INF service stop impl={"Dir":"/home/gaia/.gaia/data/cs.wal","Head":{"ID":"z3jKIFv5Kscp:/home/gaia/.gaia/data/cs.wal/wal","Path":"/home/gaia/.gaia/data/cs.wal/wal"},"ID":"group:z3jKIFv5Kscp:/home/gaia/.gaia/data/cs.wal/wal","Logger":{}} module=consensus msg={} wal=/home/gaia/.gaia/data/cs.wal/wal Jan 09 11:37:23 dante-rho-testing cosmovisor[685]: 11:37AM INF Daemon shutting down in an attempt to restart module=cosmovisor Jan 09 11:37:23 dante-rho-testing cosmovisor[685]: 11:37AM INF pre-upgrade command does not exist. continuing the upgrade. module=cosmovisor Jan 09 11:37:23 dante-rho-testing cosmovisor[685]: 11:37AM INF upgrade detected, relaunching app=gaiad module=cosmovisor Jan 09 11:37:23 dante-rho-testing cosmovisor[685]: 11:37AM INF running app args=["start","--x-crisis-skip-assert-invariants","--home","/home/gaia/.gaia"] module=cosmovisor path=/home/gaia/.gaia/cosmovisor/upgrades/v8-Rho/bin/gaiad Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: Error: failed to initialize database: resource temporarily unavailable Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: Usage: Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: gaiad start [flags] Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: Flags: Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --abci string specify abci transport (socket | grpc) (default "socket") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --address string Listen address (default "tcp://0.0.0.0:26658") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --consensus.create_empty_blocks set this to false to only produce blocks when there are txs or when the AppHash changes (default true) Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --consensus.create_empty_blocks_interval string the possible interval between empty blocks (default "0s") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --consensus.double_sign_check_height int how many blocks to look back to check existence of the node's consensus votes before joining consensus Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --cpu-profile string Enable CPU profiling and write to the provided file Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --db_backend string database backend: goleveldb | cleveldb | boltdb | rocksdb | badgerdb (default "goleveldb") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --db_dir string database directory (default "data") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --fast_sync fast blockchain syncing (default true) Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --genesis_hash bytesHex optional SHA-256 hash of the genesis file Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --grpc-only Start the node in gRPC query only mode (no Tendermint process is started) Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --grpc-web.address string The gRPC-Web server address to listen on (default "0.0.0.0:9091") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --grpc-web.enable Define if the gRPC-Web server should be enabled. (Note: gRPC must also be enabled.) (default true) Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --grpc.address string the gRPC server address to listen on (default "0.0.0.0:9090") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --grpc.enable Define if the gRPC server should be enabled (default true) Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --halt-height uint Block height at which to gracefully halt the chain and shutdown the node Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --halt-time uint Minimum block time (in Unix seconds) at which to gracefully halt the chain and shutdown the node Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: -h, --help help for start Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --home string The application home directory (default "/home/gaia/.gaia") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --iavl-disable-fastnode Disable fast node for IAVL tree (default true) Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --inter-block-cache Enable inter-block caching (default true) Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --inv-check-period uint Assert registered invariants every N blocks Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --min-retain-blocks uint Minimum block height offset during ABCI commit to prune Tendermint blocks Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --minimum-gas-prices string Minimum gas prices to accept for transactions; Any fee in a tx must meet this minimum (e.g. 0.01photino;0.0001stake) Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --moniker string node name (default "dante-rho-testing") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --p2p.external-address string ip:port address to advertise to peers for them to dial Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --p2p.laddr string node listen address. (0.0.0.0:0 means any interface, any port) (default "tcp://0.0.0.0:26656") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --p2p.persistent_peers string comma-delimited ID@host:port persistent peers Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --p2p.pex enable/disable Peer-Exchange (default true) Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --p2p.private_peer_ids string comma-delimited private peer IDs Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --p2p.seed_mode enable/disable seed mode Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --p2p.seeds string comma-delimited ID@host:port seed nodes Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --p2p.unconditional_peer_ids string comma-delimited IDs of unconditional peers Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --p2p.upnp enable/disable UPNP port forwarding Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --priv_validator_laddr string socket address to listen on for connections from external priv_validator process Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --proxy_app string proxy app address, or one of: 'kvstore', 'persistent_kvstore', 'counter', 'e2e' or 'noop' for local testing. (default "tcp://127.0.0.1:26658") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --pruning string Pruning strategy (default|nothing|everything|custom) (default "default") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --pruning-interval uint Height interval at which pruned heights are removed from disk (ignored if pruning is not 'custom') Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --pruning-keep-every uint Offset heights to keep on disk after 'keep-every' (ignored if pruning is not 'custom') Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --pruning-keep-recent uint Number of recent heights to keep on disk (ignored if pruning is not 'custom') Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --rpc.grpc_laddr string GRPC listen address (BroadcastTx only). Port required Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --rpc.laddr string RPC listen address. Port required (default "tcp://127.0.0.1:26657") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --rpc.pprof_laddr string pprof listen address (https://golang.org/pkg/net/http/pprof) Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --rpc.unsafe enabled unsafe rpc methods Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --state-sync.snapshot-interval uint State sync snapshot interval Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --state-sync.snapshot-keep-recent uint32 State sync snapshot to keep (default 2) Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --trace Provide full stack traces for errors in ABCI Log Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --trace-store string Enable KVStore tracing to an output file Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --transport string Transport protocol: socket, grpc (default "socket") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --unsafe-skip-upgrades ints Skip a set of upgrade heights to continue the old binary Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --with-tendermint Run abci app embedded in-process with tendermint (default true) Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --x-crisis-skip-assert-invariants Skip x/crisis invariants check on startup Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: Global Flags: Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --log_format string The logging format (json|plain) (default "plain") Jan 09 11:37:23 dante-rho-testing cosmovisor[1407]: --log_level string The logging level (trace|debug|info|warn|error|fatal|panic) (default "info") Jan 09 11:37:23 dante-rho-testing cosmovisor[685]: 11:37AM ERR error="exit status 1" module=cosmovisor Jan 09 11:37:23 dante-rho-testing systemd[1]: cosmovisor.service: Main process exited, code=exited, status=1/FAILURE Jan 09 11:37:23 dante-rho-testing systemd[1]: cosmovisor.service: Killing process 1073 (gaiad) with signal SIGKILL. Jan 09 11:37:23 dante-rho-testing systemd[1]: cosmovisor.service: Failed with result 'exit-code'. Jan 09 11:37:23 dante-rho-testing systemd[1]: cosmovisor.service: Consumed 19min 43.785s CPU time. Jan 09 11:37:26 dante-rho-testing systemd[1]: cosmovisor.service: Scheduled restart job, restart counter is at 1. Jan 09 11:37:26 dante-rho-testing systemd[1]: Stopped cosmovisor. Jan 09 11:37:26 dante-rho-testing systemd[1]: cosmovisor.service: Consumed 19min 43.785s CPU time. Jan 09 11:37:26 dante-rho-testing systemd[1]: Started cosmovisor. Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: 11:37AM INF Configuration is valid: Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: Configurable Values: Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: DAEMON_HOME: /home/gaia/.gaia Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: DAEMON_NAME: gaiad Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: DAEMON_ALLOW_DOWNLOAD_BINARIES: false Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: DAEMON_RESTART_AFTER_UPGRADE: true Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: DAEMON_POLL_INTERVAL: 300ms Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: UNSAFE_SKIP_BACKUP: true Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: DAEMON_PREUPGRADE_MAX_RETRIES: 0 Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: Derived Values: Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: Root Dir: /home/gaia/.gaia/cosmovisor Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: Upgrade Dir: /home/gaia/.gaia/cosmovisor/upgrades Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: Genesis Bin: /home/gaia/.gaia/cosmovisor/genesis/bin/gaiad Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: Monitored File: /home/gaia/.gaia/data/upgrade-info.json Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: module=cosmovisor Jan 09 11:37:26 dante-rho-testing cosmovisor[1419]: 11:37AM INF running app args=["start","--x-crisis-skip-assert-invariants","--home","/home/gaia/.gaia"] module=cosmovisor path=/home/gaia/.gaia/cosmovisor/upgrades/v8-Rho/bin/gaiad Jan 09 11:37:26 dante-rho-testing cosmovisor[1425]: 11:37AM INF starting node with ABCI Tendermint in-process Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF service start impl=multiAppConn module=proxy msg={} Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF service start connection=query impl=localClient module=abci-client msg={} Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF service start connection=snapshot impl=localClient module=abci-client msg={} Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF service start connection=mempool impl=localClient module=abci-client msg={} Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF service start connection=consensus impl=localClient module=abci-client msg={} Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF service start impl=EventBus module=events msg={} Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF service start impl=PubSub module=pubsub msg={} Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF service start impl=IndexerService module=txindex msg={} Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF ABCI Handshake App Info hash="h�\x1d m\x12��e�G��\uee54ڭ�u��¼�\x12" height=13562449 module=consensus protocol-version=0 software-version=HEAD-b1bc14f2dc1fa0dcf2b3abb995b1b0a1faa92a5d Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF ABCI Replay Blocks appHeight=13562449 module=consensus stateHeight=13562449 storeHeight=13562450 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF Replay last block using real app module=consensus Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF applying upgrade "v8-Rho" at height: 13562450 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF start to run module migrations... Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF adding a new module: globalfee Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF adding a new module: packetforwardmiddleware Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF migrating module transfer from version 1 to version 2 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF upgrade complete Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF minted coins from module account amount=34318916uatom from=mint module=x/bank Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM DBG absent validator height=13562450 missed=90 module=x/slashing threshold=500 validator=cosmosvalcons14sk4vptumprktehmuvvf0yynarjy4gv08t64t4 ... Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM DBG absent validator height=13562450 missed=81 module=x/slashing threshold=500 validator=cosmosvalcons1kgekmjr2wjn0s4fd0a5x4syc8m6wpvxwp9kvln Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: 11:37AM INF executed block height=13562450 module=consensus num_invalid_txs=0 num_valid_txs=0 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: panic: cannot delete latest saved version (1) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: goroutine 1 [running]: Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).PruneStores(0xc0011191e0, 0x1, {0x0?, 0x0?, 0xc0011e8700?}) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/cosmos-sdk@v0.45.11/store/rootmulti/store.go:460 +0x285 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc0011191e0) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/cosmos-sdk@v0.45.11/store/rootmulti/store.go:429 +0x18b Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc000571880) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/cosmos-sdk@v0.45.11/baseapp/abci.go:314 +0x1e9 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc0010baa20) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint@v0.34.24/abci/client/local_client.go:264 +0xb6 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc0012526b0?) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint@v0.34.24/proxy/app_conn.go:93 +0x22 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint/state.(*BlockExecutor).Commit(_, {{{0xb, 0x0}, {0xc000c08b50, 0x8}}, {0xc000c08b60, 0xd}, 0xcef203, 0xcef252, {{0xc0010beb60, ...}, ...}, ...}, ...) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint@v0.34.24/state/execution.go:228 +0x269 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc000c08b50, 0x8}}, {0xc000c08b60, 0xd}, 0xcef203, 0xcef252, {{0xc0010beb60, ...}, ...}, ...}, ...) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint@v0.34.24/state/execution.go:180 +0x6ee Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc000c08b50, 0x8}}, {0xc000c08b60, 0xd}, 0xcef203, 0xcef251, {{0xc000fac360, ...}, ...}, ...}, ...) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:503 +0x23c Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc000c08b50, 0x8}}, {0xc000c08b60, 0xd}, 0xcef203, 0xcef251, {{0xc000fac360, ...}, ...}, ...}, ...) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:416 +0x7ae Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc001253d90, {0x2564928, 0xc0000c61a0}) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:268 +0x3c8 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc000c08b50, 0x8}}, {0xc000c08b60, 0xd}, 0xcef203, 0xcef251, ...}, ...) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint@v0.34.24/node/node.go:329 +0x1b8 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint/node.NewNode(0xc00111f180, {0x2554320, 0xc00141f400}, 0xc0001b5f60, {0x253f6a0, 0xc000fc2288}, 0x0?, 0x0?, 0xc001120540, {0x2558300, ...}, ...) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/tendermint/tendermint@v0.34.24/node/node.go:777 +0x597 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x256b258, 0xc000c00bd0}, {0xc000053520, 0xe}, {0x255b588, 0xc0010e36b0}, ...}, ...) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/cosmos-sdk@v0.45.11/server/start.go:280 +0x7db Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc000d94300?, {0xc000beecc0?, 0x0?, 0x3?}) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/cosmos-sdk@v0.45.11/server/start.go:128 +0x169 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/spf13/cobra.(*Command).execute(0xc000d94300, {0xc000bee9f0, 0x3, 0x3}) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/spf13/cobra@v1.6.1/command.go:916 +0x862 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/spf13/cobra.(*Command).ExecuteC(0xc001096c00) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/spf13/cobra.(*Command).Execute(...) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/spf13/cobra@v1.6.1/command.go:968 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/spf13/cobra.(*Command).ExecuteContext(...) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/spf13/cobra@v1.6.1/command.go:961 Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc0010cd400, 0x10}) Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/cosmos-sdk@v0.45.11/server/cmd/execute.go:36 +0x1eb Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: main.main() Jan 09 11:37:43 dante-rho-testing cosmovisor[1425]: github.com/cosmos/gaia/v8/cmd/gaiad/main.go:16 +0x2c Jan 09 11:37:43 dante-rho-testing cosmovisor[1419]: 11:37AM ERR error="exit status 2" module=cosmovisor Jan 09 11:37:43 dante-rho-testing systemd[1]: cosmovisor.service: Main process exited, code=exited, status=1/FAILURE Jan 09 11:37:43 dante-rho-testing systemd[1]: cosmovisor.service: Failed with result 'exit-code'. Jan 09 11:37:43 dante-rho-testing systemd[1]: cosmovisor.service: Consumed 19.054s CPU time. ```