---
tags: rho-upgrade
---
# Rho Upgrade Testing
## 2022-10-04
* Result : **FAILING**
* Local stateful upgrade
* Genesis used: `https://files.polypore.xyz/genesis/mainnet-genesis-tinkered/tinkered-genesis_2022-11-01T03%3A09%3A50.48336835Z_v7.1.0_12669839.json.gz`
* Upgrading from `v7.1.0` -> `6c3be2e24b3d4d224205f9ac560bb5932d4c66c7` (RC modifled to print denom metadata)
* Error message: `panic: atom denom not found`
```
ov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: 12:13PM INF Configuration is valid:
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: Configurable Values:
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: DAEMON_HOME: /home/gaia/.gaia
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: DAEMON_NAME: gaiad
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: DAEMON_ALLOW_DOWNLOAD_BINARIES: true
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: DAEMON_RESTART_AFTER_UPGRADE: true
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: DAEMON_POLL_INTERVAL: 300ms
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: UNSAFE_SKIP_BACKUP: true
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: DAEMON_PREUPGRADE_MAX_RETRIES: 0
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: Derived Values:
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: Root Dir: /home/gaia/.gaia/cosmovisor
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: Upgrade Dir: /home/gaia/.gaia/cosmovisor/upgrades
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: Genesis Bin: /home/gaia/.gaia/cosmovisor/genesis/bin/gaiad
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: Monitored File: /home/gaia/.gaia/data/upgrade-info.json
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: module=cosmovisor
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1260]: 12:13PM INF running app args=["start","--x-crisis-skip-assert-invariants","--home","/home/gaia/.gaia"] module=cosmovisor path=/home/gaia/.gaia/cosmovisor/upgrades/v8-Rho/bin/gaiad
Nov 04 12:13:56 udit-rho-test-stateful cosmovisor[1266]: 12:13PM INF starting node with ABCI Tendermint in-process
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: 12:14PM INF service start impl=multiAppConn module=proxy msg={}
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: 12:14PM INF service start connection=query impl=localClient module=abci-client msg={}
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: 12:14PM INF service start connection=snapshot impl=localClient module=abci-client msg={}
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: 12:14PM INF service start connection=mempool impl=localClient module=abci-client msg={}
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: 12:14PM INF service start connection=consensus impl=localClient module=abci-client msg={}
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: 12:14PM INF service start impl=EventBus module=events msg={}
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: 12:14PM INF service start impl=PubSub module=pubsub msg={}
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: 12:14PM INF service start impl=IndexerService module=txindex msg={}
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: 12:14PM INF ABCI Handshake App Info hash="�,G�!A\vP\x0f7\x1f\x16���\x19�i��w^\x1bX�N2��\x1e\v4" height=12669898 module=consensus protocol-version=0 software-version=HEAD-6c3be2e24b3d4d224205f9ac560bb5932d4c66c7
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: 12:14PM INF ABCI Replay Blocks appHeight=12669898 module=consensus stateHeight=12669898 storeHeight=12669899
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: 12:14PM INF Replay last block using real app module=consensus
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: 12:14PM INF applying upgrade "v8-Rho" at height: 12669899
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: base is: 'uatom'
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: wasn't able to retrieve with the same string that was just retrieved!!!
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: panic: atom denom not found
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: goroutine 1 [running]:
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk/x/upgrade/keeper.Keeper.ApplyUpgrade({{0x7ffc36ae4e1d, 0x10}, 0xc001063a10, {0x2c5e760, 0xc0016586b0}, {0x2c82e38, 0xc00061e260}, 0xc0012fec30, {0x2c55aa0, 0xc000da1340}, ...}, ...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk@v0.46.4/x/upgrade/keeper/keeper.go:339 +0x41a
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0x7ffc36ae4e1d, 0x10}, 0xc001063a10, {0x2c5e760, 0xc0016586b0}, {0x2c82e38, 0xc00061e260}, 0xc0012fec30, {0x2c55aa0, 0xc000da1340}, ...}, ...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk@v0.46.4/x/upgrade/abci.go:62 +0x66c
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk@v0.46.4/x/upgrade/module.go:134
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x2c75bc0, 0xc000052070}, {0x2c849a0, 0xc001623300}, {{0xb, 0x0}, {0xc0006379d0, 0xd}, 0xc153cb, ...}, ...}, ...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk@v0.46.4/types/module/module.go:484 +0x398
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/gaia/v8/app.(*GaiaApp).BeginBlocker(_, {{0x2c75bc0, 0xc000052070}, {0x2c849a0, 0xc001623300}, {{0xb, 0x0}, {0xc0006379d0, 0xd}, 0xc153cb, ...}, ...}, ...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/gaia/v8/app/app.go:252 +0x7b
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc00161c2c0, 0x20, 0x20}, {{0xb, 0x0}, {0xc0006379d0, 0xd}, 0xc153cb, {0x1b437c06, ...}, ...}, ...})
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk@v0.46.4/baseapp/abci.go:183 +0x843
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc00161c2c0, 0x20, 0x20}, {{0xb, 0x0}, {0xc0006379d0, 0xd}, 0xc153cb, {0x1b437c06, ...}, ...}, ...})
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint@v0.34.22/abci/client/local_client.go:280 +0x118
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc00161c2c0, 0x20, 0x20}, {{0xb, 0x0}, {0xc0006379d0, 0xd}, 0xc153cb, {0x1b437c06, ...}, ...}, ...})
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint@v0.34.22/proxy/app_conn.go:81 +0x55
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x2c76cd0?, 0xc001170cc0}, {0x2c7cf10, 0xc0015bdaf0}, 0xc000fec3c0, {0x2c84120, 0xc000178078}, 0xc153ca?)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint@v0.34.22/state/execution.go:307 +0x3dd
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0004f9fd8, 0x8}}, {0xc000636050, 0xd}, 0xc15390, 0xc153ca, {{0xc00050c820, ...}, ...}, ...}, ...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint@v0.34.22/state/execution.go:140 +0x171
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc0004f9fd8, 0x8}}, {0xc000636050, 0xd}, 0xc15390, 0xc153ca, {{0xc00050c820, ...}, ...}, ...}, ...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint@v0.34.22/consensus/replay.go:503 +0x23c
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc0004f9fd8, 0x8}}, {0xc000636050, 0xd}, 0xc15390, 0xc153ca, {{0xc00050c820, ...}, ...}, ...}, ...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint@v0.34.22/consensus/replay.go:416 +0x7ae
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc0019e3948, {0x2c850e0, 0xc0012fc1a0})
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint@v0.34.22/consensus/replay.go:268 +0x3c8
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc0004f9fd8, 0x8}}, {0xc000636050, 0xd}, 0xc15390, 0xc153ca, ...}, ...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint@v0.34.22/node/node.go:330 +0x1b8
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint/node.NewNode(0xc000187900, {0x2c724d0, 0xc0012ebc20}, 0xc0005463c0, {0x2c57160, 0xc000fbf848}, 0x1?, 0x0?, 0xc000546620, {0x2c76cd0, ...}, ...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/tendermint/tendermint@v0.34.22/node/node.go:778 +0x597
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x2c8c8f0, 0xc001281e60}, 0x0, {0xc000e9a660, 0xe}, {0x2c87428, ...}, ...}, ...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk@v0.46.4/server/start.go:313 +0x7fb
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc0010f0900?, {0xc001281800?, 0x0?, 0x3?})
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk@v0.46.4/server/start.go:143 +0x148
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/spf13/cobra.(*Command).execute(0xc0010f0900, {0xc001281770, 0x3, 0x3})
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/spf13/cobra.(*Command).ExecuteC(0xc0010bc900)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/spf13/cobra.(*Command).Execute(...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/spf13/cobra@v1.6.1/command.go:968
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/spf13/cobra.(*Command).ExecuteContext(...)
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/spf13/cobra@v1.6.1/command.go:961
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x1d48050?, {0x0, 0x0}, {0xc0004f91b0, 0x10})
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/cosmos-sdk@v0.46.4/server/cmd/execute.go:36 +0x20f
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: main.main()
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1266]: github.com/cosmos/gaia/v8/cmd/gaiad/main.go:16 +0x30
Nov 04 12:14:10 udit-rho-test-stateful cosmovisor[1260]: 12:14PM ERR error="exit status 2" module=cosmovisor
```
## 2022-Nov-03
* Result : **FAILING**
* Local stateful upgrade
* Genesis used: `https://files.polypore.xyz/genesis/mainnet-genesis-tinkered/tinkered-genesis_2022-11-01T03%3A09%3A50.48336835Z_v7.1.0_12669839.json.gz`
* Upgrading from `v7.1.0` -> `16249d8b92d49b209db105df2feaae68d8c97b5c` (RC modifled to print denom metadata)
* Print output: `actualMetadata [{The native staking token of the Cosmos Hub. [denom:"uatom" aliases:"microatom" denom:"matom" exponent:3 aliases:"milliatom" denom:"atom" exponent:6 ] uatom atom }]`
* Error message: `panic: atom denom not found`
Full log:
```
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: 4:27PM INF Configuration is valid:
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: Configurable Values:
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: DAEMON_HOME: /home/gaia/.gaia
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: DAEMON_NAME: gaiad
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: DAEMON_ALLOW_DOWNLOAD_BINARIES: true
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: DAEMON_RESTART_AFTER_UPGRADE: true
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: DAEMON_POLL_INTERVAL: 300ms
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: UNSAFE_SKIP_BACKUP: true
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: DAEMON_PREUPGRADE_MAX_RETRIES: 0
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: Derived Values:
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: Root Dir: /home/gaia/.gaia/cosmovisor
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: Upgrade Dir: /home/gaia/.gaia/cosmovisor/upgrades
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: Genesis Bin: /home/gaia/.gaia/cosmovisor/genesis/bin/gaiad
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: Monitored File: /home/gaia/.gaia/data/upgrade-info.json
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: module=cosmovisor
Nov 03 16:27:26 rho-udit-test cosmovisor[6344]: 4:27PM INF running app args=["start","--x-crisis-skip-assert-invariants","--home","/home/gaia/.gaia"] module=cosmovisor path=/home/gaia/.gaia/cosmovisor/upgrades/v8-Rho/bin/gaiad
Nov 03 16:27:26 rho-udit-test cosmovisor[6350]: 4:27PM INF starting node with ABCI Tendermint in-process
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: 4:27PM INF service start impl=multiAppConn module=proxy msg={}
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: 4:27PM INF service start connection=query impl=localClient module=abci-client msg={}
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: 4:27PM INF service start connection=snapshot impl=localClient module=abci-client msg={}
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: 4:27PM INF service start connection=mempool impl=localClient module=abci-client msg={}
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: 4:27PM INF service start connection=consensus impl=localClient module=abci-client msg={}
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: 4:27PM INF service start impl=EventBus module=events msg={}
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: 4:27PM INF service start impl=PubSub module=pubsub msg={}
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: 4:27PM INF service start impl=IndexerService module=txindex msg={}
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: 4:27PM INF ABCI Handshake App Info hash="4�Q�\x16l=���*|\x03t����/G�����\x17���\x17z[" height=12670399 module=consensus protocol-version=0 software-version=HEAD-16249d8b92d49b209db105df2feaae68d8c97b5c
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: 4:27PM INF ABCI Replay Blocks appHeight=12670399 module=consensus stateHeight=12670399 storeHeight=12670400
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: 4:27PM INF Replay last block using real app module=consensus
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: 4:27PM INF applying upgrade "v8-Rho" at height: 12670400
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: actualMetadata [{The native staking token of the Cosmos Hub. [denom:"uatom" aliases:"microatom" denom:"matom" exponent:3 aliases:"milliatom" denom:"atom" exponent:6 ] uatom atom }]
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: panic: atom denom not found
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: goroutine 1 [running]:
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk/x/upgrade/keeper.Keeper.ApplyUpgrade({{0x7ffe4e87de1d, 0x10}, 0xc00156dc80, {0x2c5e6a0, 0xc0015b86b0}, {0x2c82d78, 0xc00001aeb0}, 0xc001655a70, {0x2c559e0, 0xc001294540}, ...}, ...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk@v0.46.4/x/upgrade/keeper/keeper.go:339 +0x41a
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0x7ffe4e87de1d, 0x10}, 0xc00156dc80, {0x2c5e6a0, 0xc0015b86b0}, {0x2c82d78, 0xc00001aeb0}, 0xc001655a70, {0x2c559e0, 0xc001294540}, ...}, ...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk@v0.46.4/x/upgrade/abci.go:62 +0x66c
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk@v0.46.4/x/upgrade/module.go:134
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x2c75b00, 0xc000136000}, {0x2c848e0, 0xc000fd41c0}, {{0xb, 0x0}, {0xc000545a60, 0xd}, 0xc155c0, ...}, ...}, ...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk@v0.46.4/types/module/module.go:484 +0x398
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/gaia/v8/app.(*GaiaApp).BeginBlocker(_, {{0x2c75b00, 0xc000136000}, {0x2c848e0, 0xc000fd41c0}, {{0xb, 0x0}, {0xc000545a60, 0xd}, 0xc155c0, ...}, ...}, ...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/gaia/v8/app/app.go:252 +0x7b
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc001595a80, 0x20, 0x20}, {{0xb, 0x0}, {0xc000545a60, 0xd}, 0xc155c0, {0x1fc81ce1, ...}, ...}, ...})
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk@v0.46.4/baseapp/abci.go:183 +0x843
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc001595a80, 0x20, 0x20}, {{0xb, 0x0}, {0xc000545a60, 0xd}, 0xc155c0, {0x1fc81ce1, ...}, ...}, ...})
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint@v0.34.22/abci/client/local_client.go:280 +0x118
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc001595a80, 0x20, 0x20}, {{0xb, 0x0}, {0xc000545a60, 0xd}, 0xc155c0, {0x1fc81ce1, ...}, ...}, ...})
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint@v0.34.22/proxy/app_conn.go:81 +0x55
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x2c76c10?, 0xc000dc8fc0}, {0x2c7ce50, 0xc000519ac0}, 0xc000ff6b40, {0x2c84060, 0xc0018da090}, 0xc155bf?)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint@v0.34.22/state/execution.go:307 +0x3dd
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc001251600, 0x8}}, {0xc001251610, 0xd}, 0xc15390, 0xc155bf, {{0xc0018d23c0, ...}, ...}, ...}, ...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint@v0.34.22/state/execution.go:140 +0x171
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc001251600, 0x8}}, {0xc001251610, 0xd}, 0xc15390, 0xc155bf, {{0xc0018d23c0, ...}, ...}, ...}, ...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint@v0.34.22/consensus/replay.go:503 +0x23c
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc001251600, 0x8}}, {0xc001251610, 0xd}, 0xc15390, 0xc155bf, {{0xc0018d23c0, ...}, ...}, ...}, ...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint@v0.34.22/consensus/replay.go:416 +0x7ae
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc0012d7948, {0x2c85020, 0xc0000c8410})
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint@v0.34.22/consensus/replay.go:268 +0x3c8
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc001251600, 0x8}}, {0xc001251610, 0xd}, 0xc15390, 0xc155bf, ...}, ...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint@v0.34.22/node/node.go:330 +0x1b8
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint/node.NewNode(0xc0001e4f00, {0x2c72410, 0xc0017a4280}, 0xc00001bfc0, {0x2c570a0, 0xc0018da5d0}, 0x1?, 0x0?, 0xc000d6c5f0, {0x2c76c10, ...}, ...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/tendermint/tendermint@v0.34.22/node/node.go:778 +0x597
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x2c8c830, 0xc001197b30}, 0x0, {0xc0010ea070, 0xe}, {0x2c87368, ...}, ...}, ...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk@v0.46.4/server/start.go:313 +0x7fb
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc001068600?, {0xc0011974d0?, 0x0?, 0x3?})
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk@v0.46.4/server/start.go:143 +0x148
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/spf13/cobra.(*Command).execute(0xc001068600, {0xc001197440, 0x3, 0x3})
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/spf13/cobra.(*Command).ExecuteC(0xc00102e600)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/spf13/cobra.(*Command).Execute(...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/spf13/cobra@v1.6.1/command.go:968
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/spf13/cobra.(*Command).ExecuteContext(...)
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/spf13/cobra@v1.6.1/command.go:961
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x1d47f10?, {0x0, 0x0}, {0xc000544750, 0x10})
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/cosmos-sdk@v0.46.4/server/cmd/execute.go:36 +0x20f
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: main.main()
Nov 03 16:27:39 rho-udit-test cosmovisor[6350]: github.com/cosmos/gaia/v8/cmd/gaiad/main.go:16 +0x30
Nov 03 16:27:39 rho-udit-test cosmovisor[6344]: 4:27PM ERR error="exit status 2" module=cosmovisor
```
## 2022-Nov-02
* Result : **FAILING**
* Local fresh state
* Upgrading from `v7.1.0` -> `v8.0.0-rc`
* Automated tests have also been failing:
* Fresh state upgrade: https://github.com/hyphacoop/cosmos-ansible/actions/runs/3380643940/jobs/5613657690
* Stateful upgrade: https://github.com/hyphacoop/cosmos-ansible/actions/runs/3380703187/jobs/5613787827
```
Nov 02 16:21:40 dante-rho-2 cosmovisor[20609]: 4:21PM INF Daemon shutting down in an attempt to restart module=cosmovisor
Nov 02 16:21:40 dante-rho-2 cosmovisor[20609]: 4:21PM INF pre-upgrade command does not exist. continuing the upgrade. module=cosmovisor
Nov 02 16:21:40 dante-rho-2 cosmovisor[20609]: 4:21PM INF No upgrade binary found, beginning to download it module=cosmovisor
Nov 02 16:21:42 dante-rho-2 cosmovisor[20609]: 4:21PM INF Downloading binary complete module=cosmovisor
Nov 02 16:21:42 dante-rho-2 cosmovisor[20609]: 4:21PM INF upgrade detected, relaunching app=gaiad module=cosmovisor
Nov 02 16:21:42 dante-rho-2 cosmovisor[20609]: 4:21PM INF running app args=["start","--x-crisis-skip-assert-invariants","--home","/home/gaia/.gaia"] module=cosmovisor path=/home/gaia/.gaia/cosmovisor/upgrades/v8-Rho/bin/gaiad
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF starting node with ABCI Tendermint in-process
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF service start impl=multiAppConn module=proxy msg={}
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF service start connection=query impl=localClient module=abci-client msg={}
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF service start connection=snapshot impl=localClient module=abci-client msg={}
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF service start connection=mempool impl=localClient module=abci-client msg={}
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF service start connection=consensus impl=localClient module=abci-client msg={}
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF service start impl=EventBus module=events msg={}
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF service start impl=PubSub module=pubsub msg={}
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF service start impl=IndexerService module=txindex msg={}
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF ABCI Handshake App Info hash="\x1a�\x15\x18N#;�u�\b��\a�|��(پ���\x1cl�\u007f=ZR�" height=49 module=consensus protocol-version=0 software-version=v8.0.0-rc
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF ABCI Replay Blocks appHeight=49 module=consensus stateHeight=49 storeHeight=50
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF Replay last block using real app module=consensus
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: 4:21PM INF applying upgrade "v8-Rho" at height: 50
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: panic: atom denom not found
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: goroutine 1 [running]:
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk/x/upgrade/keeper.Keeper.ApplyUpgrade({{0x7fff695bee1c, 0x10}, 0xc000eb1770, {0x2c57820, 0xc0016a2750}, {0x2c7be28, 0xc00068b0c0}, 0xc00167b770, {0x2c4eb60, 0xc000682fc0}, ...}, ...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk@v0.46.3/x/upgrade/keeper/keeper.go:339 +0x41a
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0x7fff695bee1c, 0x10}, 0xc000eb1770, {0x2c57820, 0xc0016a2750}, {0x2c7be28, 0xc00068b0c0}, 0xc00167b770, {0x2c4eb60, 0xc000682fc0}, ...}, ...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk@v0.46.3/x/upgrade/abci.go:62 +0x66c
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk@v0.46.3/x/upgrade/module.go:134
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x2c6ebc0, 0xc000136000}, {0x2c7d9a0, 0xc0011fd840}, {{0xb, 0x0}, {0xc00052ec20, 0xa}, 0x32, ...}, ...}, ...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk@v0.46.3/types/module/module.go:484 +0x398
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/gaia/v8/app.(*GaiaApp).BeginBlocker(_, {{0x2c6ebc0, 0xc000136000}, {0x2c7d9a0, 0xc0011fd840}, {{0xb, 0x0}, {0xc00052ec20, 0xa}, 0x32, ...}, ...}, ...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/gaia/v8/app/app.go:252 +0x7b
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc000eed9e0, 0x20, 0x20}, {{0xb, 0x0}, {0xc00052ec20, 0xa}, 0x32, {0x2adc46ce, ...}, ...}, ...})
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk@v0.46.3/baseapp/abci.go:183 +0x843
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc000eed9e0, 0x20, 0x20}, {{0xb, 0x0}, {0xc00052ec20, 0xa}, 0x32, {0x2adc46ce, ...}, ...}, ...})
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint@v0.34.22/abci/client/local_client.go:280 +0x118
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc000eed9e0, 0x20, 0x20}, {{0xb, 0x0}, {0xc00052ec20, 0xa}, 0x32, {0x2adc46ce, ...}, ...}, ...})
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint@v0.34.22/proxy/app_conn.go:81 +0x55
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x2c6fcd0?, 0xc0014493e0}, {0x2c75f10, 0xc000e65ae0}, 0xc0011932c0, {0x2c7d110, 0xc00000e150}, 0x31?)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint@v0.34.22/state/execution.go:307 +0x3dd
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0016a44d8, 0x8}}, {0xc0016a44f0, 0xa}, 0x1, 0x31, {{0xc000d621c0, ...}, ...}, ...}, ...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint@v0.34.22/state/execution.go:140 +0x171
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc0016a44d8, 0x8}}, {0xc0016a44f0, 0xa}, 0x1, 0x31, {{0xc000d621c0, ...}, ...}, ...}, ...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint@v0.34.22/consensus/replay.go:503 +0x23c
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc0016a44d8, 0x8}}, {0xc0016a44f0, 0xa}, 0x1, 0x31, {{0xc000d621c0, ...}, ...}, ...}, ...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint@v0.34.22/consensus/replay.go:416 +0x7ae
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc0010e7948, {0x2c7e0e0, 0xc0014ad2b0})
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint@v0.34.22/consensus/replay.go:268 +0x3c8
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc0016a44d8, 0x8}}, {0xc0016a44f0, 0xa}, 0x1, 0x31, ...}, ...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint@v0.34.22/node/node.go:330 +0x1b8
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint/node.NewNode(0xc0014ce000, {0x2c6b4c0, 0xc001441ea0}, 0xc000e651b0, {0x2c50220, 0xc00000f170}, 0x1?, 0x0?, 0xc000e653e0, {0x2c6fcd0, ...}, ...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/tendermint/tendermint@v0.34.22/node/node.go:778 +0x597
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x2c858f0, 0xc001363fb0}, 0x0, {0xc00114f8d0, 0xa}, {0x2c80428, ...}, ...}, ...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk@v0.46.3/server/start.go:311 +0x7fb
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc0011e2300?, {0xc001363950?, 0x0?, 0x3?})
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk@v0.46.3/server/start.go:143 +0x148
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/spf13/cobra.(*Command).execute(0xc0011e2300, {0xc0013638c0, 0x3, 0x3})
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/spf13/cobra.(*Command).ExecuteC(0xc0011a6600)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/spf13/cobra.(*Command).Execute(...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/spf13/cobra@v1.6.1/command.go:968
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/spf13/cobra.(*Command).ExecuteContext(...)
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/spf13/cobra@v1.6.1/command.go:961
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x1d43050?, {0x0, 0x0}, {0xc001232c10, 0x10})
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/cosmos-sdk@v0.46.3/server/cmd/execute.go:36 +0x20f
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: main.main()
Nov 02 16:21:42 dante-rho-2 cosmovisor[20777]: github.com/cosmos/gaia/v8/cmd/gaiad/main.go:16 +0x30
Nov 02 16:21:42 dante-rho-2 cosmovisor[20609]: 4:21PM ERR error="exit status 2" module=cosmovisor
Nov 02 16:21:42 dante-rho-2 systemd[1]: cosmovisor.service: Main process exited, code=exited, status=1/FAILURE
Nov 02 16:21:42 dante-rho-2 systemd[1]: cosmovisor.service: Failed with result 'exit-code'.
```
Sign in with Wallet
Connect another wallet