v11.0.0-rc1 Local Upgrade test ===== This is the procedure carried out to reproduce the error observed during the testnet upgrade from v10.0.0 to v11.0.0-rc1 To do this 2 branches were created to setup the node with pruning settings similar to the ones reported by the validators ### Perform the upgrade from branch `GAtom22/v10.0.0` to branch `GAtom22/v11.0.0-rc1` Below is the step by step procedure to reproduce the error: 1. Start localnode with v10.0.0 ```shell= # clone the evmos repo or cd into the corresponding directory git checkout GAtom22/v10.0.0 ./local_node.sh # if you have an existing ~/.tmp-evmosd directory, # type 'y' on the prompt to overwrite the existing folder ``` 2. Create upgrade proposal to v11.0.0-rc1 ```shell= evmosd tx gov submit-legacy-proposal software-upgrade "v11.0.0-rc1" --title="v11.0.0-rc1" --description="v11 upgrade proposal" --upgrade-height=25 --upgrade-info='{"binaries":{"darwin/arm64":"https://github.com/evmos/evmos/releases/download/v11.0.0-rc1/evmos_11.0.0-rc1_Darwin_arm64.tar.gz?checksum=40f49415a299f7bc598dbb35952f0a084c51b5296b128e86c050302aeb525cd8","darwin/amd64":"https://github.com/evmos/evmos/releases/download/v11.0.0-rc1/evmos_11.0.0-rc1_Darwin_amd64.tar.gz?checksum=cdaccc076b25e94d6481a8665fe1d5862acd0f9ba7afecc63733410c21695f14","linux/arm64":"https://github.com/evmos/evmos/releases/download/v11.0.0-rc1/evmos_11.0.0-rc1_Linux_arm64.tar.gz?checksum=24b6c1da488133f3fcdc0dc30a699326cecfd28f36e00d4a13352ee96802a3e5","linux/amd64":"https://github.com/evmos/evmos/releases/download/v11.0.0-rc1/evmos_11.0.0-rc1_Linux_amd64.tar.gz?checksum=57f4061d332a755e87d0595ae951d9917a2aa6771a9c8da0e23a2ebe9f728f09"}}' --chain-id=evmos_9000-1 --from=dev0 -b=block --yes --keyring-backend=test --fees=2000000aevmos --gas=auto --home ~/.tmp-evmosd --no-validate ``` 3. Make deposit to proposal ```shell= evmosd tx gov deposit 1 10000000aevmos --from=dev0 --chain-id=evmos_9000-1 -b=block --yes --keyring-backend=test --fees=20000aevmos --gas=auto --home ~/.tmp-evmosd ``` 4. Vote yes on proposal ```shell= evmosd tx gov vote 1 yes --from=dev0 --chain-id=evmos_9000-1 -b=block --yes --keyring-backend=test --fees=200000aevmos --gas=auto --home ~/.tmp-evmosd ``` 5. The corresponding upgrade block height is reached. Then checkout the v11.0.0-rc1 branch ```shell git checkout GAtom22/v11.0.0-rc1 ./local_node.sh # IMPORTANT!! Type 'n' to avoid overwriting the existing folder ``` After a few blocks (5 to 10) you will get the error message reported during the upgrade: ```shell= 8:09PM ERR CONSENSUS FAILURE!!! err="cannot delete latest saved version (6)" module=consensus server=node stack="goroutine 200 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:... ``` # Issues on other chains I've found this [issue on Gaia](https://github.com/cosmos/ibc-go/issues/1088), however I tried different things on these lines but couldn't make the error message go away. Also tried removing [the store upgrade](https://github.com/evmos/evmos/blob/e7938080eb24ce587dfa633a3031058851591bfd/app/app.go#L1223-L1228) for the ICA host, but got the same behavior.