# Pion-1 AppHash

## How to reproduce

It seems like we found the reason of apphash error. It is 100% reproducible. To reproduce it:

1) Init a brand new node:

# clear the data if necessary
rm -rf data/pion-test/ 
./build/neutrond init test --home data/pion-test
cp ${PION_GENESIS_DIR}/genesis.json data/pion-test/config/

2) Make sure you have no any `addrbook.json`, or `persisten_peer`, or seed node configured
3) Run the node `./build/neutrond start --home data/pion-test`
4) When you see the `INF No addresses to dial. Falling back to seeds` log message, stop the node, copy addbook for `pion-1` testnet to the config directory `cp ${PION_GENESIS_DIR}/addrbook.json data/pion-test/config/`. Or configure `seed/persistent_peers` config params
5) Run the node again `./build/neutrond start --home data/pion-test`
6) You get the AppHash error: `ERR Error in validation err="wrong Block.Header.AppHash.  Expected 4F55094FDE1098F7AA2AE6184C70508A3211CFBC6A16EAE4679C5486DD30AEC0, got 68CD22391908C4896894DD0DB0F3AD48EFA4FAE6D9B803A089C5C2D0D0E72100" module=blockchain`

## Background information

1) Neutron uses `wasmd` genesis messages to instantiate its DAO and TGE contracts;
2) During genesis creation, genesis was formatted to have newlines, indentation, etc.
3) During the first run (3 from the list above) a node reads the genesis from the file (`genesis.json`) and dumps it into the LevelDB state(`state.db`). While doing that, the `json.Marshaler` [encoded some fields (baked insatntiate msgs for our TGE contracts) with no formatting, as one line](https://github.com/skip-mev/mev-cometbft/blob/v0.34.27-mev.17/node/node.go#L1524). As a result, the instatiate (execute) wasm msgs format differs in `genesis.json` and genesis dumped into state
4) You run the node again (5 from the list above), the genesis is read from the state, and not from the file. The init msgs are not formatted.
5) The node executes the messages and they are stored into the state with no formatting.

### Demontration

In order to inspect the issue, we traced all state changes that happended until `height=2`:

We found that some writes with the same unique keys were present both in the normal node run (no restart) and in the restart scenario, but the *values assigned to those keys were different*.

#### Write for a specific key without a restart (no AppHash error)

`{'operation': 'write', 'key': b'\x05\xcf\xf6\x88\xbc\xa9c{\x07\xbdBQ;\xc8\x10\x88\x81\x13u\x94\x01!\xb3\xd6\x80\xd24N\xbe\x88d\x92\xd8\x00\x00\x00\x00\x00\x00\x00\x01', 'value': b'\x08\x01\x10 \x1a\x00"\x90\x05{\n "pair_configs": [\n {\n "code_id": 34,\n "pair_type": {\n "xyk": {}\n },\n "total_fee_bps": 0,\n "maker_fee_bps": 0,\n "is_disabled": false,\n "is_generator_disabled": false\n }\n ],\n "token_code_id": 35,\n "owner": "neutron19y9uedlq0cpugg5a5jtxn8vs5rdwepnk7v863qmyc0p0899dfxxqq3wuwe",\n "whitelist_code_id": 0,\n "coin_registry_address": "neutron1g9pvwvm0zmgjj02zeyklxc9pzyh8kret8sklk72xrljl8ffvjmaqmagyqt"\n }', 'metadata': {'store_name': 'wasm'}}`

#### Write for the same key after restart (AppHash error after that)

`{'operation': 'write', 'key': b'\x05\xcf\xf6\x88\xbc\xa9c{\x07\xbdBQ;\xc8\x10\x88\x81\x13u\x94\x01!\xb3\xd6\x80\xd24N\xbe\x88d\x92\xd8\x00\x00\x00\x00\x00\x00\x00\x01', 'value': b'\x08\x01\x10 \x1a\x00"\xe1\x02{"pair_configs":[{"code_id":34,"pair_type":{"xyk":{}},"total_fee_bps":0,"maker_fee_bps":0,"is_disabled":false,"is_generator_disabled":false}],"token_code_id":35,"owner":"neutron19y9uedlq0cpugg5a5jtxn8vs5rdwepnk7v863qmyc0p0899dfxxqq3wuwe","whitelist_code_id":0,"coin_registry_address":"neutron1g9pvwvm0zmgjj02zeyklxc9pzyh8kret8sklk72xrljl8ffvjmaqmagyqt"}', 'metadata': {'store_name': 'wasm'}}`

To get these messages we run node with `--trace_store` flag
`./build/neutrond start --home data/pion-test --trace-store /tmp/trace_good_run --halt-height 2` for both cases: the successful run and the apphash scenario.

To decode the files use python code snippet below.

#### Simple script to decode the keys and values from the state dump

import json
import base64

with open("./good_run_sorted.crash", "r") as f:
    for line in f:
        jline = json.loads(line)
        jline["key"] = base64.b64decode(jline["key"])
        jline["value"] = base64.b64decode(jline["value"])
        print(jline)

## Why the apphash error happens?

Most of the validators did not interrupt a node in steps 3-5 and they got executed msgs from `genesis.json` written into the storage. Well formatted msgs were written. The AppHash was equal to `68CD22391908C4896894DD0DB0F3AD48EFA4FAE6D9B803A089C5C2D0D0E72100`.

Some of the validators interupted for some reason node right after the init (4 step) - genesis dumped into the state. After restart, the genesis was read from the state and not formatted msgs were executed and written into the state. The AppHash was equal to 4F55094FDE1098F7AA2AE6184C70508A3211CFBC6A16EAE4679C5486DD30AEC0.

The whole thing would not have happened if the genesis was distributed without formatting.

## Fix
The bug was fixed in the latest release [v1.0.0-rc1](https://github.com/neutron-org/neutron/releases/tag/v1.0.0-rc1)

The idea is to always format the [messages saved into the state](https://github.com/neutron-org/wasmd/commit/c4c2f0be8ceca3fcbcc744a4129c2bc5e566c4a5) to avoid inconsistencies between `genesis.json` and the genesis from node state.