# Coldstack hard fork operation guide

[TOC]

Note: This document is a work in progress.

# Implementation notes

This section is more of notes that I keep for myself. If you are uninteresting in how it works and only want to know how to get the job done, I recommend jumping to the [Operation guide](#Operation-guide) section and start from there

## Preparations

1. Historical block archive exported from `export-blocks` subcommand: `9689999.json`
2. Original binary for launching the network: `coldstack`
3. Original json chain spec that was used in production: `prod.json`
4. HTTP RPC node with all historical blocks, for example http://127.0.0.1:9933
5. Polkadot.js Apps Frontend: https://blockchain.coldstack.io/?rpc=wss://blockchain-ws.coldstack.io/

### Launch an archive rpc node at http://127.0.0.1:9933

1. Import the blocks into `db` directory from the json backup:

$ coldstack import-blocks --chain=prod.json -d db 9689999.json --pruning=archive

2. Launch the archive node with imported `db`

$ coldstack -d db --chain=prod.json --rpc-cors=all --pruning=archive

Now the HTTP RPC endpoint http://127.0.0.1:9933 will be used by [fork-off-substrate](https://github.com/maxsam4/fork-off-substrate) to create a fork

## Goal 1: Creating a basic fork

We are going to used https://github.com/maxsam4/fork-off-substrate to do a basic fork

First we clone the repository into `fork-off-substrate`

$ git clone https://github.com/maxsam4/fork-off-substrate

Then enter the directory

$ cd fork-off-substrate

Install dependencies

$ yarn

Create a `data` directory inside the current one (`fork-off-substrate`)

$ mkdir -p data

Copy the `coldstack` binary to `data/binary`

$ cp `which coldstack` ./data/binary

Create `data/schema.json` with the following content:

{
  "types": {
    "Gateway": {
      "address": "Vec<u8>",
      "seedAddress": "Option<Vec<u8>>",
      "storage": "u8"
    }
  }
}

Generate `data/runtime.wasm` from `prod.json`:

$ cat prod.json | jq '.genesis.raw.top["0x3a636f6465"]' -r | xxd -r -p > data/runtime.wasm 

Now run the script to start the fork:

$ HTTP_RPC_ENDPOINT=http://127.0.0.1:9933 npm start

Wait around 15 minutes for the script to generate a new chain spec located at `data/fork.json`

Then you will be able to start the forked chain:

$ coldstack --chain data/fork.json

Now the node can be started with a new genesis config, but it has a few problems:

- it can't produce blocks
- the system balances are not recovered
- the coldStack balances are not recovered

In the following sections we will solve them one by one.

## Goal 2: Customizing initial authorities

$ coldstack build-spec --disable-default-bootnode > local.json

$ coldstack build-spec --disable-default-bootnode --dev > dev.json 

Suppose we want to use `//Alice` as the seed phrase for our default validator stash/controller/aura/grandpa keys, we can use this command to derive these keys respectively:

stash:

$ coldstack key inspect --scheme=sr25519 //Alice --output-type json | jq -r .ss58Address
5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY

controller:

$ coldstack key inspect --scheme=sr25519 //Alice --output-type json | jq -r .ss58Address
5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY

aura:

$ coldstack key inspect --scheme=sr25519 //Alice --output-type json | jq -r .ss58Address
5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY

grandpa:

$ coldstack key inspect --scheme=ed25519 //Alice --output-type json | jq -r .ss58Address
5FA9nQDVg267DEd8m1ZypXLBnvN7SFxYwV7ndqSYGiN9TTpu

then update the value in `local.json` corresponding to path `.genesis.runtime.session.keys` to

[
  [
    "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
    "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
    {
      "aura": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
      "grandpa": "5FA9nQDVg267DEd8m1ZypXLBnvN7SFxYwV7ndqSYGiN9TTpu"
    }
  ]
]

`.genesis.runtime.validatorSet.validators` to

[
  "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"
]

Then start the node with `--alice` flag, it will start producing blocks:

$ coldstack --chain-spec=local.json --alice --pruning=archive --tmp

If you want to customize it and make each key different, for example:

- stash: `//Alice/stash`
- controller: `//Alice/controller`
- aura: `//Alice/aura`
- grandpa: `//Alice//grandpa`

the session keys will be:

[
  [
    "5DZnGRAr28KP4GvbuxW2cBNo9Aodcm4QKUMj3Zqj67YjYStr",
    "5CFcgb22HdU9iKXsZyn89cmrqJRVNHbHqhED47KxCAxV5DWH"
  ],
  {
    "aura": "5DskCSEQbmr8iMaGHtU8roPiTpYhmH6iEe9G9dZD4zLPqVY7",
    "grandpa": "5EZAkmxARDqRz5z5ojuTjacTs2rTd7WRL1A9ZeLvwgq2STA2"
  }
]

then insert the custom session keys to the keystore

$ coldstack key insert --key-type gran --scheme ed25519

$ coldstack key insert --key-type audi --scheme sr25519

## Goal 3: Restoring system account balances

Restoring the system account balances is relatively easy. Similar to the previous section, the account balances can be set by modifying the key value pairs located at json path `.genesis.runtime.balances.balances`

[
  [
    "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
    1152921504606846976
  ],
  [
    "5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty",
    1152921504606846976
  ],
  [
    "5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY",
    1152921504606846976
  ],
  [
    "5HpG9w8EBLe5XCrbczpwq5TSXvedjrBGCwqxK1iQ7qUsSWFc",
    1152921504606846976
  ]
]

The four keys correspond to //Alice, //Bob, //Alice//Stash, //Bob//Stash

If you want to only allocate 1 Unit of token for //Alice and //Bob, the json field should be updated to:

[
  [
    "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
    1000000000000
  ],
  [
    "5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty",
    1000000000000
  ]
]

Save changes to the chain spec and launch the node:

$ coldstack --chain=dev.json --validator --telemetry-url 'wss://telemetry.rs/submit 1'  --rpc-cors=all

You will see the total issuance is 2 Unit, with Alice and Bob each holding 1 Unit

## Goal 4: Restoring coldStack balances

Unlike the system balances pallet, coldStack pallet doesn't support loading initial balances from the human readable chain spec json file, so we need to convert the chain spec to its raw format, then hardcode key value pairs into it.

First there should be a json file containing eth hex addresses and their corresponding u128 balances, like this:

{
  "0xc0ffee254729296a45a3885639AC7E10F9d54979": "1234132412341",
  "0x999999cf1046e68e36E1aA2E0E07105eDDD1f08E": "4312431431342"
}

Then we need to generate raw key value pairs from it.

The key will have three parts:

- module twox128("ColdStack") = `0x05e8b43efc523aff1c9b6d80f6edee7a`
- method twox128("Balances") = `0xc2261276cc9d1f8598ea4b6a74b15c2f`
- blake2_128concat(bytes)

where bytes is the SCALE-encoded eth address of type `Bytes`

For example, the blake2_128 hash of address `0xc0ffee254729296a45a3885639AC7E10F9d54979` is `0x06eba8679342530672298a53b531532650`, so `blake2_128concat(bytes)` will be the hash concatenated with the address: `0x06eba8679342530672298a53b531532650c0ffee254729296a45a3885639ac7e10f9d54979`, and the full key is made from the formula

twox128("ColdStack") + twox128("Balances") + blake2_128concat(bytes)

so the final key is

`0x05e8b43efc523aff1c9b6d80f6edee7ac2261276cc9d1f8598ea4b6a74b15c2f06eba8679342530672298a53b531532650c0ffee254729296a45a3885639ac7e10f9d54979`

The value is simply the SCALE-encoded `u128` value converted to hex, so it can be computed with:

console.log(u8aToHex(registry.createType('u128', '1234132412341').toU8a()))
// 0xb52706581f0100000000000000000000

Thus, the key value pair `"0xc0ffee254729296a45a3885639AC7E10F9d54979": "1234132412341"` is converted to its raw form

"0x05e8b43efc523aff1c9b6d80f6edee7ac2261276cc9d1f8598ea4b6a74b15c2f06eba8679342530672298a53b531532650c0ffee254729296a45a3885639ac7e10f9d54979": "0xb52706581f0100000000000000000000"

Repeat the same operation for the second pair, we get

"0x05e8b43efc523aff1c9b6d80f6edee7ac2261276cc9d1f8598ea4b6a74b15c2f86790263af74e08c49ab894530df18bd50999999cf1046e68e36e1aa2e0e07105eddd1f08e": "0xae8ef110ec0300000000000000000000"

Now insert the raw key value pairs to the `.genesis.raw.top` path of the raw json spec:

{
  "0x05e8b43efc523aff1c9b6d80f6edee7ac2261276cc9d1f8598ea4b6a74b15c2f06eba8679342530672298a53b531532650c0ffee254729296a45a3885639ac7e10f9d54979": "0xb52706581f0100000000000000000000",
  "0x05e8b43efc523aff1c9b6d80f6edee7ac2261276cc9d1f8598ea4b6a74b15c2f86790263af74e08c49ab894530df18bd50999999cf1046e68e36e1aa2e0e07105eddd1f08e": "0xae8ef110ec0300000000000000000000"
}

$ coldstack build-spec --chain=dev.json --raw --disable-default-bootnode > devRaw.json

and launch the chain with the raw spec

$ coldstack --chain=devRaw.json --validator --rpc-cors=all

## Goal 5: Putting it all together

I created a repository https://github.com/btwiuse/coldstack-recovery which contains scripts that will automate the previous steps. Here is now to use it

Clone the repository and enter the project directory

$ git clone https://github.com/btwiuse/coldstack-recovery && cd coldstack-recovery

Install necessary dependencies: `jq`, `deno`, `coldstack` then run

$ make

If the command finished successfully, the chain spec will be located at `./build/fork.json`

We can use this json file to launch the forked network:

$ coldstack --chain=./build/fork.json --tmp --validator --alice --rpc-cors=all

I will explain how to do customizations in the hard fork process in the next section

# Operation guide

This repository https://github.com/btwiuse/coldstack-recovery/ has been created to facilitate the forking process.

The following steps assume you that already cloned it and is inside the coldstack-recovery directory

## Prerequisites

### Installing Dependencies

A few dependencies must be installed before you proceed: `jq`, `deno`

You can install `jq` with

$ sudo apt install -y jq

Then follow this page to install deno: https://deno.land/manual@v1.36.1/getting_started/installation

Also, the `coldstack` binary should be added to your `PATH` environment variable

### Running the project with default configuration

With dependencies installed, you can create a fork by simply running the `make` command

$ make

The forked json spec will be created at `./build/fork.json`

You can start the forked network by running the following command:

$ coldstack --chain=./build/fork.json --tmp --validator --alice --rpc-cors=all

Please follow the following sections to perform customizations while forking the network.

## Step 1: Restoration of storage with balances:

The evm address balances are stored in file `genesis.raw.top.coldStack.balances.json`, with the following format

{
  "0x2802907ab681298d35ce2eaf2415550137c40c8a": "9993742781506984771",
  "0x26b7927d8bc3ad508e9e39adbeeb340699b3fb27": "9998668483351390258",
  "0xba404cfefe3433c7d577b2062fb542dc170955bf": "9998684145089755642",
  ...
}

where the key is the wallet address, and value is user's balance, both encoded as a string.

If you want to customize the values, you can replace this file with your own version, then run `make` command to regenerate the `./build/fork.json` chain spec with your custom balances

## Step 2: Restoration of wallet balances:

The system balances config is located at `genesis.runtime.balances.balances.json`

It has a slightly different format:

[
  [
    "5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY",
    29999374999361
  ],
  [
    "5FviQTMcu22GmvqhhQZothTk2gcFwjY1YUQi6swwKUyEtdft",
    1152921404231846608
  ],
  [
    "5CZhSbpeeFW6Vew9A21VVAphPVpL3BrAeofhoRCVAfcDxfoq",
    49749999656
  ],
  ...
]

User's balances are stored in a two-dimentional JSON array. Each sub-array represents the user address and his account balance, respectively.

Like the previous step, you can replace this file with your own version before running the `make` command.

## Step 3: Nodes reactivation: start of generation and validation of the new blocks

By default, the forked chain will start with only one validator account `//Alice`.

$ coldstack --chain=./build/fork.json --tmp --validator --alice --rpc-cors=all

This is great for testing purposes, but in production you may want to replace it with something else, or increase the validator count.

In order to customize the initial validator set, you will need to edit the `genesis.runtime.session.keys.json` file:

[
  [
    "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", //Alice stash account
    "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", //Alice controller account
    {
      "aura": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", //Alice aura key
      "grandpa": "5FA9nQDVg267DEd8m1ZypXLBnvN7SFxYwV7ndqSYGiN9TTpu" //Alice grandpa key
    }
  ]
]

and `genesis.runtime.validatorSet.validators.json`:

[
  "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY" //Alice stash account
]

rebuild the chain spec by running `make`, then insert the keys to the keystore, and restart your node for the key insertion to take effect.

### Example

For example, if you want to launch the network with two validator nodes:

Node1:
  stash: 5FQzhjQxFbSXoMNeKJbAWb9byoPoyi4Mdh2pTJ4uQQ9GyeP2
  controller: 5FWLtuYrv8hmeeEJmWLxKbsn9SUqQvWikJWU96mBZojcfaJU
  aura: 5ENm4MUUgA1ucmjyVdz1q5xyKwe2pge5cR29PfqeEmfCkAU5
  grandpa: 5FnUQ7qjyE6pPKxfdmwz2Y1HEFBMgpNbyKg1ZGzTHG9MbcxW
Node2:
  stash: 5F27QzgngoXGxVk8op4mT3CK74rtn86rGvQSTAXS8j6WoC5w
  controller: 5DUp5fnDJAzTTZgw5ejKFbNHDsPzz3aLi6tMKwytYNH7oFQE
  aura: 5FbWjav6m79aXkEVfbLgwzcFjFB72ba8wwnBAcoB4trgTGDy
  grandpa: 5H5uXgwczdXjTydof1rb4tQaRdWVf6sKNiYEmaxsspmme2yU

You need to update the content of `genesis.runtime.session.keys.json` to

[
  [
    "5FQzhjQxFbSXoMNeKJbAWb9byoPoyi4Mdh2pTJ4uQQ9GyeP2",
    "5FWLtuYrv8hmeeEJmWLxKbsn9SUqQvWikJWU96mBZojcfaJU",
    {
      "aura": "5ENm4MUUgA1ucmjyVdz1q5xyKwe2pge5cR29PfqeEmfCkAU5",
      "grandpa": "5FnUQ7qjyE6pPKxfdmwz2Y1HEFBMgpNbyKg1ZGzTHG9MbcxW"
    }
  ],
  [
    "5F27QzgngoXGxVk8op4mT3CK74rtn86rGvQSTAXS8j6WoC5w",
    "5DUp5fnDJAzTTZgw5ejKFbNHDsPzz3aLi6tMKwytYNH7oFQE",
    {
      "aura": "5DUp5fnDJAzTTZgw5ejKFbNHDsPzz3aLi6tMKwytYNH7oFQE",
      "grandpa": "5H5uXgwczdXjTydof1rb4tQaRdWVf6sKNiYEmaxsspmme2yU"
    }
  ]
]

and `genesis.runtime.validatorSet.validators.json` to

[
  "5FQzhjQxFbSXoMNeKJbAWb9byoPoyi4Mdh2pTJ4uQQ9GyeP2",
  "5F27QzgngoXGxVk8op4mT3CK74rtn86rGvQSTAXS8j6WoC5w"
]

Don't forget to regenerate the chain spec with `make` command

Now follow https://docs.substrate.io/tutorials/build-a-blockchain/add-trusted-nodes/#add-keys-to-the-keystore to insert the keys to your nodes:

Node1:

(NODE1) $ coldstack key insert --chain build/fork.json -d db1 --key-type gran --scheme ed25519 --suri ...
(NODE1) $ coldstack key insert --chain build/fork.json -d db1 --key-type aura --scheme sr25519 --suri ...

Node2:

(NODE2) $ coldstack key insert --chain build/fork.json -d db2 --key-type gran --scheme ed25519 --suri ...
(NODE2) $ coldstack key insert --chain build/fork.json -d db2 --key-type aura --scheme sr25519 --suri ...

Then start both nodes and they will start producing blocks

$ coldstack --chain build/fork.json -d db1 --validator --pruning=archive --name Node1
$ coldstack --chain build/fork.json -d db2 --validator --pruning=archive --name Node2

# Appendix

## Generating node keys

For each validator node, you will need two types of keys: `gran` (ed25519), `aura` (sr25519).

You can use any valid bip39 mnemonic phrases (arbitrary derivation paths are also allowed) as node keys, which can be generated by

$ coldstack key generate --output-type json | jq .secretPhrase -r
amateur prize version fee hold twist expect remind cash casual route dumb

It is possible to use the same mnemonic phrases to derive both keys, for example:

gran: `5HaKu7vUctrLG1yxtbpx4vPjobzStFcd6wN4JN8LY5xd1ujb`

$ coldstack key inspect --scheme ed25519 "amateur prize version fee hold twist expect remind cash casual route dumb"
Secret phrase `amateur prize version fee hold twist expect remind cash casual route dumb` is account:
  Secret seed:       0x93b4f944f25779a32f470dc9597f821a6dc4806e2eb4b96cd902dc6bb44f51cb
  Public key (hex):  0xf3c49cf25b7b2bedf4fd6775466bb3e82585f537be18cb46b091bf6284e1f04b
  Public key (SS58): 5HaKu7vUctrLG1yxtbpx4vPjobzStFcd6wN4JN8LY5xd1ujb
  Account ID:        0xf3c49cf25b7b2bedf4fd6775466bb3e82585f537be18cb46b091bf6284e1f04b
  SS58 Address:      5HaKu7vUctrLG1yxtbpx4vPjobzStFcd6wN4JN8LY5xd1ujb

aura:`5Gq7vMNC6zvoSrZq9jnJsLi5KkjXHYAtLSboWpgcesK5RzCN`

$ coldstack key inspect --scheme sr25519 "amateur prize version fee hold twist expect remind cash casual route dumb"
Secret phrase `amateur prize version fee hold twist expect remind cash casual route dumb` is account:
  Secret seed:       0x93b4f944f25779a32f470dc9597f821a6dc4806e2eb4b96cd902dc6bb44f51cb
  Public key (hex):  0xd2d0bc904cb628cae073e7973c94ce6a7c15760a8d6e543dacd81ec860aba721
  Public key (SS58): 5Gq7vMNC6zvoSrZq9jnJsLi5KkjXHYAtLSboWpgcesK5RzCN
  Account ID:        0xd2d0bc904cb628cae073e7973c94ce6a7c15760a8d6e543dacd81ec860aba721
  SS58 Address:      5Gq7vMNC6zvoSrZq9jnJsLi5KkjXHYAtLSboWpgcesK5RzCN

You can then follow [step 3](#Step-3-Nodes-reactivation-start-of-generation-and-validation-of-the-new-blocks) to add them to your chain spec json file

## Inserting node keys to keystore

You can use the `coldstack key insert` subcommand to insert the generated keys to the keystore of your node. By default the keystore directory is located at `$HOME/.local/share/coldstack/chains/local_testnet/keystore`

To insert the `gran` key, run:

$ coldstack key insert --keystore-path $KEYSTORE_PATH --key-type gran --suri "amateur prize version fee hold twist expect remind cash casual route dumb"

To insert the `aura` key, run:

$ coldstack key insert --keystore-path $KEYSTORE_PATH --key-type aura --suri "amateur prize version fee hold twist expect remind cash casual route dumb"

You need to restart the node to reload the inserted keys.

## Detailed guide for inserting node keys

Let's say you have four nodes: A B C D, the keystore path is located at `~/.coldstack/keystore`

You need to execute the following commands in order to setup the keystore (assuming node A, B are on the `65.108.7.232` machine, and C, D on `65.21.94.81`):

# ssh root@65.108.7.232

$ coldstack key insert --keystore-path ~/.coldstack/node-A/keystore --key-type gran --suri "gran key of A" --scheme ed25519
$ coldstack key insert --keystore-path ~/.coldstack/node-A/keystore --key-type aura --suri "aura key of A" --scheme sr25519

$ coldstack key insert --keystore-path ~/.coldstack/node-B/keystore --key-type gran --suri "gran key of B" --scheme ed25519
$ coldstack key insert --keystore-path ~/.coldstack/node-B/keystore --key-type aura --suri "aura key of B" --scheme sr25519

# ssh root@65.21.94.81

$ coldstack key insert --keystore-path ~/.coldstack/node-C/keystore --key-type gran --suri "gran key of C" --scheme ed25519
$ coldstack key insert --keystore-path ~/.coldstack/node-C/keystore --key-type aura --suri "aura key of C" --scheme sr25519

$ coldstack key insert --keystore-path ~/.coldstack/node-D/keystore --key-type gran --suri "gran key of D" --scheme ed25519
$ coldstack key insert --keystore-path ~/.coldstack/node-D/keystore --key-type aura --suri "aura key of D" --scheme sr25519