# Storage Migration From A Solo Chain To A Parachain.

## Data Storage In Substrate

Substrate uses a simple key-value data store implemented as a database-backed, modified Merkle tree. All of Substrate's higher-lever storage abstractions are built on top of this simple key-value store.
Key-Value database

Substrate implements its storage database with RocksDB, a persistent key-value store for fast storage environments. It also supports an experimental Parity DB.

### Storage Value Keys
To calculate the key for a simple Storage Value, take the TwoX 128 hash
of the name of the pallet that contains the Storage Value and append to it the TwoX 128 hash of the name of the Storage Value itself. For example, the Sudo pallet exposes a Storage Value item named Key:

twox_128("Sudo")                  = "0x5c0d1176a568c1f92944340dbfed9e9c"
twox_128("Key")                    = "0x530ebca703c85910e7164cb7d1c9e47b"
twox_128("Sudo") + twox_128("Key") = "0x5c0d1176a568c1f92944340dbfed9e9c530ebca703c85910e7164cb7d1c9e47b"

### Storage Map Keys
Like Storage Values, the keys for Storage Maps are equal to the TwoX 128 hash of the name of the pallet that contains the map prepended to the TwoX 128 hash of the name of the Storage Map itself. To retrieve an element from a map, simply append the hash of the desired map key to the storage key of the Storage Map. For maps with two keys (Storage Double Maps), append the hash of the first map key followed by the hash of the second map key to the Storage Double Map's storage key.
The next example shows how to get the Free Balance from the pallet named "Balances" for the balance of the familiar Alice account. In this example, the FreeBalance map is using the transparent Blake2 128 Concat hashing algorithm.

twox_128("Balances")      = "0xc2261276cc9d1f8598ea4b6a74b15c2f"
twox_128("FreeBalance")   = "0x6482b9ade7bc6657aaca787ba1add3b4"
scale_encode("5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY") = "0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"

blake2_128_concat("0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d") = "0xde1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d"

state_getStorage("0xc2261276cc9d1f8598ea4b6a74b15c2f6482b9ade7bc6657aaca787ba1add3b4de1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d") = "0x0000a0dec5adc9353600000000000000"

Notice that before hashing Alice's account ID it has to be SCALE-encoded. Also notice that the output of the `blake2_128_concat` function consists of 32 hexadecimal characters followed by the function's input. This is because the Blake2 128 Concat is a transparent hashing algorithm.

In Substrate you have the flexibility to change which hasher gets used for each different storage map, and as a result, change the way your underlying storage keys are saved in the Substrate database.

`blake2_128_concat` should be the default choice for any storage key. Because the prefix of the final key uses Blake2, a cryptographically secure hashing algorithm, we need not worry about the content of the starting key. This hasher will work for anything.

/// The full account information for a particular account ID.
pub Account get(fn account):
  map hasher(blake2_128_concat) T::AccountId => AccountInfo<T::Index, T::AccountData>;

---
**NOTE**

Big thanks to the Centrifuge team for their great work on storage migration and their help to get this guide done.

---

### Deep Dive Into Storage Migration Video

["Sub0 Online: Storage Migration - Standalone chain to Parachain"](https://www.youtube.com/watch?v=sIgvyRFs-N4) by Frederik Schulz, Centrifuge.

## Steps To take Into Account For A Successful Migration.

1. Network Stability
After getting on-boarded into Polkadot wait a few days until your parachain is stable.
* Get your deployments and DevOps processes ready.
* Setup monitoring and alerting.
* Ensure block times are stable.
* Wallets supported and tested.
* Exchanges aligned.

2. Bootstrap Council
Update State with a copy of the Council from the solo chain:
* Select addresses of all current Council members of the solo chain.
* Either use sudo to set the new council members in the parachain or do a runtime upgrade.

3. State Migration
* Disabling transfers on solochain by using a filter.
* Take Snapshot and execute migration of data with Sudo.

4. Remove Sudo + Enable All Calls
* Prepare runtime upgrade that enables all calls and removes Sudo.
* Vote with fastrack of 3-24h to propose this upgrade.

5. Enable Claiming
* Prepare and propose Runtime Upgrade to enable crowdloan claiming.

## Migration Using Genesis State

This method is the easiest, but has the most downtime compared to key value storage migration in terms of parachain auctions.

One of the best examples to understand this approach would be to use a fork-off substrate.
This project shows how to extract all state from an existing blockchain and start a new chain from scratch. https://github.com/maxsam4/fork-off-substrate

While this approach sounds very desirable, it has more restrictions that will not make it the best option for parachains. This will work if you want to disable all transactions on your chain during the entire auction and crowdlending process. The actual state of your chain is required before any process above can be started, as it will be the initial state of your parachain.

## Migration Using Storage Keys

This method will be best suited for a solochain to parachain migration.

### Low level considerations:

The solo chain and the parachain should be running using the same commit. This prevents any possible issue with incompatible types. You can take a look to this example in case you need to migrate data types in your chain. [Data type migration docs.](https://docs.substrate.io/how-to-guides/v3/storage-migrations/basics/)

1. Disable All Transactions
This step requires adding filters to your chain to prevent any transaction execution. [BasecallFilter, Stack Exchange](https://substrate.stackexchange.com/questions/1119/how-to-set-up-basecallfilter-to-only-allow-a-limited-set-of-calls-of-one-palle)
* Disable all calls in with the Call Filter in the Solo-Chain
* (Optional) Add Code that disables blocks production in the future, breaking the solo chain.
* Your target runtime (parachain) must have a filter that disables all calls except sudo and runtime upgrade calls. You will remove this filter whenever you want to enable calls on the destination chain.

*Filter example for the target parachain*:

pub struct BaseFilter;

impl Contains<Call> for BaseFilter {
	fn contains(c: &Call) -> bool {
		matches!(
			c,
			// Calls from Sudo
			Call::Sudo(..)
			// Calls for runtime upgrade
			| Call::System(frame_system::Call::set_code{..})
			| Call::System(frame_system::Call::set_code_without_checks{..})
			// Calls that are present in each block
			| Call::ParachainSystem(
				cumulus_pallet_parachain_system::Call::set_validation_data{..}
			)
			| Call::Timestamp(pallet_timestamp::Call::set{..})
		)
	}
}

2. State Migration
This step is the most critical one, since it involves migrating the state from the current Standalone chain.
+ Before starting the migration, your runtime should have the pallets that will get the new storage. If you used the Shell or Seedling runtime, then do an upgrade to replace it your runtime with all the pallets.
+ Add Pallet Migration to your Runtime
+ Run Migration using Centrifuge-cli and choose the forked block.
* This script as well checks data consistency between both final states.
* Should be ran on a deployed host to maximize networking & computing stability.
3. Enable Most Calls
+ Prepare Code Remove Sudo pallet + Enable all calls
+ Checkpoint to halt block production on Standalone chain.
4. Enable Crowdloan Claims
* Add Crowdloan Pallet to enable claiming
5. Collators & Staking
Council will begin to add external collators and will enable staking.
* Implement Collator Selection as PoA

## Centrifuge-Cli For Storage Migration
https://github.com/centrifuge/centrifuge-cli

After a succesful build you should get the available commands by typing:

node ./packages/cli/bin/run.js --help

Output:

Centrifuge-cli tool. Allows to interact with different components of the centrifuge infrastructure and provides different tools as plugins.

VERSION
  @centrifuge-cli/cli/0.1.0 linux-x64 node-v16.14.0

USAGE
  $ centrifuge [COMMAND]

COMMANDS
  commands   list all the commands
  crowdloan  Centrifuge cli command that allows to fetch crowdloan contributions from a relay chain and initialize the crowdloan modules on the parachain
  fork       describe the command here
  help       display help for centrifuge
  launch     describe the command here
  migration  Centrifuge cli command that allows to migrate from a stand-alone to a parachain.

Most of the displayed commands are just placeholders as you can read on their description. The one we'll focus on is **migration**. It does a lot of cool stuff for us.

### Migration Command Flags

-b/--block : Defines the blocknumber the state is fetched at. If not defined latest is used.
--config : Defines the modules and the sequence of migration for them.
--creds : Credentials that will be used to execute the extriniscs. I.e. the root-key.
--verify : If flag is given, the migration will be verified directly afterwards.
--just-verify : This option expects a path to a json file of the following form in order to verify a past migration.
--finalize : If flag is present, then the call-filters will be disabled after the migration.

Before using the cli tool you need to have the migration pallet as part of the destination runtime. [Centrifuge's migration pallet.](https://github.com/centrifuge/centrifuge-chain/blob/04ebc0475a467af829cb3462c0271fdd8802f0a9/pallets/migration/src/lib.rs)
Since the tool was developed for Centrifuge's needs, it covers most of the important pallets, but it might be the case that you need to customize it for your specific use-case.

E.g. - Balance and Accounts migration between a local solo chain and a local parachain.

node ./packages/cli/bin/run.js migration ws://127.0.0.1:9944 ws://127.0.0.1:9946 -b 14956 --config ./migrations.json --creds ./credentials.json --verify

**migrations.json** - Set the pallets to be migrated and the sequence of how it will happen.

{
    "modules": [
        {
            "name": "Balances",
            "item": {
                "name": "TotalIssuance"
            }
        },
        {
            "name": "System",
            "item": {
                "name": "Account"
            }
        }
    ],
    "sequence": [
        {
            "name": "Balances",
            "item": "TotalIssuance"
        },
        {
            "name": "System",
            "item": "Account"
        }
    ]
}

**credentials.json** - Contains the seed from the SUDO account in the target chain.

{
    "rawSeed": "cheese lunch region spot fresh term glare oppose thing lonely excite inhale"
}

---

## E.g. Westend's Accounts and Balances, Migration and Verification.

Fetched keys: `7184`
Origin chain: `wss://westend-rpc.polkadot.io`
Chosen block: `6000000`
Destination chain: `ws://127.0.0.1:9946`
Duration: `03m, 35s`
Target test chain blocks time: `6s`

node --max-old-space-size=8192 ./packages/cli/bin/run.js migration wss://westend-rpc.polkadot.io ws://127.0.0.1:9946 -b 6000000 --config ./migration.json --creds ./credentials.json --verify

### Logs

2022-03-20 15:01:38.118  INFO  [Migration packages/plugins/migration/src/commands/migration.ts:26  Migration.run] Connecting to source network:  wss://westend-rpc.polkadot.io 
2022-03-20 15:01:38.940  INFO  [Migration packages/plugins/migration/src/commands/migration.ts:36  Migration.run] Connecting to destination network:  ws://127.0.0.1:9946 
2022-03-20 15:01:39.137  INFO  [Migration packages/plugins/migration/src/commands/migration.ts:48  Migration.run] Starting migration from stand-alone chain block with hash 0xbe3c54448de70f4f652a2dde1a9e2e8a88e0a116f1648ae182a44990730740b1 
2022-03-20 15:01:39.191  INFO  [Migration packages/plugins/migration/src/commands/migration.ts:53  Migration.run] Fetching storage from stand-alone chain block with hash 0x906fdd056f990beb81705be460cc5107845faecc3149d99f70026ca3b996e70f 
2022-03-20 15:01:41.607  INFO  [Migration packages/plugins/migration/src/commands/migration.ts:63  Migration.run] Starting migration from parachain block with hash 0xd2e0da971127890ef91d12dc544906abdd95dc86e29419cd753bff31705a24f1 
Fetching storage for prefix: 0xc2261276cc9d1f8598ea4b6a74b15c2f57c875e4cff74148e4628f264b974c80
Fetched keys: 1
Fetched storage values: 1/1
Fetching storage for prefix: 0x26aa394eea5630e07c48ae0c9558cef7b99d880ec681799c0cf30e8886371da9
Fetched keys: 7184
Fetched storage values: 7184/7184
No checks performed, due to unavailability of `dry_run()` on chain.
Sending with nonce 0, running 1 : migrate_balances_issuance
No checks performed, due to unavailability of `dry_run()` on chain.
Sending with nonce 1, running 2 : migrate_system_account
Sending with nonce 2, running 3 : migrate_system_account
Sending with nonce 3, running 4 : migrate_system_account
Sending with nonce 4, running 5 : migrate_system_account
Sending with nonce 3 is in Block/Finalized : migrate_system_account
Sudo ok: 3
Sending with nonce 2 is in Block/Finalized : migrate_system_account
Sudo ok: 2
Sending with nonce 1 is in Block/Finalized : migrate_system_account
Sudo ok: 1
Sending with nonce 4 is in Block/Finalized : migrate_system_account
Sudo ok: 4
Sending with nonce 0 is in Block/Finalized : migrate_balances_issuance
Sudo ok: 0
Waiting in perBlock... 5
Sending with nonce 5, running 1 : migrate_system_account
Sending with nonce 6, running 2 : migrate_system_account
Sending with nonce 7, running 3 : migrate_system_account
Sending with nonce 8, running 4 : migrate_system_account
Sending with nonce 9, running 5 : migrate_system_account
Sending with nonce 5 is in Block/Finalized : migrate_system_account
Sudo ok: 5
Sending with nonce 8 is in Block/Finalized : migrate_system_account
Sudo ok: 8
Sending with nonce 7 is in Block/Finalized : migrate_system_account
Sudo ok: 7
Sending with nonce 9 is in Block/Finalized : migrate_system_account
Sudo ok: 9
Sending with nonce 6 is in Block/Finalized : migrate_system_account
Sudo ok: 6
Waiting in perBlock... 10
Sending with nonce 10, running 1 : migrate_system_account
Sending with nonce 11, running 2 : migrate_system_account
Sending with nonce 12, running 3 : migrate_system_account
Sending with nonce 13, running 4 : migrate_system_account
Sending with nonce 14, running 5 : migrate_system_account
Sending with nonce 11 is in Block/Finalized : migrate_system_account
Sudo ok: 11
Sending with nonce 12 is in Block/Finalized : migrate_system_account
Sudo ok: 12
Sending with nonce 13 is in Block/Finalized : migrate_system_account
Sudo ok: 13
Sending with nonce 14 is in Block/Finalized : migrate_system_account
Sudo ok: 14
Sending with nonce 10 is in Block/Finalized : migrate_system_account
Sudo ok: 10
...
...
...
Waiting in perBlock... 85
Sending with nonce 85, running 1 : migrate_system_account
Sending with nonce 86, running 2 : migrate_system_account
Sending with nonce 87, running 3 : migrate_system_account
Sending with nonce 88, running 4 : migrate_system_account
Sending with nonce 89, running 5 : migrate_system_account
Sending with nonce 88 is in Block/Finalized : migrate_system_account
Sudo ok: 88
Sending with nonce 87 is in Block/Finalized : migrate_system_account
Sudo ok: 87
Sending with nonce 85 is in Block/Finalized : migrate_system_account
Sudo ok: 85
Sending with nonce 89 is in Block/Finalized : migrate_system_account
Sudo ok: 89
Sending with nonce 86 is in Block/Finalized : migrate_system_account
Sudo ok: 86
Waiting in perBlock... 90
Sending with nonce 90, running 1 : migrate_system_account
Sending with nonce 91, running 2 : migrate_system_account
Sending with nonce 92, running 3 : migrate_system_account
Sending with nonce 93, running 4 : migrate_system_account
Sending with nonce 94, running 5 : migrate_system_account
Sending with nonce 93 is in Block/Finalized : migrate_system_account
Sudo ok: 93
Sending with nonce 92 is in Block/Finalized : migrate_system_account
Sudo ok: 92
Sending with nonce 90 is in Block/Finalized : migrate_system_account
Sudo ok: 90
Sending with nonce 94 is in Block/Finalized : migrate_system_account
Sudo ok: 94
Sending with nonce 91 is in Block/Finalized : migrate_system_account
Sudo ok: 91
Waiting in perBlock... 95
Sending with nonce 95, running 1 : migrate_system_account
Sending with nonce 96, running 2 : migrate_system_account
Sending with nonce 97, running 3 : migrate_system_account
Sending with nonce 98, running 4 : migrate_system_account
Sending with nonce 99, running 5 : migrate_system_account
Sending with nonce 95 is in Block/Finalized : migrate_system_account
Sudo ok: 95
Sending with nonce 97 is in Block/Finalized : migrate_system_account
Sudo ok: 97
Sending with nonce 99 is in Block/Finalized : migrate_system_account
Sudo ok: 99
Sending with nonce 96 is in Block/Finalized : migrate_system_account
Sudo ok: 96
Sending with nonce 98 is in Block/Finalized : migrate_system_account
Sudo ok: 98
Waiting in perBlock... 100
Sending with nonce 100, running 1 : migrate_system_account
Sending with nonce 101, running 2 : migrate_system_account
Sending with nonce 102, running 3 : migrate_system_account
Sending with nonce 103, running 4 : migrate_system_account
Sending with nonce 104, running 5 : migrate_system_account
Sending with nonce 102 is in Block/Finalized : migrate_system_account
Sudo ok: 102
Sending with nonce 101 is in Block/Finalized : migrate_system_account
Sudo ok: 101
Sending with nonce 100 is in Block/Finalized : migrate_system_account
Sudo ok: 100
Sending with nonce 104 is in Block/Finalized : migrate_system_account
Sudo ok: 104
Sending with nonce 103 is in Block/Finalized : migrate_system_account
Sudo ok: 103
Waiting in perBlock... 105
Sending with nonce 105, running 1 : migrate_system_account
Sending with nonce 106, running 2 : migrate_system_account
Sending with nonce 107, running 3 : migrate_system_account
Sending with nonce 108, running 4 : migrate_system_account
Sending with nonce 107 is in Block/Finalized : migrate_system_account4 
Sudo ok: 107
Sending with nonce 106 is in Block/Finalized : migrate_system_account
Sudo ok: 106
Sending with nonce 105 is in Block/Finalized : migrate_system_account
Sudo ok: 105
Sending with nonce 108 is in Block/Finalized : migrate_system_account
Sudo ok: 108
2022-03-20 15:04:07.662  INFO  [Migration packages/plugins/migration/src/commands/migration.ts:95  Migration.run] Migration was successful. 
2022-03-20 15:04:07.674  INFO  [Migration packages/plugins/migration/src/commands/migration.ts:113  Migration.run] Ending migration on parachain block with hash 0x881e62b1d387dc14c7c17bca70baefe4fa232f9811db1c4092887a5c941c4796 
2022-03-20 15:04:07.883  INFO  [Migration packages/plugins/migration/src/commands/migration.ts:118  Migration.run] 

Verifying migration. This will take some time... 
Fetching storage for prefix: 0xc2261276cc9d1f8598ea4b6a74b15c2f57c875e4cff74148e4628f264b974c80
Fetched keys: 1
Fetched storage values: 1/1
Fetching storage for prefix: 0x26aa394eea5630e07c48ae0c9558cef7b99d880ec681799c0cf30e8886371da9
Fetched keys: 7184
Fetched storage values: 7184/7184
Fetching storage for prefix: 0xc2261276cc9d1f8598ea4b6a74b15c2f57c875e4cff74148e4628f264b974c80
Fetched keys: 1
Fetched storage values: 1/1
Fetching storage for prefix: 0x26aa394eea5630e07c48ae0c9558cef7b99d880ec681799c0cf30e8886371da9
Fetched keys: 7185
Fetched storage values: 7185/7185
Starting verification of 7185 migrated storage keys.
2022-03-20 15:04:13.937  INFO  [Migration packages/plugins/migration/src/commands/migration.ts:148  Migration.run] Migration has been verified. 

### Polkadot JS Apps Events


## Final Comments

* TEST TEST TEST your migration locally. You can query your existing public endpoint and migrate all data to your local parachain.
* For the tool to successfully validate the balances migration, it must use a sudo key that does not exist in the source chain. The reason is that sudo calls are free if they succeed, but if they fail, you have to pay. Before you get the result, you have to pay the TX fee, and then get a refund. The consequence of this is that if it is successful, the refunded fee will occur after the balance is set and this will create a small difference that the cli tool will show as an error.
* Related to the previous point, the sudo key should be the only key with any small balance in the target chain. This will create a very small difference in the total balance issuance in the destination chain compared to the source chain.
* During your tests, the target environment should be as close as possible to your production environment.
* Node's VM V8 has a limited amount of memory of around 1.7 GB. You can change the memory limit with a flag. In this example, I am setting it to 8 GB. `--max-old-space-size=8192`