# Calamari Runtime Upgrade Issue

## What went wrong?

When `Calamari` launched as a parachain, for safety, we disable most functionalities, including `KMA` token transfer.
The initial structure of `KMA` distribution is:
Treasury 1 (3/5 multisig): 1/3 of the distribution
Treasury 2 (3/5 multisig): 1/3 of the distribution
Treasury 3 (3/5 multisig): 1/3 of the distribution
Sudo (multisig): 25,000 `KMA`

We allocate 25,000 `KMA` for Sudo to pay gas fee. Looks perfect so far.

When we start the first runtime upgrade, we find that Calamari is in a limbo state that can never move forward. We need `Sudo` to do the first runtime upgrade (`sudo` will be removed once governance is online).However, despite there are balances in `Sudo`, all the controller accounts of `sudo` doesn't have any balance.

In `Substrate`, when using a multisig account to sign transactions, controller accounts need to pay gas fees in-order to sign the transactions. This essentially means our `sudo` is unusable unless we have `KMA` in the controller accounts, and at the same time, since we disabled token transfer, there is no way to transfer token to controller accounts unless we use `sudo`. This is a limbo state!

This is a clear oversight from our team. We did a lot of tests before the parachain is online, however, for convenience, our tests never use a multisig sudo (despite we indeed test the multisig is functional). The issue happens exactly we neglect the tests!

## How should we fix it?

(The plan is from Basti. Super grateful for the help!)

1. Ensure that the relay chain can send a super user XCM message (https://github.com/Manta-Network/Manta/blob/manta-pc/runtime/manta-pc/src/lib.rs#L490).

2. Prepare a hotfix runtime (https://github.com/Manta-Network/Manta/pull/227), this runtime is build on top of the old runtime, with just the OnRuntimeUpgrade added. It uses the OnRuntimeUpgrade implementation of the runtime transfers some funds to the required accounts.

3. Send the `AuthorizeUpgrade` via a XCM from the relay chain. This requires root of Kusama and thus, needs to be initiated by the council.

4. After the `AuthorizeUpgrade` hit the chain, use `EnactAuthorizedUpgrade` with an unsigned transaction to trigger the runtime upgrade.

5. Do a “proper” runtime upgrade that ensures that the relay chain doesn’t has super user anymore and other stuff.

| Item | Data | Remark |
| --------------- | -------------------------- | --------------|
| Preimage Hash of XCM Call| `0x3e85c678d51cb84b87c143b8467e3219d28480506b5145201d670f2d39e37f08` | [screen shot](https://drive.google.com/file/d/1W55iLpryd_8jL5ih3k45W4V43pC_MQed/view?usp=sharing) |
| Encoded call data for `authorizeUpgrade` | `0x0103991ac477a220895061b0264eed9cb7232fbe070be79336832914da8f8a3a24dc` | [screen shot](https://drive.google.com/file/d/1LNW_tUBEK8AlVanJtFxie2MDyXifmgI1/view?usp=sharing) |
| Calamri Hotfix Runtime | [download](https://github.com/Manta-Network/Manta/releases/download/v3.0.2-hotfix.calamari/calamari-runtime-v2.compact.compressed.wasm) | SHA256: `0x61b342a87b6e8d7808346461ae98b183f85237c4151c4c1bb91de1f8806df74b` |

## Where we should do better?

This issue is clearly an oversight from the Manta team. What we learned is that we need to minize the gap between production environment and testing environtment. Any part of code (despite how trivial it is may looks at), needs to be tested and audited by 3-rd party before goes into production. We learned the lesson in a hard way, and should do better in the future development.