01/31/24 Dora Vota Incident Report and Response Timelines

# 01/31/24 Dora Vota Incident Report and Response Timeline ## OVERVIEW Dora Vota is a special-purpose appchain for decentralized governance and public goods funding. The current Dora Vota blockchain (vota-ash network) was deployed in September 2023, and has served as a pre-mainnet for the upcoming Dora Vota launch. The current network has been supporting several applications, including [Dora Vota MACI](https://vota.dorafactory.org), [Mini Bounties](https://dorahacks.io/mini-bounty), and the recently launched [quadratic funding application](https://dorahacks.io/aez/round). As the network starts to host more applications and onboards more users, the plan is to gradually expand the network's validator set before the official mainnet launch. And the first validator set expansion was scheduled on Jan 31, 2024. On January 31, 2024 the vota-ash network halted at block height **2581230** around 1:25 PM UTC, immediately after the first delegation transaction. After that, **3 subsequent delegation transactions didn't go through, but no other committed transactions lost**. On February 1, 2024, **Dora Vota's DevOps team created a hard fork from block height 2581225** and resumed the network. ## WHAT HAPPENED On January 27 2024, an [on-chain proposal](https://vota-explorer.dorafactory.org/doravota/gov/6) to expand the validator set was submitted. There would be 8 more external validators to join. On January 31, a series of transactions for validator delegation were initiated. The appchain halted when executing the first delegation tx due to voting power overflow, which led to a consensus error. After some investigation, we identified that the voting power overflow was caused by an incorrect `DefaultPowerReduction` value, which is set to 10^6 by default. However, the token's decimal on the vota-ash network is 18, resulting in an overflow when total voting power is greater than a threshold (1,152,921.504606847 DORA on the vota-ash network). ## THE NETWORK HAS BEEN RESUMED In order to make sure that the ongoing on-chain activities are not affected, we decided to resume the network. The Dora Vota DevOps team hard forked the blockchain from block height #2581225. After the hard fork, the vota-ash network resumed and has been producing blocks correctly. No user transactions were lost (block height #2581225 - #2581229 were empty blocks). The resumed network still carries the old `DefaultPowerReduction` value. So we will make sure that the total voting power does not exceed the overflow amount until a fix is deployed on the network. Currently, we are testing posssible means to fix the voting power overflow issue without interrupting the vota-ash network. A [bug bounty](https://dorahacks.io/bugbounty/4) has also been released in case anyone from the community could help accelerate the fix. ## WHAT’S NEXT After the hard fork, the vota-ash network and all services on it were resumed. Dora Vota MACI, Mini Bounty smart contracts are fully functioning. Due to the hard fork, all IBC clients have to be reset. The CryptoCrew team worked with us to setup new IBC clients. Governance proposals to unfreeze and replace IBC clients on CosmosHub and Osmosis have been submitted and are now in voting period. The recently launched AEZ Quadratic Funding round's voting will be postponed until the vota-ash network further stabilises, see [this article](https://dorahacks.io/blog/aez-qf-temporary-pause). We have identified three crucial tasks that need to be completed before the network continues to host more applications: 1. Resolve the `DefaultPowerReduction` issue and address associated risks. 2. Onboard validators and complete the first round of validator set expansion. 3. Reactivate the IBC Channel between the vota-ash network and other networks. ## DETAILED RESPONSE TIMELINE On January 31, 2024 the validator set expansion proposal was about to pass, and a scheduled validator delegation program started. Immediately after the first delegation transaction, chain halted on block height #2581230 with a consensus error. The reason for this crash was that total voting power exceeded [MaxTotalVotingPower](https://github.com/cometbft/cometbft/blob/1f430f51f0e390cd7c789ba9b1e9b35846e34642/types/validator_set.go#L25). The default value of `DefaultPowerReduction` is 10^6 while the DORA denom precision is 10^18 which caused the calculation result of voting power to be magnified by a factor of 10^12. **1:25 PM UTC, January 31st** - An internal alert notified Vota developers that the chain halted and a consensus error occurred. Developers started a VC war room and immediately began triaging the source of the issue. **2:10 PM UTC, January 31st** - Developers identified the problem from the error log produced by the CometBFT validator module. After the latest delegation transaction was executed, voting power on the vota-ash network had an overflow. Developers found out where this constant was calculated and started to figure out ways to recover. **2:20 PM UTC, January 31st** - Dora Vota DevOps team notified all node operators about this issue. **8:00 PM UTC, January 31st** - A quick hot fix was created by changing `DefaultPowerReduction`. Deployed the hot fix on a separate testnet. The blockchain succeeded in producing new blocks but halted again when a new delegation executed. It didn't work. A [GitHub thread](https://github.com/cosmos/cosmos-sdk/issues/19321) was created and got feedback from Cosmos devs. ``` AM ERR CONSENSUS FAILURE!!! err="failed to apply block; error commit failed for application: error changing validator set: duplicate entry ``` **5:10 PM UTC, February 1st** - A hard fork was created and the appchain and rolled back to blockheight #2581225. Forked chain worked correctly from block 2581225. **2:00 AM UTC, February 2nd** - A regression test began on the recovered chain. All IBC channels were confirmed to be affected by the hard fork. **6:26 AM UTC, February 2nd** - A notice of chain hard fork has been sent to all validators. **7:30 AM UTC, February 2nd** - Started to work with the CryptoCrew team to recover all impacted IBC channels. Proposals to unfreeze and replace IBC client on [CosmosHub](https://www.mintscan.io/cosmos/proposals/877) and [Osmosis](https://www.mintscan.io/osmosis/proposals/722) have been submitted. Other proposals are in draft and will be submitted in the coming days. Currently, we are still working on solutions to fix `DefaultPowerReduction` and to migrate the voting power of all validators. All solutions will be tested on separate test networks. We will provide further updates on our progress on [Twitter](https://twitter.com/DoraFactory). We would love to thank the Cosmos community for the huge and continuous support after the incident. We look forward to working with more ecosystem partners to make Dora Vota more secure, robust, and fun!