# September 14th - Post Mortem

*Greetings, Citizens of XYM City!*

## Finalization Background

Finality is an important concept in blockchain - it's an assurance that a transaction cannot be *altered, reversed, or canceled.* Most chains (like Bitcoin) utilize *probablistic finality* - that is, the chances that your transaction can be modified lowers as time passes. In Symbol we have both probalistic finality *and* we have *deterministic finality* as a seperate gadget - a transaction is considered to be final once it has been included in a finalized block.

In order to finalize a block, `2/3` of the active voting stake must vote on the block's hash. Voters are allowed to enter and exit the finalization process at will. Any membership changes are effective for the *next* epoch. In addition, the active voting stake is calculated at the end of each epoch and that amount is used as the denominator in the next epoch.

## Problem

On Sunday, September 12th we noticed the network having trouble with finalizing epoch 361. This was not entirely surprising - we had expected this epoch to be a strong stress test for the finalization process.

Voting nodes that used `symbol-bootstrap` to register voting keys near the start of the network had registered keys for epochs 1-360. Due to lack of publicity around the importance of regularly updating voting keys, many users didn't register keys for epochs 361+. As a result, a lot of nodes voting in epoch 360 were not voting in epoch 361.

Nonetheless, there was still enough agreement to proceed to the next epoch (362). Unfortunately, here's where the stall happened. A few more larger nodes stopped voting and, now, the network is unable to achieve agreement of a supermajority of voting stake.

It appeared that the stake of expired voters was being considered, which was undesirable. After an investigation, we discovered that although the client was properly ignoring the stake of accounts voting keys *ever* registered, it was not ignoring the stake of accounts with voting keys *previously* registered. We were able to quickly identify something was a miss by looking at the data stored in the ImportanceBlocks corresponding to epochs 360 and 361. If everything was working as designed, we would have expected to see a large decrease in `TotalVotingBalance`, but instead it was mostly unchanged.

## Primary Fix

We fixed this bug by filtering out accounts without voting keys registered at the current epoch. In addition, we lowered the supermajority threshold from 70% to 67%. With these changes, we expect finalization to resume as soon as this fix is deployed to a supermajority of voting stake.

There will be a hardfork at block `528,000`. This is needed to fix the calculation of statistics stored in ImportanceBlocks because need to be validated by all nodes.

Since finalization is implemented as a gadget in Symbol, the chain still progressed during this finalization stall. PoS+ block production continued on independent of finalization. When finalization is fixed, we expect all blocks to be finalized quickly.

## Secondary Changes

When a voting node runs out of voting keys, the original behavior crashes the node. In order to make this less disruptive, we made a change to simply log an error. While this is less noisy, it is not recommended to run a node in this state for a long period of time. Instead nodes that want to stop voting should set `enableVoting` to `false` and reboot.

There have been some complaints that a multisig account always needs another account (the initiator) to pay its fee. We are relaxing this constraint so that, at the fork block, the multisig account can pay its own fee indirectly by transferring its fee to the initiator. Please note that this doesn't change the behavior of partial transaction submissions that require a hash lock.

## Future Work

In the current version of Symbol, there are no rewards or penalties for voting nodes. While this simplified our finalization implementation considerably for launch, it doesn't create the correct incentives that we intend for voting nodes. We will be adding a proper system to reward voters in the (near) future.

We have realized we need to invest in better tooling. Being able to monitor the number of *active* voting nodes and a decline in voting stake would have alerted us to an issue before finalization stalled. We'll be prioritizing our infrastructure for the next few months, including a new block explorer, improved REST API's, and data analytics that are easily understandable and helpful to users and researchers alike. If you'd like to work with us, come join us at [Discord](https://www.discord.gg/xymcity).

## Translations

* Check out our Japanese translation [here](https://hackmd.io/@xymbassador/B1MeARCfK), by [Radio-Radio](https://twitter.com/RadioRa26841511).
* Check out our Spanish translation [here](https://hackmd.io/@xymbassador/Hy-xlyJXK), by [OldSkull](https://twitter.com/todo_nem).
* Check out our Chinese translation [here](https://hackmd.io/@DragonChris/Bk8JbCCfF), by [DragonChris](@NEM_Taiwanese).
* Check out our Italian translation [here](https://hackmd.io/@alediemmee/BksRxN1QF), by Alediemee.
* Check out our Russian translation [here](https://hackmd.io/@xymbassador/BJb0mnk7K), by [AngelNEM](https://twitter.com/bestgoldangel) & [Klim](https://twitter.com/GeranKlim).

## Further Information

Follow [Gimre](https://twitter.com/NCOSIGIMCITYNRE), [Hatchet](https://twitter.com/0x6861746366574) and [Jaguar](https://twitter.com/Jaguar0625) on Twitter for further Symbol (**XYM**) and NEM (**XEM**) updates.

###### tags: `Engineering` `Fork`