# September 14th - Post Mortem *Greetings, Citizens of XYM City!* ## Finalization Background Finality is an important concept in blockchain - it's an assurance that a transaction cannot be *altered, reversed, or canceled.* Most chains (like Bitcoin) utilize *probablistic finality* - that is, the chances that your transaction can be modified lowers as time passes. In Symbol we have both probalistic finality *and* we have *deterministic finality* as a seperate gadget - a transaction is considered to be final once it has been included in a finalized block. In order to finalize a block, `2/3` of the active voting stake must vote on the block's hash. Voters are allowed to enter and exit the finalization process at will. Any membership changes are effective for the *next* epoch. In addition, the active voting stake is calculated at the end of each epoch and that amount is used as the denominator in the next epoch. ## Problem On Sunday, September 12th we noticed the network having trouble with finalizing epoch 361. This was not entirely surprising - we had expected this epoch to be a strong stress test for the finalization process. Voting nodes that used `symbol-bootstrap` to register voting keys near the start of the network had registered keys for epochs 1-360. Due to lack of publicity around the importance of regularly updating voting keys, many users didn't register keys for epochs 361+. As a result, a lot of nodes voting in epoch 360 were not voting in epoch 361. Nonetheless, there was still enough agreement to proceed to the next epoch (362). Unfortunately, here's where the stall happened. A few more larger nodes stopped voting and, now, the network is unable to achieve agreement of a supermajority of voting stake. It appeared that the stake of expired voters was being considered, which was undesirable. After an investigation, we discovered that although the client was properly ignoring the stake of accounts voting keys *ever* registered, it was not ignoring the stake of accounts with voting keys *previously* registered. We were able to quickly identify something was a miss by looking at the data stored in the ImportanceBlocks corresponding to epochs 360 and 361. If everything was working as designed, we would have expected to see a large decrease in `TotalVotingBalance`, but instead it was mostly unchanged. ## Primary Fix We fixed this bug by filtering out accounts without voting keys registered at the current epoch. In addition, we lowered the supermajority threshold from 70% to 67%. With these changes, we expect finalization to resume as soon as this fix is deployed to a supermajority of voting stake. There will be a hardfork at block `528,000`. This is needed to fix the calculation of statistics stored in ImportanceBlocks because need to be validated by all nodes. Since finalization is implemented as a gadget in Symbol, the chain still progressed during this finalization stall. PoS+ block production continued on independent of finalization. When finalization is fixed, we expect all blocks to be finalized quickly. ## Secondary Changes When a voting node runs out of voting keys, the original behavior crashes the node. In order to make this less disruptive, we made a change to simply log an error. While this is less noisy, it is not recommended to run a node in this state for a long period of time. Instead nodes that want to stop voting should set `enableVoting` to `false` and reboot. There have been some complaints that a multisig account always needs another account (the initiator) to pay its fee. We are relaxing this constraint so that, at the fork block, the multisig account can pay its own fee indirectly by transferring its fee to the initiator. Please note that this doesn't change the behavior of partial transaction submissions that require a hash lock. ## Future Work In the current version of Symbol, there are no rewards or penalties for voting nodes. While this simplified our finalization implementation considerably for launch, it doesn't create the correct incentives that we intend for voting nodes. We will be adding a proper system to reward voters in the (near) future. We have realized we need to invest in better tooling. Being able to monitor the number of *active* voting nodes and a decline in voting stake would have alerted us to an issue before finalization stalled. We'll be prioritizing our infrastructure for the next few months, including a new block explorer, improved REST API's, and data analytics that are easily understandable and helpful to users and researchers alike. If you'd like to work with us, come join us at [Discord](https://www.discord.gg/xymcity). ## Translations * Check out our Japanese translation [here](https://hackmd.io/@xymbassador/B1MeARCfK), by [Radio-Radio](https://twitter.com/RadioRa26841511). * Check out our Spanish translation [here](https://hackmd.io/@xymbassador/Hy-xlyJXK), by [OldSkull](https://twitter.com/todo_nem). * Check out our Chinese translation [here](https://hackmd.io/@DragonChris/Bk8JbCCfF), by [DragonChris](@NEM_Taiwanese). * Check out our Italian translation [here](https://hackmd.io/@alediemmee/BksRxN1QF), by Alediemee. * Check out our Russian translation [here](https://hackmd.io/@xymbassador/BJb0mnk7K), by [AngelNEM](https://twitter.com/bestgoldangel) & [Klim](https://twitter.com/GeranKlim). ## Further Information Follow [Gimre](https://twitter.com/NCOSIGIMCITYNRE), [Hatchet](https://twitter.com/0x6861746366574) and [Jaguar](https://twitter.com/Jaguar0625) on Twitter for further Symbol (**XYM**) and NEM (**XEM**) updates. ###### tags: `Engineering` `Fork`
Last changed by  
 
The Symbol Syndicate
We're the team behind Symbol, a next generation protocol for market makers. Join us.
1531

Read more

Ramblings about public keys that were supposed to be twitter thread

Hello, @gimre here. We should probably change the way we display public keys in all possible applications/tools/etc. But before I dive into this, let's refresh your memory about addresses. This part is Symbol oriented, but it's also true for NEM. Addresses (again) Addresses in Symbol consist of 3 parts: 1-byte network prefix (0x68 byte in mainnet)

5/11/2023
2023 02 27 NEM Stall Post Mortem

Observation After the production of block 4129631, block production on the NEM network slowed to a crawl. This situation was instigated by an unexpected transaction entering the Unconfirmed Transactions cache. When this transaction expired, block production went back to normal. We have identified the makeup of the unexpected transaction and how it entered the Unconfirmed Transactions cache. A multisig account configured with 1-of-2 multisig sent two conflicting transactaions. Assume that the two cosigners are named A and B and the multisig account is named M. Root Cause Analysis The following sequence of events occured:

3/22/2023
A Guide to Creating a Symbol Test Network

Before any changes are deployed to Symbol mainnet or a new project gets launches on the blockchain, a test version is deployed to testnet first. Symbol test network(testnet) simulates the Symbol main network which gives developers and the community a chance to test features before real assets are used. Sai Testnet is the current testnet for Symbol. You can add a node and get test tokens from the Faucet. The Faucet allow an account to request up to 10k in test xym. If more token are needed to test or run a voting node, you can be requested from the Symbol's helpdesk on discord. It can be fun to have 3 or 4 million of test xym to splurge a bit, even though they carry no real-world value. Why should you create your own dev testnet? Creating and maintaining a Symbol testnet requires some work like certs and voting key renewals, so why bother setting up your own? These are some reasons you would want to setup your own dev testnet. Test Scenario - you might want to run specific test scenarios for your application. Let's say you want to fork the network and stall finalization to verify that your application also stops accepting deposits and withdrawals. Persistence - there is no guarantee that the testnet won't be reset. Symbol testnet is reset at least once a year to keep storage low. But it can also be reset if a major feature stops working. If your application requires the data kept for a longer time then a dev testnet would be better. Control Funds - your use case might require a large sum of funds like 500 Million xym. Even though there is the Faucet, requesting that much token is best done on dev testnet.

12/21/2022
Symbol - October 25th, Post Mortem

Greetings, Citizens of XYM City! Find the Japanese translation here. The Discovery On October 13th, Toshi reported some unexpected behavior to us on Discord in the #sdk-js channel. He had been comparing the outputs of the deprecated TypeScript SDK to the newer JavaScript SDK and noticed that the aggregate transaction hashes were being calculated differently. That...was not expected behavior. Even more concerning was the fact that they were both being accepted (and confirmed) by the network. Root Cause The team quickly identified two critical bugs - one in Catapult (the Symbol client), and one in the TypeScript and Java SDKs.

10/26/2022
Read more from The Symbol Syndicate
Published on HackMD