# A Guide to Creating a Symbol Test Network Before any changes are deployed to Symbol mainnet or a new project gets launches on the blockchain, a test version is deployed to testnet first. Symbol test network(testnet) simulates the Symbol main network which gives developers and the community a chance to test features before real assets are used. Sai Testnet is the current testnet for Symbol. You can add a node and get test tokens from the Faucet. The [Faucet](https://testnet.symbol.tools) allow an account to request up to 10k in test xym. If more token are needed to test or run a voting node, you can be requested from the Symbol's helpdesk on [discord](https://discord.com/invite/xymcity). It can be fun to have 3 or 4 million of test xym to splurge a bit, even though they carry no real-world value. ## Why should you create your own dev testnet? Creating and maintaining a Symbol testnet requires some work like certs and voting key renewals, so why bother setting up your own? These are some reasons you would want to setup your own dev testnet. * Test Scenario - you might want to run specific test scenarios for your application. Let's say you want to fork the network and stall finalization to verify that your application also stops accepting deposits and withdrawals. * Persistence - there is no guarantee that the testnet won't be reset. Symbol testnet is reset at least once a year to keep storage low. But it can also be reset if a major feature stops working. If your application requires the data kept for a longer time then a dev testnet would be better. * Control Funds - your use case might require a large sum of funds like 500 Million xym. Even though there is the Faucet, requesting that much token is best done on dev testnet. * Hardware requirements - using the Symbol testnet requires your node to sync with the blockchain. The storage requirements will be higher than simply rolling our own. ## Creating a new Symbol testnet For each Symbol test network, there are one or more properties in the ``config-network.properties`` file which needs to be unique. All the network properties can be found on the Symbol's [docs site](https://docs.symbol.dev/guides/network/configuring-network-properties.html#config-network-properties). * generationHashSeed: The nemesis generation hash seed used for signing all tx. This should be unique in a new network. * nemesisSignerPublicKey: The public key of the nemesis signer. * currencyMosaicId and harvestingMosaicId is generated from the nemesis signer. * harvestNetworkFeeSinkAddress and harvestNetworkFeeSinkAddressV1: Address of the harvest network fee sink account. * mosaicRentalFeeSinkAddress and mosaicRentalFeeSinkAddressV1: Address of the mosaic rental fee sink account. * namespaceRentalFeeSinkAddress and namespaceRentalFeeSinkAddress: Address of the namespace rental fee sink account. ### Required tools to automate the setup a dev testnet * catapult.tools.nemgen - available in the docker image * Symbol Python SDK 3.x * Symbol node resources - used by nemgen. ### Create testnet nemesis seed The nemesis block contains the initial accounts and assets that is required to start the network. Some of these accounts will be harvester to create blocks others might be voters to vote on finalization. Below is a network template that describes the accounts in the nemesis. Note: The host information is not required to create the nemesis seed. ``` network: testnet nodes: - host: host001.test.net friendly_name: host001 mode: dual amount: 3000000000000 roles: - harvester - voter - host: host002.test.net friendly_name: host002 mode: peer amount: 3000000000000 roles: - voter - host: host003.test.net friendly_name: host003 mode: dual amount: 100000000000 roles: - harvester accounts: - amount: 100000000000 count: 5 - amount: 3000000000000 count: 5 total_supply: 7842928625000000 ``` Nemgen needs account address, key pairs and assets to create the nemesis seed. We will create an generator using the Symbol Python SDK. The network generator will read the network template file and creates a new yaml with the required key pairs and assets to create the nemesis. Below is the main logic for the network generator. A key pair is created for each account which is the main private/public keys. For each role associated with an account, a handler will add the other metadata the required. ``` def _create_keys(self): key_pair = self.facade.KeyPair(PrivateKey.random()) return { 'privatekey': str(key_pair.private_key), 'publickey': str(key_pair.public_key), 'address': str(self.facade.network.public_key_to_address(key_pair.public_key)) } def _add_keys(self, account, name): account[name] = self._create_keys() def _create_account(self, amount, roles): account = self._create_keys() account.update({'amount': amount}) if roles: account.update({'roles': roles}) return account def _create_node_account(self, node): node.update({'main': self._create_account(node['amount'], [])}) if 'harvester' in node['roles']: self._add_keys(node, 'remote') self._add_keys(node, 'vrf') if 'voter' in node['roles']: self._add_keys(node, 'voting') return node def create_accounts(self, amount, count, roles): return [self._create_account( amount, roles ) for _ in range(0, count)] ``` The network generator output the yaml file below. I only added the first node since the others are the similar except with different key pairs. ``` network: testnet generation_hash_seed: 15609F28EE0E282ABF2105A5475313DF6FBACEA22B99EDD351672DB9351B5121 total_supply: 7842928625000000 currency_mosaic_id: '0x1BE3064212EBCB90' epoch_adjustment: 1671429771s nemesis: privatekey: D3BBBBA1D4BDF7834ED92D275E09E97A69DADCB98ACABB9404E3AB3D01688663 publickey: F4885DF48A71E634D2E0F3BFBB54EF3E7FF3D73EFBB7C7F10E641F8AD2AA99F1 address: TDDKGFAET5VIRX6CVJR46C3CJVTTOWYCFD3WA3I nodes: - host: host001.test.net friendly_name: host001 mode: dual amount: 3000000000000 roles: - harvester - voter main: privatekey: BEF08CEFF5A2A847FBDD2A39CA6F813D926EB41BAB93F331C9703ACB284EC336 publickey: 42F8F92259ECF5F4D5BA3F0E99C532DDDB90CFEABFBFB5F92F2DB7B2D6B98CA5 address: TCI6CN5WGZAVLQBJKYBIHRMIPPWWD5FZMAI54HA amount: 3000000000000 remote: privatekey: 94A0D302C2C70BEE0A324612211E27026BDBAEE7BE5201E3680AC8D249DAAAC0 publickey: C1FF6A475ABBD86CA6BFC0BE3B654BAA0DDC69F9BF7B30B908B277F2EC329D2D address: TBZT2JZPVHJKYN6YLM54B2V5HBSNAAG6IVI2EBI vrf: privatekey: 4C6E3D32B0FB727652FA3D2FC86D587162B997766D64DE9DB0DDFF69A5808DE5 publickey: E802474E4576596B411954F7995A44F8957FC6B955CABA0A7C644006FC9DED8B address: TDDYZBZZM5LZGOVIHFFHVA7BDDI6HO4XZX7RYNY voting: privatekey: 6A0150B41D4A9E795E2493C9E80788873287E817A5B14E85FBA7CCC9ADFB874B publickey: C3BDF8672762AB4FF706590A4C05DDF4BCD38FA927A7BCC672921D11D1575FE5 address: TCO3IMSPGS3IDEJZIVK4SKOU4KH7ZAUF6HOURVA ... accounts: - privatekey: 83D0155BCAD2BB2896B36C59DD6064277AEEEBDE785F673BB960A36A4D1612E3 publickey: 37F826D581C1397F01290C8400B1DF4AA04EB54267A1F82B3596955FE2A3BFF5 address: TAZUD3FGGMNQ4MRU6AIN6GSHU52DBJB2J5PD5EI amount: 100000000000 - privatekey: C68E1B16E347CF549524B72F5FD30E862233FF72CEC684A0A255311841528006 publickey: B03BB1FB9F361EDCBF82195DE2AC45DB3AF6D8235944D77080C41E97686D6819 address: TCDU332HMCQAWP3LIUIYL24D5EBUBXAWTKONEGQ amount: 100000000000 ... ``` Before we can run the nemgen tool to create the nemesis file, there are couple more things needed. 1. Configuration [properties file](https://github.com/symbol/symbol/blob/dev/client/catapult/tools/nemgen/resources/testnet.properties) for nemgen 2. Transaction files required for node setup - remote, vrf and voting links. 3. Update config-network.properties file in the Symbol node resources folder. #### Create the nemgen configuration file The Network.yaml has all the information needed to create the nemgen configuration file. The snippet of code below reads the Network yaml into the configuration object. This is then use to generate the nemgen configuration file. ``` def save_nemesis_configuration(self, output_filepath): accounts = '\n'.join( [f'{account["main"]["address"]} = {format_number_single_quote(account["amount"])}' for account in self.configuration['nodes']]) accounts += '\n' accounts += '\n'.join( [f'{account["address"]} = {format_number_single_quote(account["amount"])}' for account in self.configuration['accounts']]) configuration = f''' [nemesis] networkIdentifier = {self.configuration['network']} nemesisGenerationHashSeed = {self.configuration['generation_hash_seed']} nemesisSignerPrivateKey = {self.configuration['nemesis']['privatekey']} [cpp] cppFileHeader = [output] cppFile = binDirectory = ./seed [transactions] transactionsDirectory = ./transactions [namespaces] symbol = true symbol.xym = true [namespace>symbol] duration = 0 [mosaics] symbol:xym = true [mosaic>symbol:xym] divisibility = 6 duration = 0 supply = {format_number_single_quote(self.configuration['total_supply'])} isTransferable = true isSupplyMutable = false isRestrictable = false [distribution>symbol:xym] {accounts} ''' self._save_configuration_file(output_filepath, configuration) ``` This will create a nemgen configuration below. The main note here is that all the address under the ``distribution`` section, are the main key pair for each account that was created. These addresses will be converted to transfer transactions with the specified amount in the nemesis. ``` [nemesis] networkIdentifier = testnet nemesisGenerationHashSeed = 15609F28EE0E282ABF2105A5475313DF6FBACEA22B99EDD351672DB9351B5121 nemesisSignerPrivateKey = D3BBBBA1D4BDF7834ED92D275E09E97A69DADCB98ACABB9404E3AB3D01688663 [cpp] cppFileHeader = [output] cppFile = binDirectory = ./seed [transactions] transactionsDirectory = ./transactions [namespaces] symbol = true symbol.xym = true [namespace>symbol] duration = 0 [mosaics] symbol:xym = true [mosaic>symbol:xym] divisibility = 6 duration = 0 supply = 7'842'928'625'000'000 isTransferable = true isSupplyMutable = false isRestrictable = false [distribution>symbol:xym] TCI6CN5WGZAVLQBJKYBIHRMIPPWWD5FZMAI54HA = 3'000'000'000'000 TARZLLBVHJZB3XYKDXMS6JOAAM645PR2NPL34TA = 3'000'000'000'000 TCHHP5GF2AGHCTRTPQUYVYZDMNZHEMMQ6ZX5JPI = 100'000'000'000 TAZUD3FGGMNQ4MRU6AIN6GSHU52DBJB2J5PD5EI = 100'000'000'000 TCDU332HMCQAWP3LIUIYL24D5EBUBXAWTKONEGQ = 100'000'000'000 TCTIOEMNEJAJXWF75QGGQV6JM4DAL6YENMOXG2Y = 100'000'000'000 TBVLDD5ZY4776JPF7AVJZRQYFPIKJDZJTXBKOLY = 100'000'000'000 TACPMJ7J5RASCMFQT4RCXXADHPCS5X7ICGHE5XA = 100'000'000'000 TDQBOFCN7P7NNKZKULQ4UMBXDNKGDW2VFJZH3FA = 3'000'000'000'000 TAJGUDFCC6UI2OAVVUFWICWR7FR5NHEQKQMRIDQ = 3'000'000'000'000 TD4ZRFEQABZP2DPXOKCKQYKGNIW5HPKTFV3LW4I = 3'000'000'000'000 TD5S6LJEF7S6JFB7APNDYZSUMMG7A3GHTA5NZNA = 3'000'000'000'000 TAQ6RQCME4DWVLOI4AWCLKEK5S5GLEAQLTHGZLA = 3'000'000'000'000 TC44DTLHL43XSCCAJR5QI6B4J35G3QQ4WWCJU2Y = 0 TCHVSKDSYAEPITMHBFECE7HF3Y65IACRPQDQHPQ = 0 TAYKKEU6VR7IXQV536VQZLULI6FWSUF5G65UMDQ = 0 TBMQUG2N7B4DU7G7VUDTDL6EZTPNNXHASFCBG6I = 7'821'328'625'000'000 ``` Note: ``transactionsDirectory`` is the folder where the extra transactions to configure the nodes will be stored. #### Create the node's configuration transactons Lets create all the extra transactions(remote, vrf, voting) needed to setup each node and place them in the transactionsDirectory. The trick here is that all the transactions needs to be signed with the generation hash seed of the new network. After creating the facade for Symbol, the network information needs to be updated. ``` self.facade = SymbolFacade(self.configuration['network']) self.facade.network = Network( 'testnet', 0x98, Hash256(self.configuration['generation_hash_seed']) ) ``` Now that the SymbolFacade is created and updated to use the new network generation hash seed, creating the extra transactions for the nemesis is no different than setting up your current node on the Symbol blockchain. Note: for transaction in the nemesis, the deadline is 1 and fee is 0. ``` def _create_vrf_transaction(self, signer_public_key, account_descriptor): vrf_key_pair = self._get_key_pair_from_private_key(account_descriptor['vrf']['privatekey']) return self.facade.transaction_factory.create({ 'signer_public_key': signer_public_key, 'deadline': 1, 'type': 'vrf_key_link_transaction', 'linked_public_key': vrf_key_pair.public_key, 'link_action': 'link' }) def _create_account_key_link_transaction(self, signer_public_key, account_descriptor): remote_key_pair = self._get_key_pair_from_private_key(account_descriptor['remote']['privatekey']) return self.facade.transaction_factory.create({ 'signer_public_key': signer_public_key, 'deadline': 1, 'type': 'account_key_link_transaction', 'linked_public_key': remote_key_pair.public_key, 'link_action': 'link' }) def _create_voting_key_link_transaction(self, signer_public_key, account_descriptor): voting_key_pair = self._get_key_pair_from_private_key(account_descriptor['voting']['privatekey']) return self.facade.transaction_factory.create({ 'signer_public_key': signer_public_key, 'deadline': 1, 'type': 'voting_key_link_transaction', 'linked_public_key': voting_key_pair.public_key, 'start_epoch': 1, 'end_epoch': 720, 'link_action': 'link' }) def _create_transaction(self, transaction_type, account_descriptor, output_path): signer_key_pair = self._get_key_pair_from_private_key(account_descriptor['main']['privatekey']) transaction_factory = { 'vrf': self._create_vrf_transaction, 'remote': self._create_account_key_link_transaction, 'voting': self._create_voting_key_link_transaction } transaction = transaction_factory[transaction_type](signer_key_pair.public_key, account_descriptor) transaction.fee = Amount(0) signature = self.facade.sign_transaction(signer_key_pair, transaction) self.facade.transaction_factory.attach_signature(transaction, signature) transaction_buffer = transaction.serialize() transaction_hash = self.facade.hash_transaction(transaction) file_path = Path(output_path) / f'{transaction_type}_{transaction_hash}.bin' self._save_transaction_file(file_path, transaction_buffer) def create_nemesis_transactions(self, output_path): mkdirs(output_path) for node_descriptor in self.configuration['nodes']: if 'harvester' in node_descriptor['roles']: self._create_transaction('vrf', node_descriptor, output_path) self._create_transaction('remote', node_descriptor, output_path) if 'voter' in node_descriptor['roles']: self._create_transaction('voting', node_descriptor, output_path) ``` #### Update the Symbol node resources We are almost ready to run the nemgen. The last step is to update the Symbol node resources. We will create a patcher which updates the config-network.properties file to match the information of the new testnet network. ``` config = RawConfigParser(comment_prefixes=None, empty_lines_in_values=False, allow_no_value=True) config.optionxform = lambda option: option filename = f'{resources_folder}/resources/config-network.properties' config.read(filename) config['network']['identifier'] = account_config['network'] config['network']['nemesisSignerPublicKey'] = account_config['nemesis']['publickey'] config['network']['generationHashSeed'] = account_config['generation_hash_seed'] config['network']['epochAdjustment'] = account_config['epoch_adjustment'] config['chain']['currencyMosaicId'] = account_config['currency_mosaic_id'] config['chain']['harvestingMosaicId'] = account_config['currency_mosaic_id'] config['chain']['initialCurrencyAtomicUnits'] = format_number_single_quote(account_config['total_supply']) config['chain']['totalChainImportance'] = format_number_single_quote(account_config['total_supply']) harvest_address = get_account_address(get_account_with_role(account_config, 'harvest_network_fee_sink')) config['chain']['harvestNetworkFeeSinkAddressV1'] = harvest_address config['chain']['harvestNetworkFeeSinkAddress'] = harvest_address mosaic_address = get_account_address(get_account_with_role(account_config, 'mosaic_rental_fee_sink')) config['plugin:catapult.plugins.mosaic']['mosaicRentalFeeSinkAddressV1'] = mosaic_address config['plugin:catapult.plugins.mosaic']['mosaicRentalFeeSinkAddress'] = mosaic_address namespace_address = get_account_address(get_account_with_role(account_config, 'namespace_rental_fee_sink')) config['plugin:catapult.plugins.namespace']['namespaceRentalFeeSinkAddressV1'] = namespace_address config['plugin:catapult.plugins.namespace']['namespaceRentalFeeSinkAddress'] = namespace_address print(f'update network file: {filename}') with open(filename, 'wt', encoding='utf8') as output_file: config.write(output_file) ``` Note: Only the network config needs to be update for a new testnet. If you want to change any values in any other properties file is fine as long as its valid. #### Actual Nemesis Seeds Creation Now we are ready to run the nemgen tool. If you build the catapult client code from scratch, you can find the nemgen in the bin folder. I will take the easy route and run the tool from the latest docker image. ``` @staticmethod def generate_seed(output_path, config_path): cmd = [ 'docker', 'run', '--rm', f'--user={os.geteuid()}:{os.getgid()}', f'--volume={output_path}:/output', f'--volume={config_path}:/config', '-w=/output', '-e=LD_LIBRARY_PATH=/usr/catapult/lib:/usr/catapult/deps', 'symbolplatform/symbol-server:gcc-1.0.3.5', '/usr/catapult/bin/catapult.tools.nemgen', '--resources=/config', '--nemesisProperties=/output/block-properties-file.properties', '--useTemporaryCacheDatabase' ] dispatch_subprocess(cmd) ``` After this is complete the nemesis seed should be in the output folder. This can be used to bootstrap your test network. ### What's next? Now you have three nodes that can be setup to bootstrap your test network. Each node has 1. main account 2. linked transactions in the nemesis to allow either harvesting or voting. All that is left is to generate configuration for each node. There are tools that does this already like [shoestring](https://github.com/symbol/product/tree/shoestring/dev/tools/shoestring) so won't go over this here in details. But to give an idea of the steps 1. generate config for each node using the keys for each node 2. create voting files required 3. create the peer p2p and api json files 4. copy all to each node and start up Your testnet should be up and running in a couple of seconds, which you can confirm by navigating to http://localhost:3000.
Last changed by  
 
The Symbol Syndicate
We're the team behind Symbol, a next generation protocol for market makers. Join us.
572

Read more

Ramblings about public keys that were supposed to be twitter thread

Hello, @gimre here. We should probably change the way we display public keys in all possible applications/tools/etc. But before I dive into this, let's refresh your memory about addresses. This part is Symbol oriented, but it's also true for NEM. Addresses (again) Addresses in Symbol consist of 3 parts: 1-byte network prefix (0x68 byte in mainnet)

5/11/2023
2023 02 27 NEM Stall Post Mortem

Observation After the production of block 4129631, block production on the NEM network slowed to a crawl. This situation was instigated by an unexpected transaction entering the Unconfirmed Transactions cache. When this transaction expired, block production went back to normal. We have identified the makeup of the unexpected transaction and how it entered the Unconfirmed Transactions cache. A multisig account configured with 1-of-2 multisig sent two conflicting transactaions. Assume that the two cosigners are named A and B and the multisig account is named M. Root Cause Analysis The following sequence of events occured:

3/22/2023
Symbol - October 25th, Post Mortem

Greetings, Citizens of XYM City! Find the Japanese translation here. The Discovery On October 13th, Toshi reported some unexpected behavior to us on Discord in the #sdk-js channel. He had been comparing the outputs of the deprecated TypeScript SDK to the newer JavaScript SDK and noticed that the aggregate transaction hashes were being calculated differently. That...was not expected behavior. Even more concerning was the fact that they were both being accepted (and confirmed) by the network. Root Cause The team quickly identified two critical bugs - one in Catapult (the Symbol client), and one in the TypeScript and Java SDKs.

10/26/2022
OUR LANGUAGES

Throughout Symbol's development, we've used a large number of different languages. As part of regrouping and turning the ship, we've decided to cut back on the number of primary languages. Some choices will be less surprising than others. We hope this will facilitate hiring and improve intra-organizational mobility. We want to tear down those walls, Mr. Gorbachev! C++ catapult-client is the Symbol reference client and will continue to be developed and maintained in C++. We like C++ because it is a deterministic destruction language, which gives us fine grained control over memory usage. This allows us to optimize performance. C++ before and after 0x are almost different languages. Using RAII patterns, memory problems are almost almost a thing of the past.

1/10/2022
Read more from The Symbol Syndicate
Published on HackMD