A Guide to Creating a Symbol Test Network

Before any changes are deployed to Symbol mainnet or a new project gets launches on the blockchain, a test version is deployed to testnet first. Symbol test network(testnet) simulates the Symbol main network which gives developers and the community a chance to test features before real assets are used.

Sai Testnet is the current testnet for Symbol. You can add a node and get test tokens from the Faucet. The Faucet allow an account to request up to 10k in test xym. If more token are needed to test or run a voting node, you can be requested from the Symbol's helpdesk on discord. It can be fun to have 3 or 4 million of test xym to splurge a bit, even though they carry no real-world value.

Why should you create your own dev testnet?

Creating and maintaining a Symbol testnet requires some work like certs and voting key renewals, so why bother setting up your own? These are some reasons you would want to setup your own dev testnet.

• Test Scenario - you might want to run specific test scenarios for your application. Let's say you want to fork the network and stall finalization to verify that your application also stops accepting deposits and withdrawals.

• Persistence - there is no guarantee that the testnet won't be reset. Symbol testnet is reset at least once a year to keep storage low. But it can also be reset if a major feature stops working. If your application requires the data kept for a longer time then a dev testnet would be better.

• Control Funds - your use case might require a large sum of funds like 500 Million xym. Even though there is the Faucet, requesting that much token is best done on dev testnet.

• Hardware requirements - using the Symbol testnet requires your node to sync with the blockchain. The storage requirements will be higher than simply rolling our own.

Creating a new Symbol testnet

For each Symbol test network, there are one or more properties in the config-network.properties file which needs to be unique. All the network properties can be found on the Symbol's docs site.

• generationHashSeed: The nemesis generation hash seed used for signing all tx. This should be unique in a new network.
• nemesisSignerPublicKey: The public key of the nemesis signer.
• currencyMosaicId and harvestingMosaicId is generated from the nemesis signer.
• harvestNetworkFeeSinkAddress and harvestNetworkFeeSinkAddressV1: Address of the harvest network fee sink account.
• mosaicRentalFeeSinkAddress and mosaicRentalFeeSinkAddressV1: Address of the mosaic rental fee sink account.
• namespaceRentalFeeSinkAddress and namespaceRentalFeeSinkAddress: Address of the namespace rental fee sink account.

Required tools to automate the setup a dev testnet

• catapult.tools.nemgen - available in the docker image
• Symbol Python SDK 3.x
• Symbol node resources - used by nemgen.

Create testnet nemesis seed

The nemesis block contains the initial accounts and assets that is required to start the network. Some of these accounts will be harvester to create blocks others might be voters to vote on finalization. Below is a network template that describes the accounts in the nemesis.
Note: The host information is not required to create the nemesis seed.

network: testnet
nodes:
  - host: host001.test.net
    friendly_name: host001
    mode: dual
    amount: 3000000000000
    roles:
      - harvester
      - voter
  - host: host002.test.net
    friendly_name: host002
    mode: peer
    amount: 3000000000000
    roles:
      - voter
  - host: host003.test.net
    friendly_name: host003
    mode: dual
    amount: 100000000000
    roles:
      - harvester
accounts:
  - amount: 100000000000
    count: 5
  - amount: 3000000000000
    count: 5
total_supply: 7842928625000000

Nemgen needs account address, key pairs and assets to create the nemesis seed. We will create an generator using the Symbol Python SDK. The network generator will read the network template file and creates a new yaml with the required key pairs and assets to create the nemesis.
Below is the main logic for the network generator. A key pair is created for each account which is the main private/public keys. For each role associated with an account, a handler will add the other metadata the required.

def _create_keys(self):
    key_pair = self.facade.KeyPair(PrivateKey.random())
    return {
        'privatekey': str(key_pair.private_key),
        'publickey': str(key_pair.public_key),
        'address': str(self.facade.network.public_key_to_address(key_pair.public_key))
    }

def _add_keys(self, account, name):
    account[name] = self._create_keys()

def _create_account(self, amount, roles):
    account = self._create_keys()
    account.update({'amount': amount})
    if roles:
        account.update({'roles': roles})

    return account

def _create_node_account(self, node):
    node.update({'main': self._create_account(node['amount'], [])})

    if 'harvester' in node['roles']:
        self._add_keys(node, 'remote')
        self._add_keys(node, 'vrf')

    if 'voter' in node['roles']:
        self._add_keys(node, 'voting')

    return node

def create_accounts(self, amount, count, roles):
    return [self._create_account(
        amount,
        roles
    ) for _ in range(0, count)]

The network generator output the yaml file below. I only added the first node since the others are the similar except with different key pairs.

network: testnet
generation_hash_seed: 15609F28EE0E282ABF2105A5475313DF6FBACEA22B99EDD351672DB9351B5121
total_supply: 7842928625000000
currency_mosaic_id: '0x1BE3064212EBCB90'
epoch_adjustment: 1671429771s
nemesis:
  privatekey: D3BBBBA1D4BDF7834ED92D275E09E97A69DADCB98ACABB9404E3AB3D01688663
  publickey: F4885DF48A71E634D2E0F3BFBB54EF3E7FF3D73EFBB7C7F10E641F8AD2AA99F1
  address: TDDKGFAET5VIRX6CVJR46C3CJVTTOWYCFD3WA3I
nodes:
- host: host001.test.net
  friendly_name: host001
  mode: dual
  amount: 3000000000000
  roles:
  - harvester
  - voter
  main:
    privatekey: BEF08CEFF5A2A847FBDD2A39CA6F813D926EB41BAB93F331C9703ACB284EC336
    publickey: 42F8F92259ECF5F4D5BA3F0E99C532DDDB90CFEABFBFB5F92F2DB7B2D6B98CA5
    address: TCI6CN5WGZAVLQBJKYBIHRMIPPWWD5FZMAI54HA
    amount: 3000000000000
  remote:
    privatekey: 94A0D302C2C70BEE0A324612211E27026BDBAEE7BE5201E3680AC8D249DAAAC0
    publickey: C1FF6A475ABBD86CA6BFC0BE3B654BAA0DDC69F9BF7B30B908B277F2EC329D2D
    address: TBZT2JZPVHJKYN6YLM54B2V5HBSNAAG6IVI2EBI
  vrf:
    privatekey: 4C6E3D32B0FB727652FA3D2FC86D587162B997766D64DE9DB0DDFF69A5808DE5
    publickey: E802474E4576596B411954F7995A44F8957FC6B955CABA0A7C644006FC9DED8B
    address: TDDYZBZZM5LZGOVIHFFHVA7BDDI6HO4XZX7RYNY
  voting:
    privatekey: 6A0150B41D4A9E795E2493C9E80788873287E817A5B14E85FBA7CCC9ADFB874B
    publickey: C3BDF8672762AB4FF706590A4C05DDF4BCD38FA927A7BCC672921D11D1575FE5
    address: TCO3IMSPGS3IDEJZIVK4SKOU4KH7ZAUF6HOURVA
...
accounts:
- privatekey: 83D0155BCAD2BB2896B36C59DD6064277AEEEBDE785F673BB960A36A4D1612E3
  publickey: 37F826D581C1397F01290C8400B1DF4AA04EB54267A1F82B3596955FE2A3BFF5
  address: TAZUD3FGGMNQ4MRU6AIN6GSHU52DBJB2J5PD5EI
  amount: 100000000000
- privatekey: C68E1B16E347CF549524B72F5FD30E862233FF72CEC684A0A255311841528006
  publickey: B03BB1FB9F361EDCBF82195DE2AC45DB3AF6D8235944D77080C41E97686D6819
  address: TCDU332HMCQAWP3LIUIYL24D5EBUBXAWTKONEGQ
  amount: 100000000000
...

Before we can run the nemgen tool to create the nemesis file, there are couple more things needed.

1. Configuration properties file for nemgen
2. Transaction files required for node setup - remote, vrf and voting links.
3. Update config-network.properties file in the Symbol node resources folder.

Create the nemgen configuration file

The Network.yaml has all the information needed to create the nemgen configuration file. The snippet of code below reads the Network yaml into the configuration object. This is then use to generate the nemgen configuration file.

	def save_nemesis_configuration(self, output_filepath):
		accounts = '\n'.join(
			[f'{account["main"]["address"]} = {format_number_single_quote(account["amount"])}' for account in self.configuration['nodes']])
		accounts += '\n'
		accounts += '\n'.join(
			[f'{account["address"]} = {format_number_single_quote(account["amount"])}' for account in self.configuration['accounts']])
		configuration = f'''
[nemesis]

networkIdentifier = {self.configuration['network']}
nemesisGenerationHashSeed = {self.configuration['generation_hash_seed']}
nemesisSignerPrivateKey = {self.configuration['nemesis']['privatekey']}

[cpp]

cppFileHeader =

[output]

cppFile =
binDirectory = ./seed

[transactions]
transactionsDirectory = ./transactions

[namespaces]

symbol = true
symbol.xym = true

[namespace>symbol]

duration = 0

[mosaics]

symbol:xym = true

[mosaic>symbol:xym]

divisibility = 6
duration = 0
supply = {format_number_single_quote(self.configuration['total_supply'])}
isTransferable = true
isSupplyMutable = false
isRestrictable = false

[distribution>symbol:xym]
{accounts}
		'''

		self._save_configuration_file(output_filepath, configuration)

This will create a nemgen configuration below. The main note here is that all the address under the distribution section, are the main key pair for each account that was created. These addresses will be converted to transfer transactions with the specified amount in the nemesis.

[nemesis]

networkIdentifier = testnet
nemesisGenerationHashSeed = 15609F28EE0E282ABF2105A5475313DF6FBACEA22B99EDD351672DB9351B5121
nemesisSignerPrivateKey = D3BBBBA1D4BDF7834ED92D275E09E97A69DADCB98ACABB9404E3AB3D01688663

[cpp]

cppFileHeader =

[output]

cppFile =
binDirectory = ./seed

[transactions]
transactionsDirectory = ./transactions

[namespaces]

symbol = true
symbol.xym = true

[namespace>symbol]

duration = 0

[mosaics]

symbol:xym = true

[mosaic>symbol:xym]

divisibility = 6
duration = 0
supply = 7'842'928'625'000'000
isTransferable = true
isSupplyMutable = false
isRestrictable = false

[distribution>symbol:xym]
TCI6CN5WGZAVLQBJKYBIHRMIPPWWD5FZMAI54HA = 3'000'000'000'000
TARZLLBVHJZB3XYKDXMS6JOAAM645PR2NPL34TA = 3'000'000'000'000
TCHHP5GF2AGHCTRTPQUYVYZDMNZHEMMQ6ZX5JPI = 100'000'000'000
TAZUD3FGGMNQ4MRU6AIN6GSHU52DBJB2J5PD5EI = 100'000'000'000
TCDU332HMCQAWP3LIUIYL24D5EBUBXAWTKONEGQ = 100'000'000'000
TCTIOEMNEJAJXWF75QGGQV6JM4DAL6YENMOXG2Y = 100'000'000'000
TBVLDD5ZY4776JPF7AVJZRQYFPIKJDZJTXBKOLY = 100'000'000'000
TACPMJ7J5RASCMFQT4RCXXADHPCS5X7ICGHE5XA = 100'000'000'000
TDQBOFCN7P7NNKZKULQ4UMBXDNKGDW2VFJZH3FA = 3'000'000'000'000
TAJGUDFCC6UI2OAVVUFWICWR7FR5NHEQKQMRIDQ = 3'000'000'000'000
TD4ZRFEQABZP2DPXOKCKQYKGNIW5HPKTFV3LW4I = 3'000'000'000'000
TD5S6LJEF7S6JFB7APNDYZSUMMG7A3GHTA5NZNA = 3'000'000'000'000
TAQ6RQCME4DWVLOI4AWCLKEK5S5GLEAQLTHGZLA = 3'000'000'000'000
TC44DTLHL43XSCCAJR5QI6B4J35G3QQ4WWCJU2Y = 0
TCHVSKDSYAEPITMHBFECE7HF3Y65IACRPQDQHPQ = 0
TAYKKEU6VR7IXQV536VQZLULI6FWSUF5G65UMDQ = 0
TBMQUG2N7B4DU7G7VUDTDL6EZTPNNXHASFCBG6I = 7'821'328'625'000'000

Note: transactionsDirectory is the folder where the extra transactions to configure the nodes will be stored.

Create the node's configuration transactons

Lets create all the extra transactions(remote, vrf, voting) needed to setup each node and place them in the transactionsDirectory. The trick here is that all the transactions needs to be signed with the generation hash seed of the new network. After creating the facade for Symbol, the network information needs to be updated.

self.facade = SymbolFacade(self.configuration['network'])
self.facade.network = Network(
    'testnet',
    0x98,
    Hash256(self.configuration['generation_hash_seed'])
)

Now that the SymbolFacade is created and updated to use the new network generation hash seed, creating the extra transactions for the nemesis is no different than setting up your current node on the Symbol blockchain.
Note: for transaction in the nemesis, the deadline is 1 and fee is 0.

def _create_vrf_transaction(self, signer_public_key, account_descriptor):
    vrf_key_pair = self._get_key_pair_from_private_key(account_descriptor['vrf']['privatekey'])
    return self.facade.transaction_factory.create({
        'signer_public_key': signer_public_key,
        'deadline': 1,
        'type': 'vrf_key_link_transaction',
        'linked_public_key': vrf_key_pair.public_key,
        'link_action': 'link'
    })

def _create_account_key_link_transaction(self, signer_public_key, account_descriptor):
    remote_key_pair = self._get_key_pair_from_private_key(account_descriptor['remote']['privatekey'])
    return self.facade.transaction_factory.create({
        'signer_public_key': signer_public_key,
        'deadline': 1,
        'type': 'account_key_link_transaction',
        'linked_public_key': remote_key_pair.public_key,
        'link_action': 'link'
    })

def _create_voting_key_link_transaction(self, signer_public_key, account_descriptor):
    voting_key_pair = self._get_key_pair_from_private_key(account_descriptor['voting']['privatekey'])
    return self.facade.transaction_factory.create({
        'signer_public_key': signer_public_key,
        'deadline': 1,
        'type': 'voting_key_link_transaction',
        'linked_public_key': voting_key_pair.public_key,
        'start_epoch': 1,
        'end_epoch': 720,
        'link_action': 'link'
    })

def _create_transaction(self, transaction_type, account_descriptor, output_path):
    signer_key_pair = self._get_key_pair_from_private_key(account_descriptor['main']['privatekey'])
    transaction_factory = {
        'vrf': self._create_vrf_transaction,
        'remote': self._create_account_key_link_transaction,
        'voting': self._create_voting_key_link_transaction
    }

    transaction = transaction_factory[transaction_type](signer_key_pair.public_key, account_descriptor)
    transaction.fee = Amount(0)
    signature = self.facade.sign_transaction(signer_key_pair, transaction)
    self.facade.transaction_factory.attach_signature(transaction, signature)
    transaction_buffer = transaction.serialize()
    transaction_hash = self.facade.hash_transaction(transaction)
    file_path = Path(output_path) / f'{transaction_type}_{transaction_hash}.bin'
    self._save_transaction_file(file_path, transaction_buffer)

def create_nemesis_transactions(self, output_path):
    mkdirs(output_path)
    for node_descriptor in self.configuration['nodes']:
        if 'harvester' in node_descriptor['roles']:
            self._create_transaction('vrf', node_descriptor, output_path)
            self._create_transaction('remote', node_descriptor, output_path)

        if 'voter' in node_descriptor['roles']:
            self._create_transaction('voting', node_descriptor, output_path)

Update the Symbol node resources

We are almost ready to run the nemgen. The last step is to update the Symbol node resources. We will create a patcher which updates the config-network.properties file to match the information of the new testnet network.

config = RawConfigParser(comment_prefixes=None, empty_lines_in_values=False, allow_no_value=True)

config.optionxform = lambda option: option
filename = f'{resources_folder}/resources/config-network.properties'
config.read(filename)

config['network']['identifier'] = account_config['network']
config['network']['nemesisSignerPublicKey'] = account_config['nemesis']['publickey']
config['network']['generationHashSeed'] = account_config['generation_hash_seed']
config['network']['epochAdjustment'] = account_config['epoch_adjustment']

config['chain']['currencyMosaicId'] = account_config['currency_mosaic_id']
config['chain']['harvestingMosaicId'] = account_config['currency_mosaic_id']
config['chain']['initialCurrencyAtomicUnits'] = format_number_single_quote(account_config['total_supply'])
config['chain']['totalChainImportance'] = format_number_single_quote(account_config['total_supply'])

harvest_address = get_account_address(get_account_with_role(account_config, 'harvest_network_fee_sink'))
config['chain']['harvestNetworkFeeSinkAddressV1'] = harvest_address
config['chain']['harvestNetworkFeeSinkAddress'] = harvest_address

mosaic_address = get_account_address(get_account_with_role(account_config, 'mosaic_rental_fee_sink'))
config['plugin:catapult.plugins.mosaic']['mosaicRentalFeeSinkAddressV1'] = mosaic_address
config['plugin:catapult.plugins.mosaic']['mosaicRentalFeeSinkAddress'] = mosaic_address

namespace_address = get_account_address(get_account_with_role(account_config, 'namespace_rental_fee_sink'))
config['plugin:catapult.plugins.namespace']['namespaceRentalFeeSinkAddressV1'] = namespace_address
config['plugin:catapult.plugins.namespace']['namespaceRentalFeeSinkAddress'] = namespace_address

print(f'update network file: {filename}')
with open(filename, 'wt', encoding='utf8') as output_file:
    config.write(output_file)

Note: Only the network config needs to be update for a new testnet. If you want to change any values in any other properties file is fine as long as its valid.

Actual Nemesis Seeds Creation

Now we are ready to run the nemgen tool. If you build the catapult client code from scratch, you can find the nemgen in the bin folder. I will take the easy route and run the tool from the latest docker image.

@staticmethod
def generate_seed(output_path, config_path):
    cmd = [
        'docker',
        'run',
        '--rm',
        f'--user={os.geteuid()}:{os.getgid()}',
        f'--volume={output_path}:/output',
        f'--volume={config_path}:/config',
        '-w=/output',
        '-e=LD_LIBRARY_PATH=/usr/catapult/lib:/usr/catapult/deps',
        'symbolplatform/symbol-server:gcc-1.0.3.5',
        '/usr/catapult/bin/catapult.tools.nemgen',
        '--resources=/config',
        '--nemesisProperties=/output/block-properties-file.properties',
        '--useTemporaryCacheDatabase'
    ]

    dispatch_subprocess(cmd)

After this is complete the nemesis seed should be in the output folder. This can be used to bootstrap your test network.

What's next?

Now you have three nodes that can be setup to bootstrap your test network. Each node has

1. main account
2. linked transactions in the nemesis to allow either harvesting or voting.

All that is left is to generate configuration for each node. There are tools that does this already like shoestring so won't go over this here in details. But to give an idea of the steps

1. generate config for each node using the keys for each node
2. create voting files required
3. create the peer p2p and api json files
4. copy all to each node and start up

Your testnet should be up and running in a couple of seconds, which you can confirm by navigating to http://localhost:3000.