Keep your node talking on Symbol

Background

Symbol blockchain has been up for over 9 months and nodes have been gossiping a lot. There have been over 2 million transactions and 800K blocks shared between 1400 nodes. This is a lot of talk.

Symbol nodes communicate over Secure Sockets Layer(SSL). SSL is a standard security technology for establishing an encrypted connection between two endpoints. But wait, why is an encrypted connection required when the data ends up on a public blockchain you ask? SSL protects against “man-in-the-middle” attack. This is where a malicious actor can intercept communication, and decrypt it, pretending to be another node. I will leave this for a later talk.

SSL established a secure connection using a process called an "SSL Handshake”. The SSL handshake requires each node to have a pair of keys(private/public). In Symbol, these keys are called the node's private/public key. These keys are required to be unique for the chain. Node certificates are created from its keys and are used to verify the node's communication on the chain.

You can view the public keys assigned to your node through a node/info REST call, as seen below.

curl https://401-joey-dual.symboltest.net:3001/node/info | jq
{
"version": 16777985,
"publicKey": "3E9C8CCC16DDB1EDFF9F2D0BB92D3F839CCC33D38034E526DC768F7BA3958E00",
"networkGenerationHashSeed": "7FCCD304802016BEBBCD342A332F91FF1F3BB5E902988B352697BE245F48E836",
"roles": 7,
"port": 7900,
"networkIdentifier": 152,
"host": "401-joey-dual.symboltest.net",
"friendlyName": "401-joey-dual",
"nodePublicKey": "D26336247A3D1AF89B4040C8BAF3E288209B50B9C8547587B94D020195933FA5"
}

publicKey - should be your main account public key.
nodePublicKey - node's public key which is used for SSL communication. This is also used in delegated harvesting.

Today we discuss how to:

1. Check a node's certificate expiration date
2. Update a node's certificate

Check a node's certificate expiration date

Depending on which tool you use to manage your node, there should be either a cert or certificates folder which contains the node certificate files. If you change directory to the certificate folder you will see these files

ca.cert.pem(or ca.crt.pem)
ca.pubkey.pem
node.crt.pem
node.full.crt.pem
node.key.pem

ca.pubkey.pem - Should be your main account public key.
node.key.pem - Node's private key and is used in SSL.
ca.cert.pem - CA certificate signed by the main account private key.
node.crt.pem - Node certificate that is signed by CA
node.full.crt.pem - certificate chain used by the server

A certificate has an expiration date and will become invalid after that date. When the node certificate expires, it will stop communicating with the chain leaving your node in a fork. You will see these errors in your logs.

node | 2021-12-20 19:13:06.386308 0x00007f73fcff9700: <warning> (ionet::PacketSocket.cpp@133) async_write returned an error: End of file
node | 2021-12-20 19:13:06.386488 0x00007f73fcff9700: <warning> (ionet::PacketSocket.cpp@126) async_shutdown returned an error: shutdown while in init
node | 2021-12-20 19:13:06.410177 0x00007f74057fa700: <error> (ionet::PacketSocket.cpp@44) failed when reading from socket: sslv3 alert certificate expired
node | 2021-12-20 19:13:06.410247 0x00007f74057fa700: <error> (api::RemoteRequestDispatcher.h@50) read from remote node failed for network time request

To check the expiration date on the node's cert, you can use the openssl tool. Below is a dump of my CA certificate. There is a lot of information but you really only care about start and end dates for the certificate.

openssl x509 -in ca.cert.pem -text -noout
Certificate:
Data:
Version: 1 (0x0)
Serial Number:
3f:3b:5f:fb:e6:df:5f:6d:f1:00:13:7a:9b:53:38:0e:0c:2f:f7:fc
Signature Algorithm: ED25519
Issuer: CN = node-account
Validity
Not Before: Dec 3 22:45:26 2021 GMT
Not After : Nov 28 22:45:26 2041 GMT
Subject: CN = node-account
Subject Public Key Info:
Public Key Algorithm: ED25519
ED25519 Public-Key:
pub:
11:b2:df:7d:96:f6:5d:21:ef:4b:d1:0e:99:f4:06:
8d:87:1f:ae:17:e4:74:7c:6d:49:dd:db:c4:f3:95:
a7:5d
Signature Algorithm: ED25519
08:45:46:cb:29:17:b2:d0:de:55:88:c0:db:35:9d:c5:1f:7b:
f3:a0:e1:5e:27:99:66:56:82:c4:ee:ec:1f:29:81:97:f0:91:
76:a5:ea:c1:68:83:86:ba:68:a5:71:35:eb:3f:54:fa:53:2f:
7c:38:ba:37:e9:aa:6e:cc:67:0e

To see when your CA certificate expiration date, type openssl x509 -in ca.cert.pem -enddate -noout. This should be 20 years by default and nowhere near expiring. No need to worry, yet.

openssl x509 -in ca.cert.pem -enddate -noout
notAfter=Nov 28 22:45:26 2041 GMT

Your node certificate is usually only valid for a year.

openssl x509 -in node.crt.pem --startdate -enddate -noout
notBefore=Dec 3 22:45:26 2021 GMT
notAfter=Dec 13 22:45:26 2022 GMT

The notAfter date is the certificate expiration date. You will need to create a new certificate before this date.

You could create a cron job that runs weekly to check your certificate's end date and sends an email a week before they expire. Or if you like me, just create a reminder in my calendar.

For most nodes which were created at the launch of the symbol, your node's certificate should be expiring late March/early April. Check your nodes and set a reminder to update.

Updating node's certificates

It's time to update your certificates. This is the easy part for me. Gimre has already written a great blog on creating your node's certificate. I will not go into the details here but follow steps 3-6 if you want to be adventurous and give a shout on discord if you have issue.

For security reasons, it's suggested that you create your certificates on an offline host.

For updating node's certificate, there are two options

1. Create a new node's private keys. With this option, delegated harvesters assigned to your node will be lost. Each delegated harvester will need to re-link to your node.
2. Reuse the node's private keys to create a new certificate. This will keep your delegated harvesters safe but less secure than option 1 if your node key was compromised.

Of course, you can also wait for our tools will be updated to renew your node's certificate in the new year.

With great power comes great responsibility

Remember to renew your node's certificates.