# NoS Docs

Table of contents:
1. [About NoS](#About-NoS)
2. [Why NoS?](#Polkadot-today-and-where-it’s-going)
3. [Communicating with Polkadot](#Communicating-with-Polkadot)
4. [NoS speeds up the process](#NoS-speeds-up-the-process)
5. [Dependencies](#Dependencies)
6. [Using NoS](#Using-NoS)
7. [Using Sidecar](#Using-Sidecar)
8. [Maintainance](#Maintainance)
9. [Chain Backup Support](#Chain-Backup-Support)
10. [Config Files](#Config-Files)

## About NoS
NoS stands for Node + (API) Sidecar and consists of a simple script to launch a full node of [Statemint](https://wiki.polkadot.network/docs/learn-system-chains#statemint) together with an instance of [Sidecar](https://github.com/paritytech/substrate-api-sidecar) on a machine (using Docker). But to really understand why it is useful let's first take a look at the state of Polkadot today, and where it's heading...

## Polkadot today and where it's going

Polkadot is a blockchain ecosystem that is comprised of a Relay Chain and several parachains. The main purpose for the Relay Chain is to provide [shared security](https://wiki.polkadot.network/docs/learn-parachains#shared-security) to the blockchains that are connected to it (known as parachains). Parachains are meant to provide the functionality for the system as a whole. [XCM (Cross Consensus Messaging)](https://wiki.polkadot.network/docs/learn-xcm) can then be used to pass messages, transfer assets, and execute transactions across parachains.

This architecture is drastically different from the current paradigm, and with that a new challenge arises...

Exchanges, custodians or other entities that want to provide services that support tokens in the Polkadot ecosystem essentially need to undergo a seperate integration process for each parachain one-by-one. This is a tougher task to accomplish in comparison to integrating with an ERC20 contract that exists on a chain they already support.

One way this process is being simplified is by creating Statemint, a "[System-level parachain](https://wiki.polkadot.network/docs/learn-system-chains)" that is designed to be the place where users can mint and manage assets. With Statemint as the asset reserve parachain, service providers would be able to integrate with a single parachain, and use XCM mesasges to move assets to/from the desired locations (parachains).

For a deeper look at the functionality that Statemint will provide, you can read this [Roadmap article](https://forum.polkadot.network/t/statemint-update-roadmap/1200/11), but the first step is to add support for parachain native assets and bridged assets.

## Communicating with Polkadot

In order to help builders connect their dapps to Substrate-based chains a generic API was created called [polkadot{.js} API](https://github.com/polkadot-js/api). However, due to the fast rate of development on the Polkadot protocol, this library also gets updated often, which can cause breaking changes for developers downstream. This is why Parity built [Substrate API Sidecar](https://github.com/paritytech/substrate-api-sidecar) as a wrapper around polkadot{.js}. The team that maintains Sidecar takes care of fixing any of the breaking changes, so that developers only need to make sure that their version of Sidecar is kept up to date.

## NoS speeds up the process

We wanted to make it as simple as possible for service providers to maintain their Polkadot infrastructure, and this is where [NoS](https://github.com/paritytech/nos) comes into play. With NoS, you are able to spin up the nodes you want and quickly be able to communicate with them through an instance of Sidecar. All that is required is a configuration file that contains the desired settings for the network. When you provide this config file to NoS, it will do some of the heavy lifting for you, which includes updating versions.

## Dependencies

NoS runs on a Linux machine (Debian), that uses Docker. It requires the following software installed in the system:

- [ ] Node.js
- [ ] NPM
- [ ] Yarn
- [ ] Git
- [ ] Docker

NoS downloads and maintains archive nodes for at least one Relay Chain and one parachain. This will be a considerable amount of data, so it is important to make sure you are provisioning enough hard drive space to accomodate this. Below is the approximate size of each blockchain at the time of writing.

polkadot (1000 GB) + statemint (52GB) = 1052GB
kusama (1530GB) + statemine (73G) = 1603GB
westend (244GB) + westmint (45GB) = 289GB
rococo (366GB) + rococo-contracts (41GB) = 407GB

Once you've provisioned enough resources and booted up the machine you can proceed below.

First you'll want to make sure your system packages are all installed and up to date by running:

`sudo apt update` - update local package cache
`sudo apt upgrade` - upgrades any packages that have new updates

**Install npm and node.js:**

`sudo apt-get install -y nodejs`

**Latest compatible versions:**

`npm -v` - v9.5.1 (as of writing)

`node -v` - v19.8.1 (as of writing)

**Install yarn:**
`sudo npm install --global yarn`

**Latest compatible version**:

`yarn -v` - 1.22.19 (as of writing)

**Install Git:**

`sudo apt install git`

**Latest compatible version:**

`git --version` - v2.30.2 (as of writing)

**Install Docker:**

https://docs.docker.com/engine/install/debian/#install-using-the-repository

**Install Docker-compose:**

`sudo curl -L "https://github.com/docker/compose/releases/download/v2.0.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose`

**Give it permissions:**

`sudo chmod +x /usr/local/bin/docker-compose`

**Ensure latest compatible version:**

`docker-compose --version` - v2.0.1 (as of writing)

For help with Docker, check out their [Documentation](https://dockerlabs.collabnix.com/docker/cheatsheet/)

## Using NoS

#### Option 1 - Makerfile:
For your convenience there is a Makerfile which simplifies the commands you need to execute:

`make sync-chain` - Sync chain data with values from .env file (this will take some time based on your machine specs, network conditions and database size.)

`make start` - Start parachain with a Sidecar for the Relay Chain and parachain in the background

`make start-foreground` - Start parachain with a sidecar for the Relay Chain and parachain in the foreground to see logs in realtime

`make stop` - Stop parachain and Sidecar containers

You can check if you already have `make` installed with the following command:

`make -version`

If it is installed you should see this:

GNU Make 4.3
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

If it is not installed you will need to run:

`apt install make`

After that you will be able to proceed with `make sync-chain` followed by starting your nodes with your choice of the 2 commands above. When you want to tear down the build, you can just run, `make stop`.

#### Option 2 - more verbose:
*(If you prefer going step-by-step you can follow the steps below)*

Once you have all the dependancies you can go forward with configuring and running NoS.

You'll start by cloning the [Github Repo](https://github.com/paritytech/nos), then inside the `nos` folder execute the following steps:

1. Run: `yarn --cwd ./src/ install && yarn --cwd ./src/ generate-env` to generate a necessary `.env.example` file. It also fetches the latest versions of collator and substrate-api-sidecar images and sets the current system's non-root user.
2. `mv .env.example .env`
3. Edit your `.env` file to contain prune mode and your relaychain and parachain details. Some examples are given below.
4. `mkdir chain-data`, make sure it has the current user's permissions.
5. (optional, for the faster sync) Download the chain database using the command:
`docker-compose -f sync-chain.yml up --remove-orphans --force-recreate`
it may take 40+ minutes.
*NOTE: check [Chain Back-up Support](#Chain-Back-up-Support) for a list of blockchains that Parity maintains daily snapshots for.*
6. Start the containers with `docker-compose up -d --remove-orphans --force-recreate`

After this process finishes you will want to check to see that your docker containers are running properly by running `Docker ps`. This should return something that looks like this:

CONTAINER ID   IMAGE                                           COMMAND                  CREATED      STATUS      PORTS                                                                                NAMES
bcb84c40a39e   parity/polkadot-parachain:0.9.400-269fb073caf   "/usr/local/bin/polk…"   6 days ago   Up 6 days   9933/tcp, 30333/tcp, 0.0.0.0:9944-9945->9944-9945/tcp, :::9944-9945->9944-9945/tcp   collator
db818dc64127   parity/substrate-api-sidecar:v14.5.3            "docker-entrypoint.s…"   6 days ago   Up 6 days   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp                                            para-sidecar
4eeb6bd19006   parity/substrate-api-sidecar:v14.5.3            "docker-entrypoint.s…"   6 days ago   Up 6 days   0.0.0.0:8081->8080/tcp, :::8081->8080/tcp                                            relay-sidecar

Take note on which ports your instances of Sidecar are running. These are the ports you will submit GET and POST requests to. By default The parachain will be served on port `8080` while the Relay Chain will be served on port `8081`.

If you chose to do the faster sync option (step 5 above), then you will need to sync your nodes using the Parity hosted daily backups you downloded. To do that you will need to run:

`time docker-compose -f sync-chain.yml up`

When done you should see something like this:

Starting chain ... done
Attaching to chain
chain              | polkadot
chain              | statemint
chain              | Syncing location:
gs://parity-polkadot-backups/statemint-rocksdb-archive
chain              | /root/google-cloud-sdk/bin  cp -r
gs://parity-polkadot-backups/statemint-rocksdb-archive/20230106-010000/*
/polkadot/chains/statemint/db/
chain              | Syncing location:
gs://parity-polkadot-backups/polkadot-rocksdb-prune
chain              | /root/google-cloud-sdk/bin  cp -r
gs://parity-polkadot-backups/polkadot-rocksdb-prune/20230105-012013/*
/polkadot/chains/polkadot/db/
chain              | stdout:
chain              | stdout:
chain exited with code 0

real    33m34.411s
user    0m2.723s
sys     0m0.339s

Lastly, you will start your nodes by running:

`docker-compose up -d`

You should then see something like this:

Creating polkadot             ... done
Attaching to nos-latest_sidecar_1, polkadot
[..]
polkadot    | 2023-01-06 02:24:24 [Parachain] 🔍 Discovered new
external address for our node:
/ip4/35.233.54.19/tcp/30333/ws/p2p/12D3KooWRMwMcgNqiq86gkwexeYGitkF7qJL2fXkecxGMjaxYWLc
polkadot    | 2023-01-06 02:24:29 [Relaychain] ⚙️  Syncing,
target=#13676949 (40 peers), best: #13661945 (0xe97f…0eaf), finalized
#13661911 (0x23dc…86b5), ⬇ 12.1MiB/s ⬆ 404.8kiB/s
polkadot    | 2023-01-06 02:24:29 [Parachain] ⚙️  Preparing,
target=#2929799 (8 peers), best: #2929382 (0x77e2…c49c), finalized
#2929381 (0xd754…cfb8), ⬇ 814.6kiB/s ⬆ 38.4kiB/s
polkadot    | 2023-01-06 02:24:30 [Parachain] ✨ Imported #2929384 (0x369d…6e9d)
sidecar_1   | 2023-01-06 02:24:30 warn: API/INIT: RPC methods not
decorated: transaction_unstable_submitAndWatch,
[...]
polkadot    | 2023-01-06 02:36:34 [Parachain] 💤 Idle (8 peers), best:
#2929857 (0xc33d…4999), finalized #2929599 (0x429f…20cf), ⬇ 1.1kiB/s ⬆
49 B/s
polkadot    | 2023-01-06 02:36:34 [Relaychain] ✨ Imported #13677067
(0x8ad8…d8b6)
polkadot    | 2023-01-06 02:36:34 [Relaychain] ✨ Imported #13677068
(0x4980…0fcc)
polkadot    | 2023-01-06 02:36:34 [Relaychain] ✨ Imported #13677069
(0x0896…b64e)
polkadot    | 2023-01-06 02:36:34 [Relaychain] ✨ Imported #13677070
(0xe398…7ad0)
polkadot    | 2023-01-06 02:36:34 [Parachain] ✨ Imported #2929860

*Note that if you chose the faster sync option (step 5 above) the chain will already be nearly synced to the tip, and you will just need to sync the blocks that happened since the last snapshot was taken. If you did not download the backups, then your node will have to sync from block 0 which will can take a considerable amount of time.*

## Using Sidecar

Once the Sidecar containers are running you will be able to send and receive requests through the ports they expose.

As a quick check you can see that both are operational using the following command:

`curl http://0.0.0.0:8081/node/version`

This should return something like:

"clientVersion":"0.9.40-95fe4c88628","clientImplName":"parity-polkadot","chain":"Polkadot"}

You should run that on port 8080 to make sure the parachain instance is responding as well.

For more information on Sidecar, please refer to the [Github Repo](https://github.com/paritytech/substrate-api-sidecar) and for in-depth documentation on the endpoints available to you, please check out the [Sidecar API Spec](https://paritytech.github.io/substrate-api-sidecar/dist/).

## Maintainance

Maintaining your instances will be as simple as updating your config file to reflect the new versions you wish to upgrade (or downgrade) to. You can edit your `.env` manually, or you can generate an `.env.example` file that contains the latest versions using `yarn generate-env`. You just need to remember to remove your old file and rename the new example file to `.env`. After that run `docker-compose up -d --remove-orphans --force-recreate` again and NoS will take care of upgrading your nodes and Sidcare versions and recreating the containers for them to run in.

*It is worth noting that if you wish to downgrade your node version for any reason, and there was a database migration at any point between your current node version and the version you wish to downgrade to, a full sync of the chain will be neccessary.*

## Chain Backup Support

#### Relaychains
- Polkadot
- Kusama
- Westend
- Rococo

#### Parachains
- Statemint
- Statemine
- Westmint
- Collectives Polkadot

## Config Files

### Pruning Options
- `archive` - default value
- `1000` - value for block pruning from backup

### Custom Chains

If you want to use your own chain backups you will need to add the following variables and provide to appropriate locations:

- `RELAYCHAIN_UP_LOCATION`
- `PARACHAIN_BACKUP_LOCATION`
- `RELAYCHAIN_RESTORE_PATH`
- `PARACHAIN_RESTORE_PATH`

*NOTE: `BACKUP_LOCATION`s above should be .tar files*

### Examples
#### Polkadot/Statemint

POLKADOT_VERSION=v0.9.36
POLKADOT_PARACHAIN_VERSION=polkadot-v0.9.36-afe528af
RELAYCHAIN="polkadot"
PARACHAIN="statemint"
SIDECAR_VERSION=latest
CONTAINER_UID=10000
RELAYCHAIN_PRUNING=archive

#### Kusama/Statemine

POLKADOT_VERSION=v0.9.36
POLKADOT_PARACHAIN_VERSION=polkadot-v0.9.36-afe528af
RELAYCHAIN="kusama"
PARACHAIN="statemine"
SIDECAR_VERSION=latest
CONTAINER_UID=10000
RELAYCHAIN_PRUNING=archive

#### Westmint/Westmine

POLKADOT_VERSION=v0.9.36
POLKADOT_PARACHAIN_VERSION=polkadot-v0.9.36-afe528af
RELAYCHAIN="westend"
PARACHAIN="westmint"
SIDECAR_VERSION=latest
CONTAINER_UID=10000
RELAYCHAIN_PRUNING=archive

#### Custom

POLKADOT_VERSION=v0.9.36
POLKADOT_PARACHAIN_VERSION=polkadot-v0.9.36-afe528af
RELAYCHAIN="randomRelaychain"
PARACHAIN="randomParachain
RELAYCHAIN_BACKUP_LOCATION="http://mycompany.com/randomRelaychain.tar"
PARACHAIN_BACKUP_LOCATION="http://mycompany.com/randomRelaychain.tar"
RELAYCHAIN_RESTORE_PATH="randomRelaychaindb"
PARACHAIN_RESTORE_PATH="randomParachaindb"
SIDECAR_VERSION=latest
CONTAINER_UID=10000
RELAYCHAIN_PRUNING=archive