# Set Up Two Nodes l2 Network and Disaster Recovery
Steps below is the processes of setup up l2 network on **multiple** computers.
## Start the first node (bootnode)
### Before deployment
***1. Environment preparing:***
(**Notes: Currently history feature branch haven't been merge to main so please go to Indexer and Server for History sections to read them if you want to deploy history feature**)
```
git clone git@github.com:DelphinusLab/zkcross-lerna.git
cd zkcross-lerna
bash environment_linux.sh
bash setup.sh
// Do repo branches changing to those you want deploy
// Do not forget to change UI repo to correct branch before redo setup
bash redobootstrap.sh
```
***2. Change rpcSource of the system for deployment***
Please check the eth-config.ts comments and change the rpcSource of bsctestnet and goerli for deployment
***3. Modify Credential of the system:***
3.1. Update all **monitorAccount** for all networks in `eth-config.ts` and also update **deployer priv key** in the `monitor-secrets.json`.
3.2. Run: `npx tsx` to compile.
*Attention:* *For **default monitorAccount privkey**, please ask.*
***4. Change substrate-node.json url in `deployment/config`***
Change the address to the server wws address.
### Setup zkp and get verifier contract
***1. To generate PTAU, in `zkp/circom`, run command:***
```
bash tools/setup.sh
```
***Tips:** you can also copy your backup pot23_final.ptau file to the cirom folder instead of runing this command for time saving purpose*
***1.2 To create `verifier.sol`, in `zkp/circom`, run command:***
```
bash tools/compile.sh
```
### Deploy layer1 contracts
***1. Run these commands in `zkcross-lerna` to check if we have enough eth for the testnetworks:***
```
node packages/monitors/src/tools/ethBalanceCheck/eth-balance-check-tool.js goerli
node packages/monitors/src/tools/ethBalanceCheck/eth-balance-check-tool.js bsctestnet
node packages/monitors/src/tools/ethBalanceCheck/eth-balance-check-tool.js cronostestnet
node packages/monitors/src/tools/ethBalanceCheck/eth-balance-check-tool.js rolluxtestnet
```
***2. Init bridge***
**2.1. If you deploy delphinus for the first time (normally we should skip this and go to point 2, especially when we deploy prod test server by default networks)**
In `zkcross-lerna`, run **manually**:
```
cp packages/zkp/circom/verifier.sol packages/solidity/contracts/ZKPVerifier.sol
sed -i "s/\^0.6.11/^0.8.0/" packages/solidity/contracts/ZKPVerifier.sol
cd packages/monitors/src/tools
node init_BlockHeight.js
cd -
cd packages/solidity
npx truffle migrate --network goerli //Only do it if this is new network, do `npx truffle migrate --f 2 --to 2 --network goerli` instead if goerli is not new.
npx truffle migrate --network bsctestnet //Only do it if this is new network, do `npx truffle migrate --f 2 --to 2 --network bsctestnet` instead if bsctestnet is not new.
npx truffle migrate --network cronostestnet //Only do it if this is new network, do `npx truffle migrate --f 2 --to 2 --network cronostestnet` instead if cronostestnet is not new.
npx truffle migrate --network rolluxtestnet //Only do it if this is new network, do `npx truffle migrate --f 2 --to 2 --network rolluxtestnet` instead if rolluxtestnet is not new.
cd -
cd packages/solidity/clients/
node config-contracts-info.js
cd -
cd packages/solidity/clients/tools/bridge
node init.js goerli
node init.js bsctestnet
node init.js cronostestnet
node init.js rolluxtestnet
cd -
```
**2.2. It is not the first time to deploy delphinus on the config networks**
Run `bash init_bridge.sh` in lerna folder (If your network is not that stable, normally we **manually** run each line inside).
***3. Confirm onchain status***
```
node packages/solidity/clients/tools/bridge/status.js chainName
```
***4. Check l1 deployment succeed***
```
node packages/solidity/clients/tools/bridge/validate_l1_deployment.js
```
### Deploy substrateNode
***1. Stop docker container and delete images***
**Tips**: docker builder may take huge disk space. `docker system df` can show the usage. If there are huge amount of disk space used by build cache, we can clean it by `docker builder prune -a`.
```
//Stop substrateNode
docker ps -a
docker rm [container id]
docker ps -a
docker volume ls
docker volume rm [volume name]
docker volume ls
docker image ls
docker image rm [repository name]
docker image ls
```
***2. Custom l2 account***
You can **skip** this step if you would like to use **default** config.
***2.1. Modify `deployment/config/substrate-account-config.json`:***
```
{
"secret_key_uri": [
"//Smith",
"//Frank",
"//Cindy//stash",
"//Richard"
],
"authorities_seeds": [
"Eve//stash",
"Frank//stash"
],
"pre_funded_seeds": [
"Smith",
"Frank",
"Cindy//stash",
"Richard"
],
"sudo_account_seed": "sudo"
}
```
You must **set two different authorities_seeds** for 2 validators. `Eve//stash` is authorities_seeds for bootnode. `Frank//stash` is authorities_seeds for the second node. Add new authorities_seeds if you want to add new node.
***2.2 Modify `deployment/config/eth-config.js`***
Or modify eth-config.ts and run `npx tsc`
```
{
chainName: "bsctestnet",
mongodbUrl: "mongodb://localhost:27017",
syncEventsStep: 20000,
rpcSource:
"https://bsc.getblock.io/testnet/?api_key=" + secrets.getblock_key,
wsSource:
"wss://bsc.getblock.io/testnet/?api_key=" + secrets.getblock_key,
privateKey: secrets.accounts.deployer.priv,
monitorAccount: "0xea9225F0013D64d12e75B89c19da9e3922C86556",
deviceId: "97",
l2Account: "//Smith",
enabled: testnet,
isSnap: false,
},
{
chainName: "goerli",
mongodbUrl: "mongodb://localhost:27017",
syncEventsStep: 0, //default step 0: sync to latest directly
gasWarningAmount: "1",
rpcSource: "https://eth.getblock.io/goerli/?api_key=" + secrets.getblock_key_goerli,
wsSource: "wss://eth.getblock.io/goerli/?api_key=" + secrets.getblock_key_goerli,
privateKey: secrets.accounts.deployer.priv,
monitorAccount: "0xea9225F0013D64d12e75B89c19da9e3922C86556",
deviceId: "5",
l2Account: "//Frank",
enabled: testnet,
isSnap: true,
},
{
chainName: "cronostestnet",
mongodbUrl: "mongodb://localhost:27017",
syncEventsStep: 2000,
rpcSource: "https://cronos-testnet-3.crypto.org:8545",
wsSource: "wss://cronos-testnet-3.crypto.org:8546",
privateKey: secrets.accounts.deployer.priv,
monitorAccount: "0xea9225F0013D64d12e75B89c19da9e3922C86556",
deviceId: "338",
l2Account: "//Cindy//stash",
enabled: testnet,
isSnap: true,
},
{
chainName: "rolluxtestnet",
mongodbUrl: "mongodb://localhost:27017",
syncEventsStep: 20000,
rpcSource: "https://testnet.rollux.com:2814/",
wsSource: "",
privateKey: secrets.accounts.deployer.priv,
monitorAccount: "0xea9225F0013D64d12e75B89c19da9e3922C86556",
deviceId: "2814",
l2Account: "//Richard",
enabled: testnet,
isSnap: true,
}
```
***3. Apply custom account to substrate-node repo***
You **must do** this step even if you are using **default** config.
Run `bash generate-config.sh` in `substrate-node`.
**Tips**: **Do not modify** `substrate-node/generated_config` manually because `sed` in `generate-config.sh` will search specific string.
**4. Check if Accounts and Admins configs have been added successfully**
Run following command in `substrate-node`
```
bash config-check.sh
```
***5. Generate chain specification file: `customSpec.json`***
1. Run `cargo build` in `substrate-node/`
2. Run `./target/debug/node-swap build-spec --disable-default-bootnode --chain dev > customSpec.json` in `substrate-node/`.
3. For the purpose of using `customSpec.json` in second node, push the generated json to a new branch.
***6. Modify `deployment/config/substrate-node.json`:***
*Case I. Access Substrate-node on local machine*
Substrate node default websocket connection: port 9944 on localhost. We bind port 9944 of the docker container "delphinus-node" to port 9966 on 127.0.0.1 of the host machine (check `zkcross-lerna/docker-compose.yml`).
```
{
"local_address": "ws://127.0.0.1",
"address": "ws://127.0.0.1",
"port": "9966"
}
```
*Case II. Access Substrate-node on remote host*
Assume `bootnode.example.com` is pointed to remote host on which the node is deployed.
```
{
"local_address": "ws://127.0.0.1",
"address": "wss://bootnode.example.com",
"port": "9944"
}
```
`bootnode.example.com:9944` will also be used in Caddy later on.
***7. Modify `substrate-node/Dockerfile`***
* **Add customSpec.json to Dockerfile**
Add the following line (Do not omit `.` in the end of the command)
`COPY --from=builder /delphinuslab/packages/substrate-node/customSpec.json .`
After the following line
`COPY --from=builder /delphinuslab/bin/ /usr/local/bin/`
* **Set authorities_seeds**
You can **skip** this step if you would like to use **default** config.
Change `//Eve//stash` to `authorities_seeds` in `step 2.1(Modify deployment/config/substrate-account-config.json)` in `substrate-node/Dockerfile`:
```
RUN /usr/local/bin/node-swap key insert --chain customSpec.json --scheme Ed25519 --suri //Eve//stash --key-type gran && \
/usr/local/bin/node-swap key insert --chain customSpec.json --scheme Sr25519 --suri //Eve//stash --key-type aura
```
**explanations of `key insert`**
1. Add the aura authority keys to enable block production.
2. Add the grandpa authority keys to enable block finalization.
**Note**
**Add** `//` before the value(`Eve//stash`). Be sure to use
unique keys for each validator. If two validators have the same keys, they produce conflicting blocks.
* **Set CMD in `substrate-node/Dockerfile`**
Change
`CMD ["--dev", "--ws-external"]`
to
`CMD ["--chain", "customSpec.json", "--validator", "--unsafe-ws-external"]`
***chain:** Specifies the chain specification to use*
***validator:** Starts the node with the authority role and enables it to actively participate in any consensus task that it can*
***unsafe-ws-external:** Listens to all Websocket interfaces*
Here is an example of Dockerfile:
```
FROM docker.io/paritytech/ci-linux:production as builder
WORKDIR /delphinuslab
COPY . /delphinuslab
ENV CARGO_HOME /delphinuslab/.cargo-home
RUN --mount=type=cache,target=/delphinuslab/.cargo-home \
--mount=type=cache,target=/delphinuslab/packages/substrate-node/target \
cargo build --manifest-path=/delphinuslab/packages/substrate-node/Cargo.toml --locked --release && install -Dt ./bin/ /delphinuslab/packages/substrate-node/target/release/node-swap
# This is the 2nd stage: a very small image where we copy the delphinuslab node binary."
FROM docker.io/library/ubuntu:20.04
LABEL description="Zhenxunge node"
COPY --from=builder /delphinuslab/bin/ /usr/local/bin/
COPY --from=builder /delphinuslab/packages/substrate-node/customSpec.json .
RUN useradd -m -u 1000 -U -s /bin/sh -d /delphinuslab delphinuslab && \
mkdir -p /data /delphinuslab/.local/share && \
chown -R delphinuslab:delphinuslab /data /delphinuslab && \
ln -s /data /delphinuslab/.local/share/node-swap && \
# Sanity checks
/usr/local/bin/node-swap --version
USER delphinuslab
# Insert session key
RUN /usr/local/bin/node-swap key insert --chain customSpec.json --scheme Ed25519 --suri //Eve//stash --key-type gran && \
/usr/local/bin/node-swap key insert --chain customSpec.json --scheme Sr25519 --suri //Eve//stash --key-type aura
EXPOSE 30333 9933 9944 9615
VOLUME ["/data"]
ENTRYPOINT ["/usr/local/bin/node-swap"]
CMD ["--chain", "customSpec.json", "--validator", "--unsafe-ws-external"]
```
***8. Create docker image***
Run following command in `zkcross-lerna`:
```
DOCKER_BUILDKIT=1 docker build . -t delphinus-node -f packages/substrate-node/Dockerfile
```
***9. Modify `zkcross-lerna/docker-compose.yml`:***
```
version: "3.7"
services:
node:
image: delphinus-node
ports:
- "9966:9944"
- "30336:30333"
volumes:
- delphinus-node-data:/data
restart: unless-stopped
volumes:
delphinus-node-data:
```
Expose container port 9944 to the host machine port 9966.
Expose container port 30333 to the host machine port 30336
**9944:** Default port of Substrate to listen on for incoming WebSocket traffic.
**30333:** Default port of Substrate to listen on for peer-to-peer (p2p) traffic
*(30336 and 9966 can be any other ports not in use）*
***10. Run the docker image***
Run following command in `zkcross-lerna`:
```
docker-compose -f docker-compose.yml up
```
(***Attention:** Record the local node identity here for the purpose of runing the second node. In this case, it is:`12D3KooWDAZnqMiHrosp5xxEK4Aob5ezgZvRsvsnJ9teb5tqagfd`.*)
***11. Setkey and addPool in l2***
Before do this, please check the "how to add pool" section to see whether you need add additional pool for new networks.
```
node packages/monitors/src/tools/init-substrate.js
```
### Start Caddy
If you want to run Substrate-node on **remote host**. A remote Substrate node can be accessible **only** after we set up a secure proxy for websocket connections. The TLS certificate is obtained automatically with Caddy.
***1. Create new file called "Caddyfile" in `zkcross-lerna`***
***Run `ip addr show` in bootnode host:***
*(All the docker containers will be connected to the **docker0** bridge by default. Use the inet in Caddy)*
Assume `bootnode.example.com` is the **domain name**(without protocol). We will use it in the `bootnode` option in `Dockerfile` of the second node.
```
{
auto_https disable_redirects
}
bootnode.example.com:9944 {
reverse_proxy 172.16.200.1:9966
}
bootnode.example.com:8080 {
reverse_proxy 172.16.200.1:3000
}
bootnode.example.com:30333 {
reverse_proxy 172.16.200.1:30336
}
```
For example, we can visit `bootnode.example.com:30333` to communicate with host port 30336. Visiting port 30336 means visiting container port 30333.
***2. Run Caddy***
```
sudo caddy run --config ./Caddyfile
```
### Run MongoDB
1. Clear mongodb
```
cd [DB_PATH_NAME]
rm -r *
```
2. Start mongodb
```
mongod --dbpath [DB_PATH_NAME]
```
### Change rpcSource back in eth-config.js
We can change back the rpcSource of bsctestnet and goerli to ankr for monitor reading.
### Run Monitors
1. Start l1 monitors. In `packages/monitors`, run:
```
sh run_l1monitor.sh goerli
sh run_l1monitor.sh bsctestnet
sh run_l1monitor.sh cronostestnet
sh run_l1monitor.sh rolluxtestnet
```
2. Start l2 monitor. In `packages/monitors`, run:
```
sh run_l2monitor.sh -n // normal mode
sh run_l2monitor.sh -v // verbose mode
```
`sh run_l2monitor.sh` is normal mode too.
### Check if L2 has been deployed successfully
Run following command in `zkcross-lerna`
```
bash l2_deployed_check.sh
```
### Start indexer monitor
This is monitor to collect history info for UI
To start the indexer, run the following from from `packages/monitors` repo
bash run_indexer.sh
To restart the indexer and clear the mongodb collection, please run
bash run_indexer.sh new
and this will clear the DB, and start checking from block number 1.
### Start server for history
For remote host deployment, you must add this to the Caddyfile in `lerna` as the UI will query this address.
test.example.com:8090 {
reverse_proxy 127.0.0.1:8091
}
and restart Caddy.
Ensure that in `/config/server.json`, the `address` property is same as this Caddy address.
```
{
"address": "https://test.example.com:8090",
"port": "8091"
}
```
in order for UI to query the correct endpoint.
If server and UI are local, you may use the default server address of `http://127.0.0.1:8091`.
To start the server, from `deployment` run
npm run server
### Start UI
In `packages/ui`, run:
```
npm run start
```
For UI to utilise transaction history, please
git checkout history
in `ts-sdk` and `ui`.
### Access Delphinus user interface
**Local machine:**
Visit "http://localhost:3000" in the browser and do testing:
* Activate
* Charge
* Deposit
* WithDraw
* ...
**Server**
Visit
Here is the document of Delphinus user interface: https://github.com/DelphinusLab/delphinus-training/blob/main/Documents/DelphinusUI.md.
## Add the Second node
The second node is only for disaster recovery, so we only need to start docker on the second node host.
### Before deployment:
***1. Copy nessessary files from bootnode host to the second node host for disaster recovery.***
We should not change those files if we use a new host as the bootnode host. You should copy following files:
* directory `deployment/config/`
* directory `solidity/build/`
* directory `zkcross-lerna/db/`
* directory `zkp/circom/main_js/`
* directory `solidity/contracts/`
* `zkcross-lerna/blockNumberBeforeDeployment.json`
* `substrate-node/customSpec.json`
* `zkp/circom/verification_key.json`
* `zkp/circom/verifier.sol`
* `zkp/circom/main_0003.zkey`
* `zkp/circom/pot23_final.ptau`
**Note**:
1. Please check you already copy those files before executing later steps because if the bootnode host is damaged. We cannot get those files anymore!
2. You'd beter save those files in at least two places as a precaution.
***2. Environment preparing:***
(**Notes: Currently history feature branch haven't been merge to main so please go to Indexer and Server for History sections to read them if you want to deploy history feature**)
```
git clone git@github.com:DelphinusLab/zkcross-lerna.git
cd zkcross-lerna
bash environment_linux.sh
bash setup.sh
// Do repo branches changing to those you want deploy
// Do not forget to change UI repo to correct branch before redo setup
bash redobootstrap.sh
```
### Deploy substrateNode
***1. Stop docker container and delete images***
**Tips**: docker builder may take huge disk space. `docker system df` can show the usage. If there are huge amount of disk space used by build cache, we can clean it by `docker builder prune -a`.
```
//Stop substrateNode
docker ps -a
docker rm [container id]
docker ps -a
docker volume ls
docker volume rm [volume name]
docker volume ls
docker image ls
docker image rm [repository name]
docker image ls
```
***2. Custom l2 account***
COPY `deployment/config/` from bootnode host to the second node host to use same `deployment/config/substrate-account-config.json` and `deployment/config/eth-config.js` in all nodes.
***3. Apply custom account to substrate-node repo***
You **must do** this step even you are using **default** config.
Run `bash generate-config.sh` in `substrate-node`.
**Tips**: **Do not modify** `substrate-node/generated_config` manually because `sed` in `generate-config.sh` will search specific string.
***4. Check if Accounts and Admins configs have been added successfully***
Run following command in `substrate-node`
```
bash config-check.sh
```
***5. Pull `customSpec.json` that generated by bootnode from the new branch created above in `substrate-node/`***
***6. Modify `substrate-node/Dockerfile`***
* **Add customSpec.json to Dockerfile**
Add the following line (Do not omit `.` in the end of the command)
`COPY --from=builder /delphinuslab/packages/substrate-node/customSpec.json .`
After the following line
`COPY --from=builder /delphinuslab/bin/ /usr/local/bin/`
* **Set authorities_seeds**
You can **skip** this step if you would like to use **default** config.
Change `//Eve//stash` to `//Frank//stash` in `step 2.1(Modify deployment/config/substrate-account-config.json)` in `substrate-node/Dockerfile`:
```
RUN /usr/local/bin/node-swap key insert --chain customSpec.json --scheme Ed25519 --suri //Frank//stash --key-type gran && \
/usr/local/bin/node-swap key insert --chain customSpec.json --scheme Sr25519 --suri //Frank//stash --key-type aura
```
**explanations of `key insert`**
1. Add the aura authority keys to enable block production.
2. Add the grandpa authority keys to enable block finalization.
**Note**
**Add** `//` before the value(`Eve//stash`). Be sure to use
unique keys for each validator. If two validators have the same keys, they produce conflicting blocks.
* **Set CMD in `substrate-node/Dockerfile`**
Change
`CMD ["--dev", "--ws-external"]`
to
`CMD ["--chain", "customSpec.json", "--validator", "--unsafe-ws-external", "--bootnodes", "/dns/bootnode.example.com/tcp/30333/p2p/12D3KooWDAZnqMiHrosp5xxEK4Aob5ezgZvRsvsnJ9teb5tqagfd"]`
***chain:** Specifies the chain specification to use*
***validator:** Starts the node with the authority role and enables it to actively participate in any consensus task that it can*
***unsafe-ws-external:** Listens to all Websocket interfaces*
***bootnodes**: Specifies a list of boot nodes identifiers for peer-to-peer communication.`12D3KooWDAZnqMiHrosp5xxEK4Aob5ezgZvRsvsnJ9teb5tqagfd` is node identify of bootnode. `bootnode.example.com:30333` is set in [here](https://hackmd.io/9CWC23PjSMeP9mkFB62tfA?both#Start-Caddy).*
**Do not** add `http` or `https` before `bootnode.example.com:30333`. Read the first item in [Addtional Explanations](https://hackmd.io/9CWC23PjSMeP9mkFB62tfA?both#Addtional-Explanations).
Here is an example of Dockerfile:
```
FROM docker.io/paritytech/ci-linux:production as builder
WORKDIR /delphinuslab
COPY . /delphinuslab
ENV CARGO_HOME /delphinuslab/.cargo-home
RUN --mount=type=cache,target=/delphinuslab/.cargo-home \
--mount=type=cache,target=/delphinuslab/packages/substrate-node/target \
cargo build --manifest-path=/delphinuslab/packages/substrate-node/Cargo.toml --locked --release && install -Dt ./bin/ /delphinuslab/packages/substrate-node/target/release/node-swap
# This is the 2nd stage: a very small image where we copy the delphinuslab node binary."
FROM docker.io/library/ubuntu:20.04
LABEL description="Zhenxunge node"
COPY --from=builder /delphinuslab/bin/ /usr/local/bin/
COPY --from=builder /delphinuslab/packages/substrate-node/customSpec.json .
RUN useradd -m -u 1000 -U -s /bin/sh -d /delphinuslab delphinuslab && \
mkdir -p /data /delphinuslab/.local/share && \
chown -R delphinuslab:delphinuslab /data /delphinuslab && \
ln -s /data /delphinuslab/.local/share/node-swap && \
# Sanity checks
/usr/local/bin/node-swap --version
USER delphinuslab
# Insert session key
RUN /usr/local/bin/node-swap key insert --chain customSpec.json --scheme Ed25519 --suri //Frank//stash --key-type gran && \
/usr/local/bin/node-swap key insert --chain customSpec.json --scheme Sr25519 --suri //Frank//stash --key-type aura
EXPOSE 30333 9933 9944 9615
VOLUME ["/data"]
ENTRYPOINT ["/usr/local/bin/node-swap"]
#CMD ["--chain", "customSpec.json", "--validator", "--unsafe-ws-external", "--bootnodes", "/dns/bootnode.example.com/tcp/30333/p2p/12D3KooWDAZnqMiHrosp5xxEK4Aob5ezgZvRsvsnJ9teb5tqagfd"]
```
***7. Generate docker image***
Run following command in `zkcross-lerna`:
```
DOCKER_BUILDKIT=1 docker build . -t delphinus-node -f packages/substrate-node/Dockerfile
```
***8. Start the docker image***
Run following command in `zkcross-lerna`:
```
docker-compose -f docker-compose.yml up
```
**Note:**
1. If you have used the second node to do some ops **before** connecting it to the bootnode host, run `docker-compose down -v` before `docker-compose -f docker-compose.yml up` to delete docker volume. In such case, genesis block and state root hashes is different from those on bootnode which is not allowed for setting up l2 network.
1. You **do not need** to modify `zkcross-lerna/docker-compose.yml` on the second node host because all other nodes connect to port 30336(for example) of bootnode host and their port 30333 is not used by any nodes.
## Verify
Genesis block and state root hashes should be the same for both nodes. Each node should has one peer (**1 peers**), and they will produce **a block proposal** (best: #2 (0xe111…c084)). After a few seconds, new blocks will be finalized on both nodes.
## Disaster Recovery
Suppose the bootnode is damaged for some reason and the second node works well. We should copy volumes of the second node to new bootnode host to make sure we do not lose any pre-existing data.
### On the second node host
***1. Copy volumes to new bootnode host***
```
cd /var/lib/docker/volumes
tar cvf zkcross-lerna.tar zkcross-lerna_delphinus-node-data
```
Pass `zkcross-lerna.tar` to new bootnode host.
***2. Record node identity***
In `zkcross-lerna/`, run:
```
docker-compose -f docker-compose.yml up
```
Record the node identity. Suppose it is `12D3KooWDAZnqMiHrosp5xxEK4Aob5ezgZvRsvsnJ9teb5tyfgkn`.
***3. Delete node identity in the second node***
**Please make sure you already execute step 1 and step2 because this step will delete the data of the second node.**
Node identity is saved in `volumes`. Run following commands to avoid the node identity of the second node is the same as node identity of bootnode which is not allowed for setting up l2 network:
```
cd zkcross-lerna
docker-compose down -v
```
New node identity will be generated.
***4. Rebuil docker image with new Dockerfile***
**4.1 In `substrate-node/Dockerfile`, run:**
Change
`CMD ["--chain", "customSpec.json", "--validator", "--unsafe-ws-external", "--bootnodes", "/ip4/bootnode.example.com/tcp/30333/p2p/12D3KooWDAZnqMiHrosp5xxEK4Aob5ezgZvRsvsnJ9teb5tqagfd"]`
to
`CMD ["--chain", "customSpec.json", "--validator", "--unsafe-ws-external", "--bootnodes", "/ip4/bootnode.example.com/tcp/30333/p2p/12D3KooWDAZnqMiHrosp5xxEK4Aob5ezgZvRsvsnJ9teb5tyfgkn"]`
**4.2 In `zkcross-lerna`, run:**
```
DOCKER_BUILDKIT=1 docker build --no-cache . -t delphinus-node -f packages/substrate-node/Dockerfile
```
***5. Start substrate node***
In `zkcross-lerna`, run:
```
docker-compose -f docker-compose.yml up
```
If the new node identity is different from `12D3KooWDAZnqMiHrosp5xxEK4Aob5ezgZvRsvsnJ9teb5tyfgkn`, the command succeed.
### On new bootnode host
We already copy some files from bootnode host to the second node host in this [step](https://hackmd.io/9CWC23PjSMeP9mkFB62tfA?both#Before-deployment1). Suppose those files is in a directory named `backup`.
***1. Environment preparing:***
(**Notes: Currently history feature branch haven't been merge to main so please go to Indexer and Server for History sections to read them if you want to deploy history feature**)
```
git clone git@github.com:DelphinusLab/zkcross-lerna.git
cd zkcross-lerna
bash environment_linux.sh
bash setup.sh
// Do repo branches changing to those you want deploy
// Do not forget to change UI repo to correct branch before redo setup
bash redobootstrap.sh
```
***2. Modify Credential of the system:***
**2.1. Replace `deployment/config/` with `backup/config/`** to update all **monitorAccount** for all networks in `eth-config.ts` and also update **deployer priv key** in the `monitor-secrets.json`.
**2.2. Run: `npx tsx` to compile.**
***3. Change substrate-node.json url in `deployment/config`***
Change the address to the server wws address.
**4. Setup zkp and get verifier contract**
**4.1. Copy `backup/pot23_final.ptau` to `zkp/circom/`.**
You maybe need it to rerun `bash compile.sh`.
**4.2. get `verifier.sol`**
Copy `backup/main_js/`, `backup/verification_key.json`, `backup/verifier.sol`, `backup/main_0003.zkey` to `zkp/circom/`.
You will need them for shell scripts in `zkp/circom/tools`.
***5. Deploy layer1 contracts***
**5.1. Replace `backup/contracts/` with `solidity/contracts/`.**
**5.2. Copy`blockNumberBeforeDeployment.json` to `zkcross-lerna/`**
**5.3. Repalce `backup/build` with `solidity/build/`.**
**5.4. Confirm onchain status**
```
node packages/solidity/clients/tools/bridge/status.js chainName
```
**5.5. Check l1 deployment succeed**
```
node packages/solidity/clients/tools/bridge/validate_l1_deployment.js
```
***6. Deploy substrateNode***
**6.1. Stop docker container and delete images**
**Tips**: docker builder may take huge disk space. `docker system df` can show the usage. If there are huge amount of disk space used by build cache, we can clean it by `docker builder prune -a`.
```
//Stop substrateNode
docker ps -a
docker rm [container id]
docker ps -a
docker volume ls
docker volume rm [volume name]
docker volume ls
docker image ls
docker image rm [repository name]
docker image ls
```
**6.2. Apply custom account to substrate-node repo**
Run `bash generate-config.sh` in `substrate-node`.
**6.3. Check if Accounts and Admins configs have been added successfully**
Run following command in `substrate-node`
```
bash config-check.sh
```
**6.4. Copy `backup/customSpec.json` to `substrate-node/`**
**6.5. Replace `substrate-node/Dockerfile` with `backup/Dockerfile`**
**6.6. Create docker image**
Run following command in `zkcross-lerna`:
```
DOCKER_BUILDKIT=1 docker build . -t delphinus-node -f packages/substrate-node/Dockerfile
```
**6.7. Replace `zkcross-lerna/docker-compose.yml` with `backup/docker-compose.yml`**
**6.8. Start docker image with data of original bootnode**
```
cd zkcross-lerna
docker-compose down -v // delete original volumes if it exists
mkdir /var/lib/docker/volumes/zkcross-lerna_delphinus-node-data/
tar -xvf zkcross-lerna.tar -C /var/lib/docker/volumes/
docker-compose -f docker-compose.yml up
```
Node identity should be `12D3KooWDAZnqMiHrosp5xxEK4Aob5ezgZvRsvsnJ9teb5tyfgkn` which is the same as node identity get from step 2 in [On the second node](https://hackmd.io/9CWC23PjSMeP9mkFB62tfA?both#On-the-second-node-host)
***7. Start Caddy***
If you want to run Substrate-node on **remote host**. A remote Substrate node can be accessible **only** after we set up a secure proxy for websocket connections. The TLS certificate is obtained automatically with Caddy.
**7.1. Create new file called "Caddyfile" in `zkcross-lerna`**
***Run `ip addr show` in bootnode host:***
*(All the docker containers will be connected to the **docker0** bridge by default. Use the inet in Caddy)*
Assume `bootnode.example.com` is the **domain name**(without protocol). We will use it in the `bootnode` option in `Dockerfile` of the second node.
```
{
auto_https disable_redirects
}
bootnode.example.com:9944 {
reverse_proxy 172.16.200.1:9966
}
bootnode.example.com:8080 {
reverse_proxy 172.16.200.1:3000
}
bootnode.example.com:30333 {
reverse_proxy 172.16.200.1:30336
}
```
For example, we can visit `bootnode.example.com:30333` to communicate with host port 30336. Visiting port 30336 means visiting container port 30333.
**7.2. Run Caddy**
```
sudo caddy run --config ./Caddyfile
```
***8. Run MongoDB***
**8.1. Copy `backup/db` to `zkcross-lerna/`**
**8.2. Start mongodb**
```
mongod --dbpath [DB_PATH_NAME]
```
***9. Run Monitors***
**9.1. Start l1 monitors.**
In `packages/monitors`, run:
```
sh run_l1monitor.sh goerli
sh run_l1monitor.sh bsctestnet
sh run_l1monitor.sh cronostestnet
sh run_l1monitor.sh rolluxtestnet
```
**9.2. Start l2 monitor.**
In `packages/monitors`, run:
```
sh run_l2monitor.sh -n // normal mode
sh run_l2monitor.sh -v // verbose mode
```
`sh run_l2monitor.sh` is normal mode, too.
l2 monitor logs show something like this:
**The completed reqs length is not zero.**
***10. Check if L2 has been deployed successfully***
Run following command in `zkcross-lerna`
```
bash l2_deployed_check.sh
```
***11. Start indexer monitor***
This is monitor to collect history info for UI
To start the indexer, run the following from from `packages/monitors` repo
bash run_indexer.sh
To restart the indexer and clear the mongodb collection, please run
bash run_indexer.sh new
and this will clear the DB, and start checking from block number 1.
***12. Start server for history ***
For remote host deployment, you must add this to the Caddyfile in `lerna` as the UI will query this address.
test.example.com:8090 {
reverse_proxy 127.0.0.1:8091
}
and restart Caddy.
Ensure that in `/config/server.json`, the `address` property is same as this Caddy address.
```
{
"address": "https://test.example.com:8090",
"port": "8091"
}
```
in order for UI to query the correct endpoint.
If server and UI are local, you may use the default server address of `http://127.0.0.1:8091`.
To start the server, from `deployment` run
npm run server
***13. Start UI***
In `packages/ui`, run:
```
npm run start
```
For UI to utilise transaction history, please
git checkout history
in `ts-sdk` and `ui`.
***14. Access Delphinus user interface***
**Local machine:**
Visit "http://localhost:3000" in the browser and do testing:
* Activate
* Charge
* Deposit
* WithDraw
* ...
**Server**
Visit
Here is the document of Delphinus user interface: https://github.com/DelphinusLab/delphinus-training/blob/main/Documents/DelphinusUI.md.
## How to add pools
### Default pools after standard init
Add tToken in `config-contracts-info.ts` in `solidity/clients` folder
```
{
chainId: "***",
address:TokenInfo.networks["***"]?.address.replace("0x", ""),
wei:12,
name:"tToken"
}
```
**Now The whole system is recovered.**
### How to add pools for extra token pair
Add extra token in `contractsinfo.ts` in `deployment/config` folder
```
{
chainId: "***",
name: "****",
wei: **,
address: "***************************"
}
```
## Addtional Explanations
1. Explanation of using `dns` rather than `ipv4` in `Dockerfile` cannot be found on [Substrate official document](https://docs.substrate.io). Here is the explanation:
https://paritytech.github.io/substrate/master/sc_network/index.html
As you can see, a node's address is only a domain name **without** `http/https`
2. Better use latest npm version. We find some old version of npm and node will not work. (npm 8.13.2,node 16.14.2 works fine).
3. If you want to add more nodes to l2 network, the steps is the same as [Add the Second node](https://hackmd.io/9CWC23PjSMeP9mkFB62tfA?view#Add-the-Second-node).
4. If you set up l2 network **on local host**, you should use different ports in `docker-compose.yml` and different port in `Dockerfile` on the second node host.
For example,
**`zkcross-lerna/docker-compose.yml` on the bootnode host:**
```
version: "3.7"
services:
node:
image: delphinus-node
ports:
- "9944:9944"
- "30333:30333"
volumes:
- delphinus-node-data:/data
restart: unless-stopped
tty: true
volumes:
delphinus-node-data:
```
**`zkcross-lerna/docker-compose.yml` on the second node host:**
```
version: "3.7"
services:
node:
image: delphinus-node
ports:
- "9945:9944"
volumes:
- delphinus-node-data:/data
restart: unless-stopped
volumes:
delphinus-node-data:
```
**`packages/substrate-node/Dockerfile` on the second node host:**
```
CMD ["--chain", "customSpec.json", "--validator", "--port", "30334", "--unsafe-ws-external", "--bootnodes", "/ip4/172.16.200.1/tcp/30333/p2p/12D3KooWBj24ak3Dg8sdjiqjgZFeuPYTm2WCUY91p7GdHcM8JQRw"]
```
Here use port 30334 because the default port 30333 is used by the bootnode.
Sign in via Google
Sign in via Facebook
Sign in via X(Twitter)
Sign in via GitHub
Sign in via Dropbox
Sign in with Wallet
Connect another wallet