---
# System prepended metadata

title: filcollins storage provider setup

---

# filcollins storage provider setup

## table of contents

* [overview](#overview)
* [ssh config](#ssh-config)
* [firewall](#firewall)
* [hardware](#hardware)
* [required sp processes](#required-sp-processes)
* [long-term storage](#long-term-storage)
* [known issues](#known-issues)

## overview

The storage provider (SP) is running on Protocol Labs managed infrastructure.

https://filfox.info/en/address/f01953925

public ip: 209.94.92.6

public peer id: `12D3KooWNSRG5wTShNu6EXCPTkoH7dWsphKAPrbvQchHa5arfsDC`



---

## ssh config

```
Host worker-gpu-9
  User nonsense
  HostName 209.94.92.6
  ForwardAgent yes
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  LogLevel QUIET

Host worker-gpu-10

Host worker-cpu-1-1

Host worker-cpu-1-4
```

---

## firewall

ufw is enabled on `worker-gpu-9` which is our public instance

ports 22, 24001, 2345 are enabled

---

## tailscale

tailscale / wireguard is installed on all hosts and a vpn is established with `sofiaminer`

---

## hardware

4 instances - 2 cpu instances ; 2 gpu instances

### cpu instance type

#### cpu

model name: AMD EPYC 7F32 8-Core Processor

#### storage

5 x 1.8TB HDDs running as a striped array, mounted at /mnt/hddvol, designated for scratch area

2 x NVMe running as a stripped array, mounted at /mnt/nvmevol, designated for scratch area

nonsense@worker-cpu-1-1:~$ sudo vgs
  VG        #PV #LV #SN Attr   VSize  VFree
  hddgroup    5   1   0 wz--n-  8.73t  33.65g
  nvmegroup   2   1   0 wz--n- <2.62t <71.54g
  
#### memory

1 TB RAM

### gpu instance type

#### cpu

model name: Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz

#### memory

502 GB RAM

#### gpus

4 x NVIDIA Quadro RTX 6000 (24 GB GDDR6)

### gpu instance - worker-gpu-10

#### storage

500 TB Ceph system mounted at /mnt/ceph -- designated for long-term storage of sealed/unsealed sectors

3 x 900 GB HDDs running as a striped array, mounted at /storage/d1, designated for scratch area

### gpu instance - worker-gpu-9

#### storage

500 TB Ceph system mounted at /mnt/ceph -- designated for long-term storage of sealed/unsealed sectors

3 x 900 GB HDDs running as a striped array, mounted at /storage/d1

---

## required sp processes

### lotus daemon

* running in a tmux pane on worker-gpu-9
* uses / filesystem for chain storage
```
lotus daemon &>> lotus-daemon.log
```

### lotus-miner process

* running in a tmux pane on worker-gpu-9
* Make sure to use the below variable before starting the miner process
```
export CUDA_VISIBLE_DEVICES="0,1"
lotus-miner run &>> lotus-miner.log
```

### lotus-worker processes - sealing

1. running on worker-cpu-1-1 and worker-cpu-1-4
```
lotus-worker run --no-default --addpiece --precommit1  --data-cid
```


2. running on worker-gpu-10 (tmux GPU)
Sealing disk = /storage/d1/scratch
```
export CUDA_VISIBLE_DEVICES="0,1,2,3"
LOTUS_WORKER_PATH=/home/nonsense/.lotusworker lotus-worker run --no-default --precommit2 --commit --replica-update --prove-replica-update2 --regen-sector-key --name worker-gpu-10-worker0
```


3. running on worker-gpu-10  (tmux Storage)
Sealing disk = /storage/d1/scratch1
```
export CUDA_VISIBLE_DEVICES="4"
LOTUS_WORKER_PATH=/home/nonsense/.lotusworker1 lotus-worker run --no-default --no-local-storage --listen 0.0.0.0:3457 --name worker-gpu-10-STORAGE
```


### lotus-worker processes - windowpost on worker-gpu-9

```
export CUDA_VISIBLE_DEVICES="2"
LOTUS_WORKER_PATH=/home/nonsense/.lotusworkerpost lotus-worker run --no-default --windowpost --no-local-storage --listen 0.0.0.0:3457 --name worker-gpu-9-wdPost
```

### lotus-worker processes - winningPost on worker-gpu-9

```
export CUDA_VISIBLE_DEVICES="3"
LOTUS_WORKER_PATH=/home/nonsense/.lotuswinningpost lotus-worker run --no-default --winningpost --no-local-storage --listen 0.0.0.0:3458 --name worker-gpu-9-wnPost
```

### YugabyteDB docker container

* running on worker-gpu-10
```
sudo docker run -d --name yugabyte -p7000:7000 -p9000:9000 -p15433:15433 -p5433:5433 -p9042:9042 -v /home/nonsense/yb-home:/home/yugabyte yugabytedb/yugabyte:latest bin/yugabyted start --base_dir=/home/yugabyte/yb_data --daemon=false
```

### boostd-data service

* in TMUX on worker-gpu-10 after starting YugabyteDB container

```
boostd-data run yugabyte --hosts 127.0.0.1 --connect-string="postgresql://postgres:postgres@127.0.0.1:5433?sslmode=disable" --addr 0.0.0.0:8044
```

### boostd process

* running on worker-gpu-9

running as a service - `/etc/systemd/system/boostd.service`

incoming staging area is symlinked to Ceph

### additional configurations

* finalizeearly is `true`
* New sectors for deals are disabled

---

## long-term storage

we use Ceph system for long-term storage of sealed/unsealed sectors.

* 2x 500TiB is formatted with ext4
* worker-gpu-10 has access to one ceph device, mounted at `/mnt/ceph`
* worker-gpu-9 has access to another ceph device, mounted at `/mnt/ceph`
* devices are only attached to single machine, because they are currently RBD volumes (RWO). 

---

## Tailscale

in order to be able to test out multi-boost / single LID flows, we have set up a VPN between sofiaminer and filcollins.

If the VPN needs to be restarted (eg it doesn't come back up when a machine is rebooted) use:

```
$ sudo tailscale logout
$ sudo tailscale login
Login credentials:
boostteam55@gmail.com
(password is in OnePassword)
```

---

## known issues

### pruning of chain data

at the moment we must manually prune the chain data on worker-gpu-9 as we are still not using splitstore with discard store

```
cd /mnt/ceph/tmp/ && aria2c -x5 https://snapshots.mainnet.filops.net/minimal/latest
lotus daemon stop
rm -rf /home/nonsense/.lotus/datastore/chain/
rm -rf /home/nonsense/.lotus/datastore/splitstore/
lotus daemon --import-snapshot /mnt/ceph/tmp/<snapshot-name>.car
```

### sealing pipeline gets blocked when trying to AddPiece if there are no extended Available sectors for snap deals

workaround: periodically use the `./new-extend-sectors.sh` script to extend sectors

TODO: Run on a daily cron

### lotus-miner almost never shuts down gracefully

when wanting to restart `lotus-miner` we have to `kill -9` it

```
ps -ef | grep "lotus-miner run" | head -1 | awk '{print $2}' | xargs kill -9
```

### replace username to something more generic

replace user from `nonsense` to `filadmin` or something

### no backups for repos

we should add periodic backups for all important repos -- lotus-miner