# Filecoin Setup Specs and Knowledgebase

TODO(Setup):
~~1. gpu2 feature~~
~~2. ZFS for storage~~
~~3. RAID 0 of 2xNVME for chain storing~~
4. ~~Fix GPU~~
5. ~~Ensure remote connection to main router; configure loopback for remote LotusP2P address; add No/Announce fields for miner~~
6. ~~Ensure remote to IPMI~~
7. ~~Disable multithreading~in PC1~~
8. ~~SSH keep-alive~~
9. ~~Update BIOS for PC1~~
11. ~~Cron pledging~~
14. ~~Move RAM from Main/Misc to PC2~~
15.~~Optimize IO~~
16. ~~Auto snapshotting~~
17. ~~Remove swap~~
18. ~~Change router and new eth cables~~
19. ~~Rotate logs and enable colors~~
20. ~~LTS for PC1~~
21. ~~Splitstore~~
22. ~~Migrate from ZFS to madm~~ The root issue was fixed by kernel downgrade to latest LTS.
23. ~~Fix log rotation~~
24. Separate Market from Miner
26. Add MARKET_API_INFO
27. Move owner account to Multisig(Ledger)
28. Lotus Farcast
29. Smarter autopledging(seems like is not needed anymore)
30. Setup IPFS on Main
31. Remote power control
32. NFS
33. Lotus user(for now Wondertan controls and runs everything)
34. IPV6 and QUIC
35. Firewall, Whitelist

TODO(Docs):
1. Add server descriptions
2. Hardware specs and custom BIOS settings
3. NIC Names, MACs and Bonds
4. Automate file edits
5. Chain SDD and Storage HDDs mounting
6. ZFS configuration
7. Lotus user

## Servers

### Miner
CaseID: 3
Users: root, Wondertan
Host: Feel-Miner

enp68s0: a8:5e:45:e8:1d:32
enp67s0: 04:d9:f5:15:37:1c

SSH: 192.168.1.103:22 | 178.212.197.65:103
LotusAPI: 192.168.1.103:1234 | 178.212.197.65:1234
MinerAPI: 192.168.1.103:2345 | 192.168.88.103:2345 | 178.212.197.65:2345
LotusP2P: 192.168.1.103:12345 | 178.212.197.65:12345
MinerP2P: 192.168.1.103:23456 | 178.212.197.65:23456
IPFS_P2P: 192.168.1.103:4000 | 178.212.197.65:4000
IPFS_API: 192.168.1.103:5000

### PC1
CaseID: N/A
Users: root, Wondertan
Host: Feel-PC1

eno1np0: 3c:ec:ef:72:46:9e
enp129s0f0: 40:a6:b7:1b:86:18
enp129s0f1: 40:a6:b7:1b:86:19
bond1: fe:af:c4:ca:58:e2

SSH: 192.168.1.106:22 | 178.212.197.65:106
Worker: 192.168.88.106
IPMI: 192.168.1.116:80 | 178.212.197.65:8293

IPMILogin: ADMIN
IPMIPass: RHMJZFNATO

### PC2_A
CaseID: 2
Users: root, Wondertan
Host: Feel-PC2-A

enp68s0: z
enp33s0f0: 40:a6:b7:1b:84:bc
enp33s0f1: 40:a6:b7:1b:84:bd
bond1: 9a:d6:4f:68:b0:7e

SSH: 192.168.1.102:22 | 178.212.197.65:102
Worker: 192.168.88.102

### PC2_B
CaseID: 5
Users: root, Wondertan
Host: Feel-PC2-B

enp68s0: a8:5e:45:e8:1f:68
enp33s0f0: 40:a6:b7:16:00:0c
enp33s0f1: 40:a6:b7:16:00:0d
bond1: da:cb:33:82:0e:69

SSH: 192.168.1.105:22 | 178.212.197.65:105
Worker: 192.168.88.105

### C2
CaseID: 1
Users: root, Wondertan
Host: Feel-C2

enp68s0: a8:5e:45:e8:1e:a8
enp67s0: 04:d9:f5:15:37:1f

SSH: 192.168.1.101:22 | 178.212.197.65:101
Worker: 192.168.88.101

### Misc
CaseID: 4
Users: root, Wondertan
Host: Feel-Misc

enp68s0: a8:5e:45:e8:1e:d9
enp67s0: 04:d9:f5:15:35:64

SSH: 192.168.1.104:22 | 178.212.197.65:104
Worker: 192.168.88.104

// TODO: Add descriptions
## Routers

### Management Network(mgmt)
Host: Feeler
SSH: 192.168.1.1:22 | 178.212.197.65:22
Manage: 192.168.1.1:80 | 178.212.197.65:80
WinBox: 192.168.1.1:8291 | 178.212.197.65:8291
MAC(Custom): C4:AD:34:CD:E1:D1
Login: admin
Password: #################
-
Port Mappings

### Storage Network(strg)
Host: Feel.Switch
Manage: 192.168.1.100:80
WinBox: 192.168.1.100:8291 | 178.212.197.65:8292
MAC(Custom): C4:AD:34:CD:E1:D1
Login: admin
Password: #################
-
Bond Interfaces
MTU: 9000
Removed NAT Masquarade to block traffic from/to WAN

## Installation

### OS(ArchOS)

# NOTE: Disks name may vary. Use `lsblk` to find and/or check disk names/partions/mount points

wipefs -a /dev/nvme0n1 # Wipe exisiting OS, if exists

# TODO: Automate
cfdisk /dev/nvme0n1 # Disk Partioning with simple GUI
# Create 3 required partions:
#    Name           Size Type
# 1. /dev/nvme0n1p1 128M EFI System
# 2. /dev/nvme0n1p2 64G  Linux swap
# 3. /dev/nvme0n1p3 Full Linux filesystem

# Create and mount root filesystem
# Note: Create and mount *this* first 
mkfs.ext4 /dev/nvme0n1p3
mount /dev/nvme0n1p3 /mnt

# Create and enable swap 
# NOTE: In general using swap is not recommended for mining operation due to speed unregular slow downs and thus unpredictable timings. 
# IF possible do not turn on swap and calculate RAM usage in advance to ease futher miner administration. 
mkswap /dev/nvme0n1p2
swapon /dev/nvme0n1p2

# Create and mount EFI filesystem
mkfs.fat -F32 /dev/nvme0n1p1
mkdir /mnt/boot /mnt/boot/efi && mount /dev/nvme0n1p1 /mnt/boot/efi

# NOTE: Also mount all other disks, e.g. the SSD used for sealing:
# mkfs.ext4 /dev/nvme1n1
# mkdir /mnt/sealing && mount /dev/nvme1n1 /mnt/sealing

# Install linux-zen and required packages 
# NOTE: That to use supported 
pacstrap /mnt base linux linux-headers nvidia-utils nvidia-dkms cuda-tools downgrade dkms grub efibootmgr micro amd-ucode go rustup openssh man lshw sudo git make rsync wget parted nvtop htop iptop nnn mosh

# Generate /etc/fstab that tells hount mount filesystems automatically on every OS start 
genfstab -U /mnt >> /mnt/etc/fstab

# Note: Current boot is loaded in Live environment what means that we loaded OS from the USB.
# The following command loads recently installed OS 
arch-chroot /mnt

### Network configuration
micro /etc/hostname # Write hostname(Name seen from the network) for node

micro /etc/hosts # optional
#
127.0.0.1 localhost
#

micro /etc/resolv.conf
#
nameserver 8.8.8.8
nameserver 8.8.4.4
#

# List all the available network devices with their speed
lshw -class Network

# Create configurations for Management and Storage networks
micro /etc/systemd/network/enp68s0.network # TODO: Automate
#
[Match]
Name=enp68s0

[Network]
Address=<local_ip_address>/24 # prefer static address over provided by DHCP
Gateway=<router_ip_address> # fill this and below fields only for the network device that connects to WAN
DNS=8.8.8.8 # or any other preffered DNS server
DNS=8.8.4.4
#

# NOTE: In case of using 2x10GB network devices create bonding interface
# For thw setup 802.3ad bonding is used, as the most advanced mode, 
# however it requires switch/router support.

micro /etc/systemd/network/bond.netdev
#
[NetDev]
Name=bond1
Kind=bond
MTUBytes=9000 

[Bond]
Mode=802.3ad
MIIMonitorSec=1s
LACPTransmitRate=slow
UpDelaySec=2s
DownDelaySec=8s
TransmitHashPolicy=layer2+3
MinLinks=2
#

micro /etc/systemd/network/bond.network
#
[Match]
Name=bond1

[Network]
BindCarrier=enp33s0f0 enp33s0f1
Address=192.168.88.106/24
#

micro/etc/systemd/network/enp33s0f0
#
[Match]
Name=enp33s0f0

[Network]
Bond=bond1
#
 
micro /etc/systemd/network/enp33s0f1
#
[Match]
Name=enp33s0f1

[Network]
Bond=bond1
#

# Enable(autostart) network manager daemon(background process)
systemctl enable systemd-networkd systemd-resolved

### Add user
# Set password for root user
passwd

# Create user for system administrator
useradd -m -g wheel lotus # m - creates home directory; g wheel - adds user to the sudo group NOTE: Use any other name if necessary
passwd lotus

# Configure group for sudo(super users)
micro /etc/sudoers # TODO: Automate
# Uncomment in the file a line below
%wheel ALL=(ALL) ALL
#

### Remote Access(SSH)
# After reboot, copy local puclic key to remote server
ssh-copy-id lotus@<ip_address>

# or copy manually if Live environment already accessed remotely
scp ~/.ssh/id_ed25519.pub root@<ip_address>:/mnt/home/lotus/ # copy pubkey to Live environment NOTE: Use any other key algorithm, but ed25519 is recommended 
# NOTE: The command above must be run PC remotely accessing Live environment

# register key and manage permissions
su - lotus # login as new user
mkdir .ssh && chmod 700 .ssh
cat id_ed25519.pub >> .ssh/authorized_keys
rm id_ed25519.pub && touch .ssh/authorized_keys && chmod 600 .ssh/authorized_keys # types 'yes' when asked
exit # get back to root

# It is recommended to use public key cryptography instead of regular 
# password authentication and disable remote root access, so disable such options.
# Also, set serverside session keep-alives for long monitoring sessions. 
micro /etc/ssh/sshd_config
#
PermitRootLogin no
PasswordAuthentication no
ClientAliveInterval 60
ClientAliveCountMax 3
#

# Enable(autostart) SSH daemon(background process)
systemctl enable sshd
# NOTE:
# Use `ssh lotus@<ip_address>` for remote access later
###

### Miscellanous

# Setup NTP 
# NOTE: It is important to be consistent with time over the network
micro /etc/systemd/timesyncd.conf
#
[Time]
NTP=0.arch.pool.ntp.org 1.arch.pool.ntp.org 2.arch.pool.ntp.org 3.arch.pool.ntp.org
FallbackNTP=0.pool.ntp.org 1.pool.ntp.org 0.fr.pool.ntp.org
#

timedatectl set-ntp true 
systemctl enable systemd-timesyncd

# Set own timezone
timedatectl set-timezone Europe/Kiev

# Set locale(language)
locale-gen
localectl set-locale LANG=en_US.UTF-8

###

# Install GRUB bootloader(software that starts the OS)
grub-install --target=x86_64-efi --bootloader-id=GRUB --efi-directory=/boot/efi
grub-mkconfig -o /boot/grub/grub.cfg


# Finally reboot into newly installed OS
exit # get back to Live environment
reboot

//TODO: Kernel params networking tweaks

### Paru

rustup default stable
sudo pacman -S --needed base-devel
git clone https://aur.archlinux.org/paru.git
cd paru
makepkg -si
cd .. && rm -rf paru

sudo micro /etc/paru.conf
#
[bin]
FileManager=nnn
#

#### Pacman Mirrors
sudo micro /etc/pacman.d/mirrorlist

# Primary
Server = https://mirror.osbeck.com/archlinux/$repo/os/$arch

# Generated by Reflector
Server = https://mirror.pkgbuild.com/$repo/os/$arch
Server = https://phinau.de/arch/$repo/os/$arch
Server = https://archlinux.thaller.ws/$repo/os/$arch
Server = https://ftp.halifax.rwth-aachen.de/archlinux/$repo/os/$arch
Server = https://mirror.f4st.host/archlinux/$repo/os/$arch

### ZFS

micro /etc/pacman.conf # Add zfs repository to pacman
# [archzfs]
# Server = http://archzfs.com/$repo/$arch
# Server = http://mirror.sum7.eu/archlinux/archzfs/$repo/$arch
# Server = https://mirror.biocrafting.net/archlinux/archzfs/$repo/$arch
# Server = https://mirror.in.themindsmaze.com/archzfs/$repo/$arch
# Server = https://zxcvfdsa.com/archzfs/$repo/$arch

# Get, add, sign GPG key for repo
wget archzfs.com/archzfs.gpg 
pacman-key -a archzfs.gpg
pacman-key --lsign-key DDF7DB817396A49B2A2723F7403BD972F75D9D76
rm -f archzfs.gpg

# Install ZFS
pacman -S zfs-dkms
micro /etc/pacman.conf
#
# IgnorePkg = zfs-dkms
#

# Enable ZFS required services to be started on boot
systemctl enable zfs-import-cache
systemctl enable zfs-import.target
systemctl enable zfs-mount
systemctl enable zfs-mount.target

#### Sector Storage over HDDs

# List ids of desired HDDs
ls /dev/disk/by-id | grep ata

# Create ZFS Raidz2 pool over available HDDs
zpool create -f -o ashift=12 -m /lotus/sectors sectors raidz2 <ids>

sudo zpool replace -w -f -o ashift=12 sectors

#### Chain Storage over Nvmes

# Get ids of desired Nvmes
ls /dev/disk/by-id

# Create ZFS Raidz pool over two nvmes
zpool creat -f storage raidz

### Lotus

sudo git clone https://github.com/filecoin-project/lotus /usr/src/lotus
cd /usr/src/lotus && sudo git checkout v1.6.0

paru --no-confirm -Syu opencl-icd-loader gcc git bzr jq pkg-config hwloc

export RUSTFLAGS="-C target-cpu=native -g"
export FFI_BUILD_FROM_SOURCE=1

sudo micro /etc/environment
# RUSTFLAGS="-C target-cpu=native -g"
# FFI_BUILD_FROM_SOURCE=1

sudo make clean all
sudo make install

#### Daemon
Ensure you have your HDDs for sectors and SSD for chain mounted
Systemd

export LOTUS_PATH=/chain/lotus-repo
micro /etc/environment
# LOTUS_PATH=/chain/lotus-repo


#Systemd service
micro /etc/systemd/system/lotus-daemon.service
#
[Unit]
Description=Lotus Daemon
After=network-online.target
Requires=network-online.target

[Service
ExecStart=/usr/local/bin/lotus daemon
ExecStop=/usr/local/bin/lotus daemon stop
Restart=always
RestartSec=10

Environment=LOTUS_PATH=/lotus/daemon-repo
Environment=FIL_PROOFS_PARAMETER_CACHE=/lotus/params

Environment=GOLOG_OUTPUT=file
Environment=GOLOG_FILE=/lotus/logs/daemon.log
Environment=GOLOG_LOG_FMT=json
Environment=RUST_LOG=info
Environment=RUST_BACKTRACE=full
StandardOutput=append:/lotus/logs/daemon_rust.log

MemoryAccounting=true
MemoryHigh=8G
MemoryMax=10G
LimitNOFILE=1048576:1048576

[Install]
WantedBy=multi-user.target
#

# Lotus config
micro /chain/lotus-repo/config.toml
#
[API]
ListenAddress = "/ip4/192.168.1.103/tcp/1234/http"
RemoteListenAddress = "/ip4/178.212.197.65/tcp/1234/http"
Timeout = "30s"

[Backup]
DisableMetadataLog = false

[Libp2p]
ListenAddresses = ["/ip4/192.168.1.103/tcp/12345"]
AnnounceAddresses = ["/ip4/178.212.197.65/tcp/12345"]
NoAnnounceAddresses = ["/ip4/192.168.1.103/tcp/12345"]
ConnMgrLow = 150
ConnMgrHigh = 180
ConnMgrGrace = "20s"

[Pubsub]
Bootstrapper = false
RemoteTracer = "/dns4/pubsub-tracer.filecoin.io/tcp/4001/p2p/QmTd6UvR47vUidRNZ1ZKXHrAFhqTJAD27rKL9XYghEKgKX"

[Client]
UseIpfs = false
IpfsOnlineMode = false
IpfsMAddr = ""
IpfsUseForRetrieval = false
SimultaneousTransfers = 20

[Metrics]
Nickname = ""
HeadNotifs = false

[Wallet]
RemoteBackend = ""
EnableLedger = false
DisableLocal = false

[Fees]
 DefaultMaxFee = "0.007 FIL"

[Chainstore]
EnableSplitstore = false
  [Chainstore.Splitstore]
    HotStoreType = "badger"
    TrackingStoreType = ""
    MarkSetType = ""
    EnableFullCompaction = false
    EnableGC = false
    Archival = false

#### Miner TODO

#### Workers

##### Environment

sudo mkdir /lotus/logs /lotus/sealing && sudo chmod -R 0777 /lotus

sudo micro /etc/environment
#
RUSTFLAGS=-C target-cpu=native -g
FFI_BUILD_FROM_SOURCE=1
FFI_USE_CUDA=1

LOTUS_WORKER_PATH=/lotus/worker-repo
FIL_PROOFS_PARAMETER_CACHE=/lotus/params
FULLNODE_API_INFO= # sensetive info, should be gathered from lotus auth api-info --perm admin
MINER_API_INFO= # sensetive info, should be gathered from lotus-miner auth api-info --perm admin
#

sudo lotus-worker storage attach --init --seal /lotus/sealing # or --store for storing sectors

##### Service

sudo micro /etc/systemd/system/lotus-worker.service
#
[Unit]
Description=Lotus Worker
After=network.target

[Service]
ExecStart=/usr/local/bin/lotus-worker run --no-local-storage --addpiece=true --precommit1=true --unseal=true --precommit2=false --commit=false --parallel-fetch-limit 10 # Worker type defining values
Restart=always
RestartSec=10
LimitNOFILE=1048576:1048576
LimitMEMLOCK=1000000000 # This is need for PC1 workers

Environment=LOTUS_WORKER_PATH=/lotus/worker-repo # Place where all the worker data is stored
Environment=FIL_PROOFS_PARAMETER_CACHE=/lotus/params
Environment=FIL_PROOFS_PARENT_CACHE=/lotus/parents
Environment=TMPDIR=/lotus/tmp

Environment=BELLMAN_CUSTOM_GPU="GeForce RTX 3090:10496"
Environment=BELLMAN_CPU_UTILIZATION=0 # 

Environment=FIL_PROOFS_MAXIMIZE_CACHING=1
Environment=FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1
Environment=FIL_PROOFS_USE_GPU_TREE_BUILDER=1
Environment=FIL_PROOFS_USE_MULTICORE_SDR=1 # This option depends on CCX

Environment=GOLOG_OUTPUT=file
Environment=GOLOG_FILE=/lotus/logs/worker.log
Environment=GOLOG_LOG_FMT=json # Remove if better readability needed
Environment=RUST_LOG=info
Environment=RUST_BACKTRACE=full
StandardOutput=append:/lotus/logs/worker_rust.log

Environment=FULLNODE_API_INFO= # sensetive info, should be gathered from lotus auth api-info --perm admin
Environment=MINER_API_INFO= # sensetive info, should be gathered from lotus-miner auth api-info --perm admin

[Install]
WantedBy=multi-user.target
#

sudo systemctl enable lotus-worker

#### With GPU
For workers and miner with GPU it seems to be important to add a user running the worker to `render` group.

sudo usermod -a -G render lotus

#### Swapiness
It is important to lower swapiness from default 60 in order to lesser use swap when it is not needed. This would in general speedup all the process and will fix possible OOM killer issues.

sudo sysctl -w vm.swappiness=1
echo 'vm.swappiness=1' | sudo tee -a /etc/sysctl.d/99-swappiness.conf

### Network Tuning

sudo micro /etc/sysctl.d/10-network.conf
#
# For Daemon and Miner to speedup QUIC
net.core.rmem_max=26214400
net.core.rmem_default=26214400
# For faster sector transportation
net.core.default_qdisc=fq
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 16384 16777216
net.ipv4.tcp_congestion_control=bbr2
#

#### Verifying Hardware TODO
It is recommented to run tests/benchamarks against hardware to check if operative and stable.
This could be done along Filecoin bechmarking with `lotus-bench`

##### GPU
While running nvtop in backround, run benchmark to see that GPU works.
NOTE: This will download params(~2.5Gb) for minimally required sector size of 512kb

BELLMAN_CUSTOM_GPU="GeForce RTX 3090:10496" FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1 FIL_PROOFS_USE_MULTICORE_SDR=1 FIL_PROOFS_USE_GPU_TREE_BUILDER=1 FIL_PROOFS_MAXIMIZE_CACHING=1 ./lotus-bench sealing --skip-unseal --skip-commit2

### IPFS TODO

### [Lotus Farcaster](https://github.com/s0nik42/lotus-farcaster) TODO

pacman -S grafana prometheus prometheus-node-exporter

### Log Rotation

paru -S logrotate
sudo micro /etc/logrotate.conf
#
/lotus/logs/* {
    rotate 14
    daily # lotus generates tonns of logs, so daily is recommended
    compress
    copytruncate
    olddir /lotus/logs/old
    su Wondertan wheel # TODO: This should be a lotus user instead
}
#
mkdir /lotus/logs/old
chmod +x /lotus/logs/old

## Maintaining

### Renewing Sectors

To detect expiring sectors use:

lotus-miner sectors check-expire

To renew expiring sectors use:

lotus-miner sectors renew

The current setup has `/etc/systemd/system/renew.timer`,
which runs automatic sectors renewal every 30 days.

### Drives

Illumintates LED on the specific drive

sudo ledctl locate=/dev/sda

Reverses the above comand

sudo ledctl locate_off=/dev/sda

### Deals
In case hardware needs to be stopped, new storage deals has to be rejected.

lotus-miner storage-deals reject --online --offline

To turn storage-deal recieval back on

lotus-miner storage-deals reset

### Security

#### Mitigate BruteForce attacks
Setup Fail2ban or [sshguard](https://wiki.archlinux.org/title/Sshguard) for that, but latter one is more recommended.

#### Reverse path filtering

sudo micro /usr/lib/sysctl.d/50-default.conf
#
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
#

### Parallel SSH
To execute multiple commands at the same time

### Aliases

micro /etc/bash.bashrc
#
alias jobs='watch -n 5 -c -x lotus-miner sealing jobs'
alias workers='watch -n 5 -c -x lotus-miner sealing workers'
alias sectors='watch -n 5 -c -x lotus-miner sectors list --fast'
alias pledge='lotus-miner sectors pledge'
alias basefee='lotus chain head | xargs lotus chain getblock | jq -r .ParentBaseFee'
alias pending='lotus mpool pending --local --cids'
alias wallets='lotus wallet list -i'

alias minerlogs='tail -f -n 100 /lotus/logs/miner.log'
alias daemonlogs='tail -f -n 100 /lotus/logs/daemon.log'
alias minerlogsrust='tail -f -n 100 /lotus/logs/miner_rust.log'
alias daemonlogsrust='tail -f -n 100 /lotus/logs/daemon_rust.log'

# or for worker
alias workerlogs='tail -f -n 100 /lotus/logs/worker.log'
alias worker2logs='tail -f -n 100 /lotus/logs/worker2.log'
alias workerlogsrust='tail -f -n 100 /lotus/logs/worker_rust.log'
alias worker2logsrust='tail -f -n 100 /lotus/logs/worker2_rust.log'
#

### Auto pledging
Autopledging means scheduling sector production once in some time period. The metric is solely based on sector bandwidth of your setup and have to be approximated accordingly.

For auto pledging systemd timers can be used.

sudo micro /etc/systemd/system/pledge.timer
#
[Unit]
Description=Auto Pledging Timer

[Timer]
OnUnitActiveSec=21min

[Install]
Unit=pledge.service
WantedBy=timers.target
#

sudo micro /etc/systemd/system/pledge.service
#
[Unit]
Description=Auto Pledging Service
Wants=lotus-miner.service

[Service]
User=Wondertan
Type=oneshot
ExecStart=lotus-miner sectors pledge
Environment=LOTUS_MINER_PATH=/lotus/miner-repo
#

# to trigger the timer run this
sudo systemctl start pledge.timer 

### Updating

To update lotus software exec the command below, but change the version accordingly.

cd /lotus/lotus && make dist-clean && git fetch -t && git checkout v1.25.1 && make dist-clean clean all && sudo APP=lotus-shed make lotus-shed install install-app && cd -

### Disk Space Cleaning

To detect what are the most heaviest files use:

sudo du -ahxd 4 -t 1G /

To clean rust cache:

cargo install cargo-cache
cargo cache -a

To clean go cache:

go clean -modcache

To clean pacman cache:

paru -c
sudo paccache -rk 1

### Sectors stucks
In case some sectors stuck on PreCommitWait or CommitWait, check if the chain sync is going without issues first and then check mpool for stucked transactions.

# check sync is done
lotus sync wait #blocks if sync is in progress

# check mpool
lotus mpool pending --local --cids

# replace the first message if showed any
lotus mpool replace --auto <cid>

In case sectors is stuck on some failed state, e.g. PreCommitFailed, sector's state machine have to be triggered with same Failed state as an argument.

lotus-miner sectors update-state --really-do-it <sec_num> PreCommitFailed

### Chain size
> NOTE: This rule applies to the latest Lotus version by 8.06.2021 and it could be simplified in observable future. For reference check: https://docs.filecoin.io/get-started/lotus/chain/#compacting-the-chain-data

Lotus is an application that handles Filecoin blockchain and stores its data. Due to the nature of Filecoin blockchain it grows very fast and the bigger it is the more IO bandwith it requires due to some temporary software limitation. Unless it is not fixed, we need to regularly check the IO usage and track if chain does not start to fall out of sync. Currently, this happens every two weeks and have to be resolved with the instruction below.

1. Export snapshot

lotus chain export --recent-stateroots 900 --skip-old-msgs /lotus/snapshots/recent.car 

or

cd /usr/src/lotus && ulimit -n 99999 && sudo swapoff --all && make lotus-shed && ./lotus-shed export --repo /lotus/daemon-repo/ --recent-stateroots 900 --skip-old-msgs /lotus/snapshots/recent.car && sudo swapon --all && cd -

2. Stop Lotus daemon

sudo systemctl stop lotus-daemon

3. Remove old state

rm -rf /lotus/daemon-repo/datastore/chain/*

4. Apply exported snapshot

lotus daemon --halt-after-import --import-snapshot /lotus/snapshots/recent.car 

5. Once exported, start daemon again

sudo systemctl start lotus-daemon