NiM - The NixOS Machine

hosted on Hetzner Dedicated
runing NixOS
encrypred ZFS
ephemeral root FS

Resources

Setup

See: https://www.hetzner.com/dedicated-rootserver/

6 Core (12 Thread)
64 ECC RAM
2 x 512GB NVMe
~~BIOS~~ UEFI ^[1]

Partitioning:

/boot: ext2, mirrored

zpools

rpool
- local
  - nix
  - root
- system
  - journald
- data
  - data
  - app1
  - app2
  - …

# zpool: -O for FS options, -o for pool options
zpool create \
    -o ashift=12 \
    -R /mnt \
    -O mountpoint=none \
    -O canmount=off \
    -O xattr=sa \
    -O acltype=posix \
    -O atime=off \
    -f \
    rpool mirror \
    /dev/... \
    /dev/...

zfs create -o mountpoint=none rpool/local
zfs create -o mountpoint=none rpool/system
zfs create -o mountpoint=none rpool/data

# Create root FS and take an empty snapshot

zfs create -o mountpoint=legacy rpool/local/nix
zfs create -o mountpoint=legacy rpool/data/data
zfs create -o mountpoint=legacy rpool/system/root
zfs snapshot rpool/system/root@blank

zfs create -o reservation=1G -o mountpoint=none rpool/reserved

For a notebook

see https://openzfs.github.io/openzfs-docs/Getting Started/NixOS/Root on ZFS.html

zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -R /mnt \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=on \
    -O dnodesize=auto \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=none \
    rpool /dev/xxx
  
zfs create \
    -o canmount=off \
    -o mountpoint=none \
    -o encryption=on \
    -o keylocation=prompt \
    -o keyformat=passphrase \
    rpool/enc

zfs create -o canmount=off rpool/enc/local
zfs create -o canmount=off rpool/enc/system
zfs create -o canmount=off rpool/enc/user

# Root FS
zfs create -o canmount=on -o mountpoint=/ rpool/enc/system/root
zfs snapshot rpool/enc/system/root@blank

# Homedir
zfs create -o canmount=on -o mountpoint=/home rpool/enc/user/home

# System dirs
zfs create -o canmount=on -o mountpoint=/var/log rpool/enc/system/log

# Boot stuff
mkdir /mnt/boot
mount /dev/vda1 /mnt/boot

ZFS

https://nixos.wiki/wiki/ZFS
- See Unlock encrypted zfs via ssh on boot
https://grahamc.com/blog/nixos-on-zfs
Ephemeral root FS: https://grahamc.com/blog/erase-your-darlings

Dataset properties

compression=lz4
- Specifying on instead of lz4 or another specific algorithm will always pick the best available compression algorithm. ^[2]
For wherever /var lives:
- xattr=sa for Journald
- acltype=posixacl also for Journald
relatime=on for reduced stress on SSDs

The following is a list of dataset properties which are often useful, but do have drawbacks:

atime=off disables if a file's access time is updated when the file is read. This can result in significant performance gains, but might confuse some software like mailers.
- Nix doesn't use atime, so setting atime=off on the /nix FS is fine

NixOS requires (as of 2020-04-11) mountpoint=legacy for all datasets. NixOS does not yet have tooling to require implicitly created ZFS mounts to settle before booting, and mountpoint=legacy plus explicit mount points in hardware-configuration.nix will ensure all your datasets are mounted at the right time

Hetzner Dedicated

https://jappie.me/the-nix-mutli-monolith-machine-nmmm.html