Learn NixOS by turning a Raspberry Pi into a Wireless Router

I recently moved, and my new place has a relatively small footprint. (Yes, I moved during the COVID-19 pandemic. And yes, it was crazy.) I quickly realizied that was going to need a wireless router of some sort, or more formally, a wireless access point (WAP). Using my Ubuntu laptop's "wireless hotspot" capability was a nice temporary solution, but it had a few serious drawbacks:

• The wireless internet goes out whenever I would travel with the laptop,
• The laptop had to be close to the modem, so that it could be plugged into ethernet, making my laptop not even portable within the appartment,
• The SSID was my laptop's hostname,
• The WPA password would be set to a random string whenever the hotspot was started, and so
• Whenever I moved my laptop I would also need to reset the credentials on all of my wireless devices!

Additionally, some of my coworkers are Nix true believers. While I had read the NixOS pamphlets, and listened politely when the faithful came knocking on my door at inconvenient times, I had never walked the path of functional Linux enlightenment myself.

Consider this my first few steps down the /etc/nixos path, because, while I lacked a WiFi router, I did have an errant Raspberry Pi 3B+ lying around…

Too Long; Didn't Read

Be sure to fill out the SSID and WPA passphrase in the file below.

/etc/nixos/configuration.nix

{ config, pkgs, lib, ... }:
{
  # NixOS wants to enable GRUB by default
  boot.loader.grub.enable = false;
  # Enables the generation of /boot/extlinux/extlinux.conf
  boot.loader.generic-extlinux-compatible.enable = true;

  # use an older kernel, so that we can actually boot
  boot.kernelPackages = pkgs.linuxPackages_4_19;

  # Needed for the virtual console to work on the RPi 3, as the default of 16M
  # doesn't seem to be enough. If X.org behaves weirdly (I only saw the cursor)
  # then try increasing this to 256M.
  boot.kernelParams = ["cma=32M"];

  # File systems configuration for using the installer's partition layout
  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
    };
  };

  # Recommended swap file is optional
  swapDevices = [ { device = "/swapfile"; size = 1024; } ];

  # packages
  environment.systemPackages = with pkgs; [ hostapd dnsmasq bridge-utils ];

  # add wireless service
  hardware.enableRedistributableFirmware = true;
  networking.wireless.enable = true;

  # set up wireless access point
  networking.networkmanager.unmanaged = [ "interface-name:wlan*" ]
    ++ lib.optional config.services.hostapd.enable "interface-name:${config.services.hostapd.interface}";
  services.hostapd = {
    enable = true;
    interface = "wlan0";
    hwMode = "g";
    ssid = "<YOUR NETWORK NAME HERE>";
    wpaPassphrase = "<YOUR PASSWORD HERE>";
  };

  # set up wireless static IP address
  networking.interfaces.wlan0.ip4 = lib.mkOverride 0 [ ];
  networking.interfaces.wlan0.ipv4.addresses =
    lib.optionals config.services.hostapd.enable [{ address = "192.168.0.1"; prefixLength = 24; }];

  # set up wireless DNS
  services.dnsmasq = lib.optionalAttrs config.services.hostapd.enable {
    enable = true;
    extraConfig = ''
      interface=wlan0
      bind-interfaces
      dhcp-range=192.168.0.10,192.168.0.254,24h
    '';
  };
  networking.firewall.allowedUDPPorts = lib.optionals config.services.hostapd.enable [53 67];
  services.haveged.enable = config.services.hostapd.enable;

  # Finally, bridge ethernet and wifi
  networking.bridges.br0.interfaces = [ "eth0" "wlan0" ];
}

Learning to use Nix

Nix is great! It's mental model is really neat and there are some fantastic ideas under the covers. However, the documentation has some glaring holes. It almost all cases it either assumes that:

1. You already know how to use Nix, or
2. You already are on a Nix machine.

This makes it very frustrating to actually get started, especially on non-standard harware such as the Raspberry Pi. A lot of these issues can be smoothed over by a friend who can act as your spirit guide. I write this as someone who has been a Linux user 20 years and who works on open source packaging problems (conda-forge).

The following are some basic Nix tips for helping get you started.

Don't use nix-env

The first bit of philosphy to understand is that while much of the Nix documentation touts the functional nature of its underlying language, the imporant aspect of Nix is that it is declarative.

Nix wants you to specify the configuration and layout of your Operating System (OS) ahead of time in a relatively static way. This is very different from how other Linux distributrions operate, which assume that you will procedurally build up your system from various available components, as needed. The declarative approach has advantages & disadvantages.

Advanatages:

• You can write out exactly what your OS is in a single text file (see above).
• Your OS can be built and tested before booting into it.
• Errors in configuration are caught and tested during the build process.

Disadvantages:

• You have to what you want in your OS before you build it.
• Changes to the OS configuration require a rebuild.

In many cases, the advantages here outweigh the disadvanatges. It is without question that Docker, CoreOS, and conda all owe a lot of conceptual inspiration to the work Nix has been performing for years.

Of course, even a functional OS has to have an escape valve. This is called nix-env and is a command line utility for creating & managing environments (a collection of packages) in an existing OS.

We want to create a dedicated device that, when it boots up, is a WAP. To this end, it is important that we declare everything in the configuration file. If we start creating environments willy-nilly, we won't obtain the proper boot behavior. This is very different from procedural OSes, where you can modify live configuration files that affect boot processes. Not so here! All boot config needs to be in declared!

(Unfortunately, much of the Nix documentation uses nix-env, because it assumes that you are a user on an existing Nix box just trying things out.)

The configuration.nix file

So where does this mysterious OS configuration file live? Well, the full path to this file is /etc/nixos/configuration.nix. It is written in the Nix language, and is used by the nixos-rebuild command line tool. We'll see this tool later to build the router's OS.

Again, unfortunately, when people report bugs or list a configuration snippet, they are almost always refering to this file. However, they specify that they are talking about this file. It is just known.

Now you know too.

You need to be root (which is shockingly easy)

Another issue that is not clear from the docs is that any serious command you might want to run needs to be run as root (or another user in the wheel group). However, the default aarch64 image boots into a user named nixos. This requires you to sudo su to become the root user to run rebuild commands (or any of the commands prefixed by sudo).

Also, oddly, in the initial image neither the nixos nor the root user have passwords. So you end up running sudo without needing a password. You will probably want to set a password with the passwd utility, or via user management in /etc/nixos/configuration.nix.

First Boot!

Assuming you have the SD card for your Raspberry Pi handy, take it out of the Pi and plug it into another (Linux) computer. We are going to need to flash it with a basic NixOS. You can find generic instructions for NixOS on a Raspberry PI here and instructions for NixOS on ARM here. However, I'll summarize the important bits here.

First, go to the 19.09 aarm64 landing page and download the latest NixOS image. It will be called something like nixos-sd-image-19.09.2435.9642f121eb1-aarch64-linux.img. We'll assume this is in your ~/Downloads folder.

Second, figure out what device your SD card is. If you just plugged it in, you can determine this by looking at the end of the output of the dmesg command. For example,

$ dmesg
[ 4591.053095] usb 3-10: new high-speed USB device number 5 using xhci_hcd
[ 4591.201911] usb 3-10: New USB device found, idVendor=14cd, idProduct=168a, bcdDevice= 0.01
[ 4591.201915] usb 3-10: New USB device strings: Mfr=1, Product=3, SerialNumber=2
[ 4591.201917] usb 3-10: Product: USB Mass Storage Device
[ 4591.201919] usb 3-10: Manufacturer: USB Device  
[ 4591.201921] usb 3-10: SerialNumber: 816820130806
[ 4591.202955] usb-storage 3-10:1.0: USB Mass Storage device detected
[ 4591.203140] scsi host10: usb-storage 3-10:1.0
[ 4592.205933] scsi 10:0:0:0: Direct-Access     USB Mass  Storage Device  1.00 PQ: 0 ANSI: 0
[ 4592.206338] sd 10:0:0:0: Attached scsi generic sg1 type 0
[ 4592.207288] sd 10:0:0:0: [sdb] 31116288 512-byte logical blocks: (15.9 GB/14.8 GiB)
[ 4592.207421] sd 10:0:0:0: [sdb] Write Protect is off
[ 4592.207425] sd 10:0:0:0: [sdb] Mode Sense: 03 00 00 00
[ 4592.207561] sd 10:0:0:0: [sdb] No Caching mode page found
[ 4592.207567] sd 10:0:0:0: [sdb] Assuming drive cache: write through
[ 4592.212993]  sdb: sdb1 sdb2
[ 4592.214220] sd 10:0:0:0: [sdb] Attached SCSI removable disk

This let's us know that the SD card is the /dev/sdb device and has two partitions. Yours might be called /dev/sdc or something similar. It also might have more than two partitions. That is totally normal at this step.

Third, we need to copy the NixOS image over to the SD card. We'll do this with the dd command. The SD card should not be mounted right now. Run the following command with the path to the image and the SD card device replaced as appropriate.

$ sudo dd if=~/Downloads/nixos-sd-image-19.09.2435.9642f121eb1-aarch64-linux.img of=/dev/sdb

Great! At this point, we have now flashed the SD card with our new NixOS!

Fourth, it will save us a lot of typing if we copy over an existing configuration file to the SD card. Copy the text of the /etc/nixos/configuration.nix file at the top of this article to a new file, let's call it ~/Downloads/config.nix. Fill in this file with the SSID and password that you want your network to have. Then, run the following commands to copy the configuration file to the SD card. Again, modify the paths here as needed.

$ mkdir -p ~/mount
$ sudo mount /dev/sdb2 ~/mount
$ sudo mkdir -p ~/mount/etc/nixos
$ sudo cp ~/Downloads/config.nix ~/mount/etc/nixos/configuration.nix
$ sudo umount ~/mount

Fifth, now unplug the SD card from your main machine, plug it into the Raspberry Pi! Attach the ethernet, keyboard, monitor, and power supply to the Pi you will be booting up into your first NixOS!

Build the Router OS

The operating system that we have just booted into on the Raspberry Pi is a generic image that does not use the configuration file that we copied over. We need the Nix tools to be able to build the image. Luckily, we are now on a Nix machine!

Sixth, we need to be root to run a lot of these tools. However, we booted into the nixos user. To make the rest of the process easier, let's just log in as root with the following:

$ sudo su

Seventh, now let's verify that we have a working internet connection and that the network devices exist. To do so, start with a simple ping that should looks like:

$ ping 8.8.8.8 -c 3
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=19.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=19.3 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=24.2 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 3ms
rtt min/avg/max/mdev = 19.098/20.847/24.154/2.345 ms

If you only see packet loss, then this means you do not have internet on the Pi and you cannot proceed. Nix requires internet access to build in all realistic scenarios.

Next run the ifconfig command and verify that both eth0 (the ethernet device) and wlan0 (the wireless device) exist.

$ ifconfig
...
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 37.128.100.42  netmask 255.255.255.0 
        ...
wlan0: ...
...

Eighth, we finally get to build the new OS for our router! We do this with the nixos-rebuild command. All you have to do is run the following command and watch the text scroll by.

$ nixos-rebuild switch -v

Ninth, we can now reboot into our router! Just run:

$ reboot

The default boot option has been changed to be the OS we just built, which is the same as the second option in the bootloader's listing. The original image will be the last boot option (which for various reasons says it is from 1970). This allows us to always get back to a working NixOS to do another nixos-rebuild if something went terribly wrong and we need to do another rebuild. For example, this could happen if there was a typo in the configuration file. If you ever modify /etc/nixos/configuration.nix, you'll need to rebuild & reboot. The rebuild & reboot cycle is the fundemental implication of having a declarative OS.

Tenth, if you want to verify that everything is working on your router after reboort, you can log in as root and run ifconfig again. This time, you should see eth0, wlan0, and br0 devices. Of course, the ping command should also work too.

Eleventh, you should now be able to connect a wireless device like a phone or a laptop to your shiny new WAP!

Deep dive into configuration.nix

For the truly inquisitive who are still reading, let's breakdown what the different parts of the configuration file actually mean, and how they help define our wireless router.

Bootloader

boot.loader.grub.enable = false;
boot.loader.generic-extlinux-compatible.enable = true;

These lines are part of the standard ARM configuration and help speed up the boot process a bit by disabling the fancy GRUB bootloader.

Older Kernel

boot.kernelPackages = pkgs.linuxPackages_4_19;

This line pins our packages to use and older version of the Linux kernel (v4.19). This is super critical because, without this line, nixos-rebuild will end up grabbing a kernel in the v5.x series. Unfortunately, the Raspberry Pi 3 has problems starting up these more recent kernels and the Pi will hang indefinitely on boot. Using an older kernel version avoids this problem for the time being.

Console Memory

boot.kernelParams = ["cma=32M"];

This line gives the console more memory than the default value of 16 MB. The Raspberry Pi seems to need this.

Partition Layout

fileSystems = {
  "/" = {
    device = "/dev/disk/by-label/NIXOS_SD";
    fsType = "ext4";
  };
};

The above specifies where the root file system lives. It is part of the standard ARM configuration.

Swap

swapDevices = [ { device = "/swapfile"; size = 1024; } ];

This line gives the machine some extra virtual memory, which is always a good idea.

Router packages

environment.systemPackages = with pkgs; [ hostapd dnsmasq bridge-utils ];

Unlike procedural OSes, we list all of the packages that we need inside the configuration file itself. This ensures that our router is running exactly the software that what we want it to. In this case, we only need three packages to enable the Pi to act as an access point, provide a domain name service, and bridge the ethernet and the WiFi device.

For comparison, in Ubunutu, we would install these packages after we installed Ubuntu itself. In Nix, we install the OS and the packages at the same time!

Enable the wireless device

hardware.enableRedistributableFirmware = true;
networking.wireless.enable = true;

These lines simply allow the wireless card to be used on the Raspberry Pi in the simplest possible way.

Set up the wireless access point

networking.networkmanager.unmanaged = [ "interface-name:wlan*" ]
  ++ lib.optional config.services.hostapd.enable "interface-name:${config.services.hostapd.interface}";
services.hostapd = {
  enable = true;
  interface = "wlan0";
  hwMode = "g";
  ssid = "<YOUR NETWORK NAME HERE>";
  wpaPassphrase = "<YOUR PASSWORD HERE>";
};

Because we are installing the router packages along with the OS, we also need to configure these packages at the same time we configure the OS itself. The first line here tells Nix not to use normal networking management on the wlan0 device. This is beacuse we'll be managing it as a WAP ourselves.

The remaining lines configure the access point, including the SSID and password for the wireless network.

Set up aireless static IP address

networking.interfaces.wlan0.ip4 = lib.mkOverride 0 [ ];
networking.interfaces.wlan0.ipv4.addresses =
  lib.optionals config.services.hostapd.enable [{ address = "192.168.0.1"; prefixLength = 24; }];

Now, we would like the router itself to have a conistent IP address. We set this in the second line above as 192.168.0.1, though any value in 192.168.x.x would work equally well. However, just providing the static IP on it's own is not enough. This is because NixOS will verify that wlan0 does not have an IP address during the nixos-rebuild process. Since we are giving wlan0 an IP address, we need to turn off the IP address checking. If we do not remove this verification, the whole OS build process will fail. The first line in the above snippet removes this check with the mkOverride function.

Set up the wireless DNS

services.dnsmasq = lib.optionalAttrs config.services.hostapd.enable {
  enable = true;
  extraConfig = ''
    interface=wlan0
    bind-interfaces
    dhcp-range=192.168.0.10,192.168.0.254,24h
  '';
};
networking.firewall.allowedUDPPorts = lib.optionals config.services.hostapd.enable [53 67];
services.haveged.enable = config.services.hostapd.enable;

The collection of lines above allows the wlan0 device to operate as a domain name server, proxying a real DNS online. It also sets the range of IP addresses that the router will issue to other network devices. This is seen in the dhcp-range=192.168.0.10,192.168.0.254 portion. The first IP address is the lowest address the router will issue, and the second IP address is the highest.

Bridge ethernet and wifi

networking.bridges.br0.interfaces = [ "eth0" "wlan0" ];

Lastly, we 'bridge' the ethernet and wireless devices. This allows network traffic to flow through the eth0 connection and into the wlan0.

Reflections

This was a really fun weekend project! I certainly learned a lot about Nix, Raspberry Pis, how to set up various parts of the Linux networking stack that I had never explored before. My main wish in this process was that Nix had better documentation that was more aimed at,

1. People who had never used Nix before, and
2. People who are trying to build dedicated devices.

A lot of the Nix documentation seems to be aimed at a very particular kind of desktop user: someone who already has Nix installed! Such users represent an important use case, and the nix build configurations are easy enough to read. However, I definitely think there is onboarding improvement work to be done in the Nix ecosystem.

So, will I ever go back? I don't think so! This router was so cheap (~$40) and the Raspberry Pi 3B+ is so powerful that I get amazing performance throughout my entire appartment. If it ever breaks, the Pi will be trivial to replace. I am really happy with what I created. Even if this little project isn't original, it solves a real problem in my day-to-day life.

In terms of NixOS as a Linux distribution, I think I have now drunk the kool-aid myself. Nix has so many incredible advantages that (as a control freak who builds his own WiFi router) I just can't ignore or give up. The feature of Ubuntu that was keeping me on that distribution for so long was that, "it just works"

But Nix "just works" too. The only catch is that you need to know what "it" is that you want working ahead of time. I am also comfortable with responsibly using environments, so I think increases my willingness to jump into a new OS framework. I am a little worried about moving from Ubuntu to Nix on an existing machine, but that is what external hard drive backups are for!

That is all folks! Thanks for reading