# Nix Flakes  **Toward Portable, Reproducible and Dependency-Isocated General Purpose Development Environment** Yueh-Shun Li (ShamrockLee) --- ## Build from source * Where's the binary release? * It's FOSS! Build it from source!  --- ## Build from source * But how? * Just run ```console $ ./configure $ make ``` * But ... ```console ... ./configure: line 17717: syntax error near unexpected token `OPENSSL,' ./configure: line 17717: `PKG_CHECK_MODULES(OPENSSL, openssl)' ``` * Run `sudo apt install libssl-dev` --- ## What's wrong with installing dependencies system-wide? * Hard to clean up * Hard to reproduce on other machines * Conflicts with system libraries / dependencies from other projects * In-project dependency conflict * Root privilege required ---- ## `apt` with `equivs` | | apt install | apt + equivs | |-|-|-| |garbage collection|&cross;|&check;| |system dep isolation|&cross;|&cross;| |in-project isolation|&cross;|&cross;| |restoreability|&cross;|&cross;| |unprivileged|&cross;|&cross;| ---- ## Virtual environment? | | apt | equivs | venv | |-|-|-|-| |garbage collection|&cross;|&check;|&check;| |system dep isolation|&cross;|&cross;|&check;| |in-project isolation|&cross;|&cross;|&cross;| |restoreability|&cross;|&triangle;|&check;| |unprivileged|&cross;|&cross;|&check;| |interactivity|&check;|&check;|&check;| |general purpose|&check;|&check;|&cross;| ---- ## How about Docker? | |apt|equivs|venv|docker| |-|-|-|-|-| |system dep iso|&cross;|&cross;|&check;|&check;| |in-project iso|&cross;|&cross;|&cross;|&triangle;| |restoreability|&cross;|&check;|&check;|&triangle;/&check;| |unprivileged|&cross;|&cross;|&check;|&cross;/&check;| |interactivity|&check;|&check;|&check;|&triangle;| |general purpose|&check;|&check;|&cross;|&check;| |patchability|&cross;|&cross;|&cross;|&check;| --- ## Why Nix Flakes * Dependency isolation * minimal conflict by design * Declarativity * text configuration contains all * Portablility * unprivileged and installation-free (on Linux) * General purpose * 80000+ packages available * builders for languages and frameworks --- ## What is Nix * A "purely functional" package manager * a purely functional, lazy-evaluated language (the Nix Language) * a directory containing hash-prefixed packkage outputs (the Nix Store) * a sandboxed build environment --- ## How Nix gets derivations ```plantuml @startuml state "Nix Language Level" as Language #lightblue { state "User-defined derivation" as UserDrv UserDrv : writeShellScriptBin ''\n echo "Hello, World!"\n'' state "Calling builtin function `derivation`" as CallingDerivation CallingDerivation : derivation {\n name = "myhello";\n system="x86_64-linux";\n builder="/nix/store/51sszq...-bash-5.2-p15/bin/bash";\n args = [ ... ];\n # ...\n} state "Derivation as an attribute set" as DrvAsAttrs DrvAsAttrs : {\n type="derivation";\n drvPath = "/nix/store/9rgj55...-myhello.drv";\n outPath = "/nix/store/8lq2rd...-myhello";\n # ...\n} UserDrv -[#DarkBlue]-> CallingDerivation : evaluate CallingDerivation -[#DarkBlue]-> DrvAsAttrs : evaluate } state "Nix Store Level" as Store #BurlyWood { state "Store derivation" as StoreDrv StoreDrv : /nix/store/9rgj55...-myhello.drv\nDerive([("...","...")],[("...")],...) state "Output store path" as OutPath OutPath : /nix/store/8lq2rd...-myhello\n└── bin\n └── myhello StoreDrv -[#DarkRed]-> OutPath : realize (as needed) } CallingDerivation -right[#DarkBlue]-> StoreDrv : instantiate @enduml ``` --- ## How Nix get derivations ```plantuml @startuml state "Nix Language Level" as Language #lightblue { state "User-defined derivation" as UserDrv UserDrv : writeShellScriptBin ''\n echo "Hello, World!"\n'' state "Derivation as an attribute set" as DrvAsAttrs DrvAsAttrs : {\n type="derivation";\n drvPath = "/nix/store/9rgj55...-myhello.drv";\n outPath = "/nix/store/8lq2rd...-myhello";\n # ...\n} UserDrv -[#DarkBlue]-> DrvAsAttrs : evaluate } @enduml ``` * `drvPath` and `outPath` calculated at eval time * Hash prefix calculated from the build instruction (with all the dependencies) --- ## Purely functional <img src=https://upload.wikimedia.org/wikipedia/commons/3/3b/Function_machine2.svg alt="a function takes inputs and produces output" style="background-color: white;" /> * Immutability * a package is never modified * definition change produces package * Deterministic * A package is determined by (the hash of) the build process and all its dependency --- ## Hash-separated store path ```graphviz digraph FortuneRunDepsGraph { "b4vbia29wmzbvsxsq8g7ygk08l5zxvr0-fortune-mod-3.18.0" [label = "fortune-mod-3.18.0", shape = box, style = filled, fillcolor = "#ff0000"]; "84xlyz6zafcff8ddb3ggqzzckyzyvxiw-recode-3.7.12" -> "b4vbia29wmzbvsxsq8g7ygk08l5zxvr0-fortune-mod-3.18.0" [color = "black"]; "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" -> "b4vbia29wmzbvsxsq8g7ygk08l5zxvr0-fortune-mod-3.18.0" [color = "red"]; "84xlyz6zafcff8ddb3ggqzzckyzyvxiw-recode-3.7.12" [label = "recode-3.7.12", shape = box, style = filled, fillcolor = "#ff0000"]; "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" -> "84xlyz6zafcff8ddb3ggqzzckyzyvxiw-recode-3.7.12" [color = "green"]; "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" [label = "glibc-2.37-8", shape = box, style = filled, fillcolor = "#ff0000"]; "4563gldw8ibz76f1a3x69zq3a1vhdpz9-libidn2-2.3.4" -> "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" [color = "blue"]; "jd99cyc0251p0i5y69w8mqjcai8mcq7h-xgcc-12.2.0-libgcc" -> "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" [color = "magenta"]; "4563gldw8ibz76f1a3x69zq3a1vhdpz9-libidn2-2.3.4" [label = "libidn2-2.3.4", shape = box, style = filled, fillcolor = "#ff0000"]; "567zfi9026lp2q6v97vwn640rv6i3n4c-libunistring-1.1" -> "4563gldw8ibz76f1a3x69zq3a1vhdpz9-libidn2-2.3.4" [color = "burlywood"]; "567zfi9026lp2q6v97vwn640rv6i3n4c-libunistring-1.1" [label = "libunistring-1.1", shape = box, style = filled, fillcolor = "#ff0000"]; "jd99cyc0251p0i5y69w8mqjcai8mcq7h-xgcc-12.2.0-libgcc" [label = "xgcc-12.2.0-libgcc", shape = box, style = filled, fillcolor = "#ff0000"]; } ``` ```console $ nix build --no-link github:NixOS/nixpkgs/ecb441f22067ba1d6312f4932a7c64efa8d19a7b#fortune $ nix path-info github:NixOS/nixpkgs/ecb441f22067ba1d6312f4932a7c64efa8d19a7b#fortune /nix/store/b4vbia29wmzbvsxsq8g7ygk08l5zxvr0-fortune-mod-3.18.0 $ nix-store --query --graph /nix/store/b4vbia29wmzbvsxsq8g7ygk08l5zxvr0-fortune-mod-3.18.0 ``` --- ## Hash-separated store path * Let's override the `recode` dependency of `fortune` to build with `gcc6`, but still build fortune with `gcc12` ```console $ cd ~/Project/NixOS/nixpkgs $ git switch -d ecb441f22067ba1d6312f4932a7c64efa8d19a7b $ nix repl Welcome to Nix 2.15.0. Type :? for help. nix-repl> let lib = import ./lib; pkgs = import ./. { }; in pkgs.fortune.override { recode = pkgs.recode.override { stdenv = pkgs.gcc6Stdenv; }; } «derivation /nix/store/a6lsym3s6jba9pd360hzkah6raymlrnq-fortune-mod-3.18.0.drv» nix-repl> :q $ nix-build --no-link /nix/store/a6lsym3s6jba9pd360hzkah6raymlrnq-fortune-mod-3.18.0.drv ... build process ... /nix/store/5il49lm4a0m0xx7kilybhsnr744fsl95-fortune-mod-3.18.0 ``` --- ## Hash-separated store path * ... and compare the resulting graphs * Notice the hash difference with the mous ecursor ```graphviz digraph FortuneRunDepsGraph { "b4vbia29wmzbvsxsq8g7ygk08l5zxvr0-fortune-mod-3.18.0" [label = "fortune-mod-3.18.0", shape = box, style = filled, fillcolor = "#ff0000"]; "84xlyz6zafcff8ddb3ggqzzckyzyvxiw-recode-3.7.12" -> "b4vbia29wmzbvsxsq8g7ygk08l5zxvr0-fortune-mod-3.18.0" [color = "black"]; "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" -> "b4vbia29wmzbvsxsq8g7ygk08l5zxvr0-fortune-mod-3.18.0" [color = "red"]; "84xlyz6zafcff8ddb3ggqzzckyzyvxiw-recode-3.7.12" [label = "recode-3.7.12", shape = box, style = filled, fillcolor = "#ff0000"]; "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" -> "84xlyz6zafcff8ddb3ggqzzckyzyvxiw-recode-3.7.12" [color = "green"]; "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" [label = "glibc-2.37-8", shape = box, style = filled, fillcolor = "#ff0000"]; "4563gldw8ibz76f1a3x69zq3a1vhdpz9-libidn2-2.3.4" -> "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" [color = "blue"]; "jd99cyc0251p0i5y69w8mqjcai8mcq7h-xgcc-12.2.0-libgcc" -> "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" [color = "magenta"]; "4563gldw8ibz76f1a3x69zq3a1vhdpz9-libidn2-2.3.4" [label = "libidn2-2.3.4", shape = box, style = filled, fillcolor = "#ff0000"]; "567zfi9026lp2q6v97vwn640rv6i3n4c-libunistring-1.1" -> "4563gldw8ibz76f1a3x69zq3a1vhdpz9-libidn2-2.3.4" [color = "burlywood"]; "567zfi9026lp2q6v97vwn640rv6i3n4c-libunistring-1.1" [label = "libunistring-1.1", shape = box, style = filled, fillcolor = "#ff0000"]; "jd99cyc0251p0i5y69w8mqjcai8mcq7h-xgcc-12.2.0-libgcc" [label = "xgcc-12.2.0-libgcc", shape = box, style = filled, fillcolor = "#ff0000"]; } ``` ```graphviz digraph FortuneDepsGraphOverriden { "5il49lm4a0m0xx7kilybhsnr744fsl95-fortune-mod-3.18.0" [label = "fortune-mod-3.18.0", shape = box, style = filled, fillcolor = "#ff0000"]; "1a75xmifw4jg4x6dzrmsdxhh7grvnvl9-recode-3.7.12" -> "5il49lm4a0m0xx7kilybhsnr744fsl95-fortune-mod-3.18.0" [color = "black"]; "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" -> "5il49lm4a0m0xx7kilybhsnr744fsl95-fortune-mod-3.18.0" [color = "red"]; "1a75xmifw4jg4x6dzrmsdxhh7grvnvl9-recode-3.7.12" [label = "recode-3.7.12", shape = box, style = filled, fillcolor = "#ff0000"]; "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" -> "1a75xmifw4jg4x6dzrmsdxhh7grvnvl9-recode-3.7.12" [color = "green"]; "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" [label = "glibc-2.37-8", shape = box, style = filled, fillcolor = "#ff0000"]; "4563gldw8ibz76f1a3x69zq3a1vhdpz9-libidn2-2.3.4" -> "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" [color = "blue"]; "jd99cyc0251p0i5y69w8mqjcai8mcq7h-xgcc-12.2.0-libgcc" -> "dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8" [color = "magenta"]; "4563gldw8ibz76f1a3x69zq3a1vhdpz9-libidn2-2.3.4" [label = "libidn2-2.3.4", shape = box, style = filled, fillcolor = "#ff0000"]; "567zfi9026lp2q6v97vwn640rv6i3n4c-libunistring-1.1" -> "4563gldw8ibz76f1a3x69zq3a1vhdpz9-libidn2-2.3.4" [color = "burlywood"]; "567zfi9026lp2q6v97vwn640rv6i3n4c-libunistring-1.1" [label = "libunistring-1.1", shape = box, style = filled, fillcolor = "#ff0000"]; "jd99cyc0251p0i5y69w8mqjcai8mcq7h-xgcc-12.2.0-libgcc" [label = "xgcc-12.2.0-libgcc", shape = box, style = filled, fillcolor = "#ff0000"]; } ``` --- ## Fetch from the Net ```plantuml @startuml state "Nix Language Level" as Language #LightBlue { state "User-defined derivation" as UserDrv UserDrv : fetchurl {\n url = "mirror://gnu/hello/hello-2.12.1.tar.gz";\n hash = "sha256-jZkUKv2...=";\n} state "Calling builtin function `derivation`" as CallingDerivation CallingDerivation : derivation {\n name = "hello-2.12.1.tar.gz";\n system="x86_64-linux";\n builder="/nix/store/7q1b1b...-bash-5.2-p15/bin/bash";\n args = [ ... ];\n outputHash = "sha256-jZkUKv2...=";\n outputHashMode="flat";\n # ...\n} state "Derivation as an attribute set" as DrvAsAttrs DrvAsAttrs : {\n type="derivation";\n drvPath = "/nix/store/7qqp4q...-hello-2.12.1.tar.gz.drv";\n outPath = "/nix/store/pa10z4...-hello-2.12.1.tar.gz";\n # ...\n} UserDrv -[#DarkBlue]-> CallingDerivation : evaluate CallingDerivation -[#DarkBlue]-> DrvAsAttrs : evaluate } state "Nix Store Level" as Store #BurlyWood { state "Store derivation" as StoreDrv StoreDrv : /nix/store/7qqp4q...-hello-2.12.1.tar.gz.drv\nDerive([("...","...")],[("...")],...) state "Output store path" as OutPath OutPath : /nix/store/pa10z4...-hello-2.12.1.tar.gz StoreDrv -[#DarkRed]-> OutPath : realize (as needed) } CallingDerivation -right[#DarkBlue]-> StoreDrv : instantiate state "Internet" as Internet Internet -[#DarkRed,dotted]-> OutPath @enduml ``` --- ## A typical package definition ```nix { lib, stdenv, fetchurl, cmake, recode, perl, rinutils , withOffensive ? false }: stdenv.mkDerivation (finalAttrs: { pname = "fortune-mod"; version = "3.18.0"; # We use fetchurl instead of fetchFromGitHub because # the release pack has some special files. src = fetchurl { url = with finalAttrs; "https://github.com/shlomif/fortune-mod/releases/download/${pname}-${version}/${pname}-${version}.tar.xz"; hash = "sha256-xaaB8aJgG3GG0fYS0vOnxC4RifQybxejS8ysqYE0xCs="; }; nativeBuildInputs = [ cmake perl rinutils ]; buildInputs = [ recode ]; cmakeFlags = [ "-DLOCALDIR=${placeholder "out"}/share/fortunes" ] ++ lib.optional (!withOffensive) "-DNO_OFFENSIVE=true"; patches = [ (builtins.toFile "not-a-game.patch" '' diff --git a/CMakeLists.txt b/CMakeLists.txt index 865e855..5a59370 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -154,7 +154,7 @@ ENDMACRO() my_exe( "fortune" "fortune/fortune.c" - "games" + "bin" ) my_exe( -- '') ]; postFixup = lib.optionalString (!withOffensive) '' rm -f $out/share/fortunes/men-women* ''; meta = with lib; { mainProgram = "fortune"; description = "A program that displays a pseudorandom message from a database of quotations"; license = licenses.bsdOriginal; platforms = platforms.unix; maintainers = with maintainers; [ vonfry ]; }; }) ``` --- ## Nixpkgs pinning ```nix let nixpkgs-pinned = builtins.fetchTarball { url = "https://github.com/NixOS/nixpkgs/archive/ecb441f22067ba1d6312f4932a7c64efa8d19a7b.tar.gz"; sha256 = "1r73q6h812hpzzy93bf6klmq30hvf7nh47v5lkjzqi1vp0imgsm2"; }; lib = import "${nixpkgs-pinned}/lib"; pkgs = import nixpkgs-pinned { }; in pkgs.hello ``` ```plantuml state "Nix Language Level" as Language #LightBlue { state "Calling builtin fetcher" as CallingBuiltinFetchers CallingBuiltinFetchers : builtins.fetchTarball {\n url = "https://github.com/NixOS/nixpkgs/archive/ecb441....tar.gz";\n sha256 = "1r73q6...";\n}\n\n state "Store path string" as StorePathStrings StorePathStrings : "/nix/store/yd0l70...-source" CallingBuiltinFetchers -[#DarkBlue]-> StorePathStrings : evaluate state "import from store path" as ImportStore : import /nix/store/yd0l70...-source state "evaluate from imported" as EvalImported : {...}: { ... } ImportStore -[#DarkBlue]-> EvalImported } state "Nix Store Level" as Store #BurlyWood { state "Store Paths" as StorePaths StorePaths : /nix/store/yd0l70...-source } CallingBuiltinFetchers -right[#DarkBlue]-> StorePaths : eval-time fetch state "Internet" as Internet Internet -[#DarkBlue,dotted]-> StorePaths StorePaths -up[#DarkBlue,dashed]-> ImportStore : import ``` --- ## Nix flakes -- automated pinning ```nix # flake.nix { inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05"; outputs = { self, nixpkgs, ... }@inputs: { packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello; } } ``` ```console $ nix flake lock path:. warning: creating lock file './flake.lock' $ nix run path:.#hello -- -g "Hello, Nix!" Hello, Nix! ``` ---- ## `flake.lock` ```json { "nodes": { "nixpkgs": { "locked": { "lastModified": 1690558459, "narHash": "sha256-5W7y1l2cLYPkpJGNlAja7XW2X2o9rjf0O1mo9nxS9jQ=", "owner": "NixOS", "repo": "nixpkgs", "rev": "48e82fe1b1c863ee26a33ce9bd39621d2ada0a33", "type": "github" }, "original": { "owner": "NixOS", "ref": "nixos-23.05", "repo": "nixpkgs", "type": "github" } }, "root": { "inputs": { "nixpkgs": "nixpkgs" } } }, "root": "root", "version": 7 } ``` --- ## Nix flakes -- automated pinning * Let's make it more cross-platform ```nix # flake.nix { inputs.flake-utils.url = "github:numtide/flake-utils"; inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05"; outputs = { self, nixpkgs, flake-utils, ... }@inputs: flake-utils.lib.eachDefaultSystem (system: let inherit (nixpkgs) lib; pkgs = nixpkgs.legacyPackages.${system}; in { packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello; devShells.default = pkgs.mkShell { packages = [ hello ]; # inputsFrom = [ # hello # ]; }; } ); } ``` --- ## Shell session with Nix * `nix develop` resembles the build environment * `nix shell` gets the specified packages ready * `nix run` and `nix shell -c` execute programs on the fly --- ## build a development enviornment * Use `mkShell` from Nixpkgs or `devshells.mkShell` by Numtide * Common attributes * `packages`: list of tools to make available to the environment * `inputsFrom`: list of packages to get their build environments * `shellHook`: extra RC when entering the shell * Works well with direnv (write `use flake` inside ``.envrc`) --- ## On-the-fly execution * Download the static Nix binary * Make it executable * `nix run flake-url#package` ```console $ curl -L "https://hydra.nixos.org/job/nix/master/buildStatic.x86_64-linux/latest/download-by-type/file/binary-dist" \ > nix $ chmod u+x ./nix $ ./nix \ --extra-experimental-features "nix-command flakes" \ run github:NixOS/nixpkgs/nixos-23.05#hello Hello, world! ```