Building reproducible Development environment

--- tags: Publication --- ![](https://i.imgur.com/VI2aul0.png) # Building reproducible Development environment One of the development team's biggest challenges is having a consistent and reproducible environment between team members. The bigger the team or more teams working on the same project codebase, the issue is amplified. Different languages try to manage dependencies using specifically inbuilt language-specific package manager. Ex. [Python](https://www.python.org/) it is using by de-facto `pip` as Python [package installer](https://pypi.org/project/pip/ ) that is using [Python Package Index repository](https://pypi.org/). All dependencies can be declared in the requirements.txt file and installed with `pip install -r requirements.txt` Pypi is limited in managing consistent Python packages and versioning. New Python package managers try to solve the limitation of the pip like Data Science oriented [Anaconda](https://www.anaconda.com) or [Poetry ](https://python-poetry.org/) that bring multiple improvements over pip with dependency management and packaging. Poetry is arguably Python's most advanced dependency management option today by using a configuration file in `toml` format and generating project scaffolding, locking dependency in the `*.lock` file. Ex. [Node.js](https://nodejs.org/en/) is another big contender in package management hell. [Npm](https://en.wikipedia.org/wiki/Npm_(software)) is a default Node.js package manager. Historically, `npm` lacked a dependency locking mechanism. New projects sprong out to help rectify the lack of functionality of npm like [Npmp](https://pnpm.io/) and [Yarn](https://classic.yarnpkg.com/en/). As I am more familiar with, Yarn was initially developed by Facebook and included lock file support and monorepo support. Npmjs community, in the past few years since the npm version 5 release, got feature parity with yarn, when it comes to lock file support. Many other languages try to cope with dependency management by introducing specialized tolls. ## Context In my career as development lead, I encountered the same pattern of problems again and again on maintaining stable, repeatable, and consistent codebase and dependency management. We can categorize the issue in two distinct buckets. The development team working on the same code base may encounter issues once the dependency version is changed by one of the developers that may break the app that may rely on a specific version only. You can mitigate this behavior primarily with test coverage or enforcing team practices on who can change the package version and how change is communicated to the rest of the group. The dependency tree and example will be if the primary package is changed and the new package is dependent on multiple other packages. As a consequence, the new dependencies version may break partially or fully application code. The only possible mitigation is to have a comprehensive test code coverage, which is not always possible. ## The Problem Let's define our problem. We need to have declarative, reliable, and reproducible management development environment managed in the source control system. Development environment codebase should be independent of the underlying system configuration or environment variable used. The development environment should be easy to maintain and configure without additional management overhead or limitations on development teams. The development environment configuration management tools should support monorepo practice and trunk-based development. The solution should be easily integrated with the CICD pipeline. ## Solution In time I tried multiple options and solutions to maintain a consistent and reproducible environment between development teams. Most of the solutions tried were language-specific, and I could not replicate them on different development languages. A few years back, I started using [NixOS](https://nixos.org/) as my primary OS for Laptop and Desktop. The main attraction was the declarative nature of the underlying configuration management language [Nix](https://nixos.org/manual/nix/stable/). One of Nix's tools is `nix-shell` that provided some ideas on how I could solve consistency and reproducibility issues for developers. [Nix-shell](https://nixos.org/manual/nix/stable/#sec-nix-shell) is a big part of Nix, and one of the downsides is it can't be used stand-alone. As we already decided to use NixOs as our development environment, it was natural to use all the advanced development toolset provided by Nix. The nix-shell official definition _"nix-shell - start an interactive shell based on a Nix expression"_ is not helpful. Probably a better definition would be "nix-shell command is used to provide an isolated repeatable environment, and it uses Nix expressions for that." If you are already running Nix on your Linux or using NixOS, `nix-shell` is installed as part of your Nix env. You can use Nix-Shell in three scenarios. ### nix-shell packages As an ad-hoc replacement for package manager (you can run an application without resorting to permanent installation) Ex. We need a Python interpreter to run a script. But Python is not installed in the system. We can use `nix-shell` to run the environment and deploy Python only for the duration of the `nix-shell` running environment. Once we exit from `nix-shell` Python will not be available anymore. ```sh ~> python python: command not found ~> nix-shell --packages python [nix-shell:~]$ python --version Python 2.7.18 [nix-shell:~]$ exit exit ~> python python: command not found ``` We can specify the version of the python we want to run. ```sh ~> nix-shell -p python39 these 4 paths will be fetched (11.03 MiB download, 53.47 MiB unpacked): /nix/store/0fgr90x0rsbl9j2rv3935mrm9mfy0zls-tzdata-2020f /nix/store/2kzvsbp8i7k3kq284cg9bv2zk6iadi2m-expat-2.2.10 /nix/store/ph6hpbx4pr31146wpb72yrk3f3yy0xcs-python3-3.9.4 /nix/store/rxcgs0xi9ngd0zsq47f4g06v2dqpqsr0-libffi-3.3 copying path '/nix/store/0fgr90x0rsbl9j2rv3935mrm9mfy0zls-tzdata-2020f' from 'https://cache.nixos.org'... copying path '/nix/store/2kzvsbp8i7k3kq284cg9bv2zk6iadi2m-expat-2.2.10' from 'https://cache.nixos.org'... copying path '/nix/store/rxcgs0xi9ngd0zsq47f4g06v2dqpqsr0-libffi-3.3' from 'https://cache.nixos.org'... copying path '/nix/store/ph6hpbx4pr31146wpb72yrk3f3yy0xcs-python3-3.9.4' from 'https://cache.nixos.org'... [nix-shell:~]$ python --version Python 3.9.4 [nix-shell:~]$ exit exit ``` We can as well directly run, in our case, Python shell by; ```sh ~> nix-shell -p python39 --run python Python 3.9.4 (default, Apr 4 2021, 18:23:51) [GCC 10.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> ``` We can run multiple packages, or in the below case, we run Python with the NumPy library already available for Python to consume. We need to specify only NumPy and Python interpreter will be deployed as a dependency. ```sh ~> nix-shell -p python3Packages.numpy --run python these 7 paths will be fetched (24.90 MiB download, 170.80 MiB unpacked): /nix/store/6fig8w8sir42gsli6zwzlp7njl2j6rbl-blas-3-dev /nix/store/761cpqh61b4zak4rl0y3sz5pa0954d3c-blas-3 /nix/store/c1lb5svhbkc7nsi8cpk6xbwgbiij38z9-openblas-0.3.13 /nix/store/fm5xm721z0jf65570g78v5xkaa6jy3kd-lapack-3-dev /nix/store/lr061z7swdw9gfgv8r2x2wzaz582m0xv-lapack-3 /nix/store/p0y5xw5yk57rwlik4yamq9xk7z5s45y6-gfortran-9.3.0-lib /nix/store/ys7ffnghl2iaza8ly4mjpycfyargxv4z-python3.8-numpy-1.20.2 copying path '/nix/store/p0y5xw5yk57rwlik4yamq9xk7z5s45y6-gfortran-9.3.0-lib' from 'https://cache.nixos.org'... copying path '/nix/store/c1lb5svhbkc7nsi8cpk6xbwgbiij38z9-openblas-0.3.13' from 'https://cache.nixos.org'... copying path '/nix/store/761cpqh61b4zak4rl0y3sz5pa0954d3c-blas-3' from 'https://cache.nixos.org'... copying path '/nix/store/lr061z7swdw9gfgv8r2x2wzaz582m0xv-lapack-3' from 'https://cache.nixos.org'... copying path '/nix/store/fm5xm721z0jf65570g78v5xkaa6jy3kd-lapack-3-dev' from 'https://cache.nixos.org'... copying path '/nix/store/6fig8w8sir42gsli6zwzlp7njl2j6rbl-blas-3-dev' from 'https://cache.nixos.org'... copying path '/nix/store/ys7ffnghl2iaza8ly4mjpycfyargxv4z-python3.8-numpy-1.20.2' from 'https://cache.nixos.org'... Python 3.8.9 (default, Apr 2 2021, 11:20:07) [GCC 10.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> ``` ### nix-shell shebangs Another use case is to embed `nix-shell` as a shell script that will help run on any Nix compatible OS even if the software specified in the script is not installed on the system. We can use `nix-shell shebangs` test.sh ```sh ~> python python: command not found ~> cat test.sh #! /usr/bin/env nix-shell #! nix-shell -i python -p python print "Hello world!" ~> chmod +x test.sh ~> ./test.sh Hello world! ``` ### shell.nix The most valuable and essential capability of the `nix-shell` is to have in your git repo code folder where you keep your sources `shell.nix` and where we can describe all dependencies and libraries needed for the project. Every time the developer is in the folder where the code resides and runs `nix-shell`, all project software dependencies will be deployed for the entire user session. Let's take the most basic example of the Node.js code base. ```sh ~> node --version node: command not found ~> yarn --version yarn: command not found ``` We need file shell.nix and populate with; ```sh ~> cat shell.nix let pkgs = import <nixpkgs> {}; in pkgs.mkShell rec { name = "nodeAapp"; buildInputs = with pkgs; [ nodejs yarn ]; } ``` Now we can run `nix-shell.` ```sh ~> nix-shell these 6 paths will be fetched (21.79 MiB download, 102.64 MiB unpacked): /nix/store/d50va99bmlng5926npmxvfmy7ly7qsjs-icu4c-69.1 /nix/store/hahkzr18y0x8vdxn4g2s2igxzga2qlar-openssl-1.1.1k-dev /nix/store/km96sj9xw1ww7jpz4055dfr92nmmq33i-libuv-1.41.0 /nix/store/m8csnvay8fj9479f4zdqnr29s9dbjdha-yarn-1.22.10 /nix/store/s42x7xflxz9br1axz9i7583m42j17piy-icu4c-69.1-dev /nix/store/vp3y4rm082app6f5ld4h9dq5z00bbapp-nodejs-14.17.2 copying path '/nix/store/d50va99bmlng5926npmxvfmy7ly7qsjs-icu4c-69.1' from 'https://cache.nixos.org'... copying path '/nix/store/hahkzr18y0x8vdxn4g2s2igxzga2qlar-openssl-1.1.1k-dev' from 'https://cache.nixos.org'... copying path '/nix/store/km96sj9xw1ww7jpz4055dfr92nmmq33i-libuv-1.41.0' from 'https://cache.nixos.org'... copying path '/nix/store/s42x7xflxz9br1axz9i7583m42j17piy-icu4c-69.1-dev' from 'https://cache.nixos.org'... copying path '/nix/store/vp3y4rm082app6f5ld4h9dq5z00bbapp-nodejs-14.17.2' from 'https://cache.nixos.org'... copying path '/nix/store/m8csnvay8fj9479f4zdqnr29s9dbjdha-yarn-1.22.10' from 'https://cache.nixos.org'... [nix-shell:~/src/test]$ node --version v14.17.2 [nix-shell:~/src/test]$ yarn --version 1.22.10 ``` We also can specify the version. shell.nix ```sh let pkgs = import <nixpkgs> {}; in pkgs.mkShell rec { name = "webdev"; buildInputs = with pkgs; [ nodejs-12_x yarn ]; } ``` _Note: advantage is by developing in monorepo team may have microservices running with different Node.js versions, and this will allow the team members to support multiple version for the project_ We can use the `shellHook` function to extend `nix-shell` functionality. shell.nix ```sh with (import <nixpkgs> {}); mkShell { buildInputs = [ nodejs yarn ]; shellHook = '' mkdir -p .nix-node export NODE_PATH=$PWD/.nix-node export NPM_CONFIG_PREFIX=$PWD/.nix-node export PATH=$NODE_PATH/bin:$PATH ''; } ``` Let's take a look at the Python example with Pandas and NumPy libraries as dependencies. shell.nix ```sh with (import <nixpkgs> {}); mkShell { buildInputs = with python39Packages; [ pip pandas numpy ]; shellHook = '' alias pip="PIP_PREFIX='$HOME/.pip_packages' \pip" ''; } ``` ```sh ~> nix-shell these 23 paths will be fetched (37.14 MiB download, 197.66 MiB unpacked): /nix/store/06mpf704f7mm0nxvvwciv7vzda1g739g-python3.9-pbr-5.5.1 /nix/store/3fzlg2lvd0vmqyy2va96796w95rqk8jn-python3.9-tables-3.6.1 /nix/store/7hk3s95mllbj2bjy680s5pbq2f657chg-python3.9-et_xmlfile-1.0.1 /nix/store/7iw8n01m46m6n4gxr46y6cnys1r3cp4z-python3.9-xlwt-1.3.0 /nix/store/8fwy5f74b20xh7zyyz875ri64z1nd86d-python3.9-numpy-1.20.2 /nix/store/8lvgm16bifq6a17vhn5fgacrwf06lvwn-python3.9-pip-21.0.1 /nix/store/8nqr6x0w568p9kxr5rz3mghb63f924j2-python3.9-Bottleneck-1.3.2 /nix/store/aiw78m14p2r7xmxdq9z7ij6am3air1nm-python3.9-python-dateutil-2.8.1 /nix/store/dz9f2hjbs16gq2h8v7zs7z23fyh11xcq-python3.9-SQLAlchemy-1.3.23 /nix/store/j4gfk78r1xalkpd1j3195x0v202vqbwi-python3.9-pytz-2021.1 /nix/store/jx9669w7109bkmjj7x249yiflzyl8c87-python3.9-html5lib-1.1 /nix/store/kpqmf12s5j0yzncp0635c07wh2y5lfkf-python3.9-beautifulsoup4-4.9.3 /nix/store/kxrspjijzc81axl0sb968z1s7ix0glvb-python3.9-soupsieve-2.2.1 /nix/store/lpjpm6y207id13d3x7y994mxdvvb2c9b-python3.9-numexpr-2.7.3 /nix/store/mxd4dd79jmldaysvkkl0hrxlk91amlci-python3.9-webencodings-0.5.1 /nix/store/mzbagdp4b6kifr2narz1zp9r5lklnzin-python3.9-openpyxl-3.0.7 /nix/store/n9rxhgjsxlqy9w82v4abhwjwiv4kcp48-python3.9-scipy-1.6.1 /nix/store/nnp5x5vf8algzq5ikwyv7j4dxqc4zrd4-python3.9-mock-4.0.3 /nix/store/qlrpcacwyfdq2sgd6z9wpmm6vzax65l6-python3.9-xlrd-2.0.1 /nix/store/qrfg4g4l8skb6yj1b1gws1xrvjzkf99p-python3.9-jdcal-1.4.1 /nix/store/v7a9l919llggn700sn9mzdvma731akfg-python3.9-lxml-4.6.3 /nix/store/wfvd4l92z04chhd3rdq959npvn3bs9kg-python3.9-six-1.15.0 /nix/store/z3j94srpg9pxljb301mhbdlg0nibddh7-python3.9-pandas-1.2.3 copying path '/nix/store/8fwy5f74b20xh7zyyz875ri64z1nd86d-python3.9-numpy-1.20.2' from 'https://cache.nixos.org'... copying path '/nix/store/8lvgm16bifq6a17vhn5fgacrwf06lvwn-python3.9-pip-21.0.1' from 'https://cache.nixos.org'... copying path '/nix/store/v7a9l919llggn700sn9mzdvma731akfg-python3.9-lxml-4.6.3' from 'https://cache.nixos.org'... copying path '/nix/store/dz9f2hjbs16gq2h8v7zs7z23fyh11xcq-python3.9-SQLAlchemy-1.3.23' from 'https://cache.nixos.org'... copying path '/nix/store/7hk3s95mllbj2bjy680s5pbq2f657chg-python3.9-et_xmlfile-1.0.1' from 'https://cache.nixos.org'... copying path '/nix/store/qrfg4g4l8skb6yj1b1gws1xrvjzkf99p-python3.9-jdcal-1.4.1' from 'https://cache.nixos.org'... copying path '/nix/store/06mpf704f7mm0nxvvwciv7vzda1g739g-python3.9-pbr-5.5.1' from 'https://cache.nixos.org'... copying path '/nix/store/j4gfk78r1xalkpd1j3195x0v202vqbwi-python3.9-pytz-2021.1' from 'https://cache.nixos.org'... copying path '/nix/store/wfvd4l92z04chhd3rdq959npvn3bs9kg-python3.9-six-1.15.0' from 'https://cache.nixos.org'... copying path '/nix/store/mzbagdp4b6kifr2narz1zp9r5lklnzin-python3.9-openpyxl-3.0.7' from 'https://cache.nixos.org'... copying path '/nix/store/kxrspjijzc81axl0sb968z1s7ix0glvb-python3.9-soupsieve-2.2.1' from 'https://cache.nixos.org'... copying path '/nix/store/8nqr6x0w568p9kxr5rz3mghb63f924j2-python3.9-Bottleneck-1.3.2' from 'https://cache.nixos.org'... copying path '/nix/store/nnp5x5vf8algzq5ikwyv7j4dxqc4zrd4-python3.9-mock-4.0.3' from 'https://cache.nixos.org'... copying path '/nix/store/lpjpm6y207id13d3x7y994mxdvvb2c9b-python3.9-numexpr-2.7.3' from 'https://cache.nixos.org'... copying path '/nix/store/aiw78m14p2r7xmxdq9z7ij6am3air1nm-python3.9-python-dateutil-2.8.1' from 'https://cache.nixos.org'... copying path '/nix/store/kpqmf12s5j0yzncp0635c07wh2y5lfkf-python3.9-beautifulsoup4-4.9.3' from 'https://cache.nixos.org'... copying path '/nix/store/n9rxhgjsxlqy9w82v4abhwjwiv4kcp48-python3.9-scipy-1.6.1' from 'https://cache.nixos.org'... copying path '/nix/store/mxd4dd79jmldaysvkkl0hrxlk91amlci-python3.9-webencodings-0.5.1' from 'https://cache.nixos.org'... copying path '/nix/store/qlrpcacwyfdq2sgd6z9wpmm6vzax65l6-python3.9-xlrd-2.0.1' from 'https://cache.nixos.org'... copying path '/nix/store/3fzlg2lvd0vmqyy2va96796w95rqk8jn-python3.9-tables-3.6.1' from 'https://cache.nixos.org'... copying path '/nix/store/7iw8n01m46m6n4gxr46y6cnys1r3cp4z-python3.9-xlwt-1.3.0' from 'https://cache.nixos.org'... copying path '/nix/store/jx9669w7109bkmjj7x249yiflzyl8c87-python3.9-html5lib-1.1' from 'https://cache.nixos.org'... copying path '/nix/store/z3j94srpg9pxljb301mhbdlg0nibddh7-python3.9-pandas-1.2.3' from 'https://cache.nixos.org'... [nix-shell:~/src/test]$ python Python 3.9.4 (default, Apr 4 2021, 18:23:51) [GCC 10.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pandas as pd >>> import numpy as np >>> ``` Python's case can get tricky as Python provided virtualenv that can be managed by PyPI or poetry. nix-shell as well creates an isolated environment that can easily replace virtualenv. The dilemma is mostly around how familiar the team is with nix expressions. If you are starting your journey into Nix you can find a comprehensive guide in [NixWiki Python page](https://nixos.wiki/wiki/Python) and use [Mach-nix](https://github.com/DavHau/mach-nix) that makes it easy to create and share reproducible python environments or packages. Another option is to use [poetry2nix](https://github.com/nix-community/poetry2nix) to turn [Poetry](https://poetry.eustace.io/) projects into Nix derivations without writing Nix expressions. Nix-shell supports the latest nix flake option that will remove dependency on nix-channels and allow a better consistency between nix environments with direct integration with git. Example from The blog post [Practical Nix Flakes](https://serokell.io/blog/practical-nix-flakes) ```sh ~> nix shell https://github.com/nixos/nixpkgs/archive/nixpkgs-unstable.tar.gz#hello > ~nix shell 'git+https://github.com/nixos/nixpkgs?ref=nixpkgs-unstable#hello' ``` Nix Wiki provides more in depth examples on [NixWiki Flake Page](https://nixos.wiki/wiki/Flakes#Super_fast_nix-shell) Our team implemented a native `nix shell` with flake integration and poetry to manage python package dependencies. flake.nix ```sh { description = "A flake using poetry2nix"; nixConfig.bash-prompt = "\[nix-develop\]$ "; inputs.nixpkgs.url = "github:NixOS/nixpkgs"; inputs.utils.url = "github:numtide/flake-utils"; inputs.poetry2nix-src.url = "github:nix-community/poetry2nix"; outputs = {nixpkgs, utils, poetry2nix-src, self}: utils.lib.eachDefaultSystem (system: let pkgs = import nixpkgs { inherit system; overlays = [ poetry2nix-src.overlay ]; }; matrixInfraEnv = pkgs.poetry2nix.mkPoetryEnv { projectDir = ./.; python = pkgs.python39; editablePackageSources = { matrixInfra = ./.; }; }; in { devShell = pkgs.mkShell { buildInputs = [ matrixInfraEnv pkgs.python39Packages.pip ]; }; }); } ``` To make it easy for the developers, we start using [direnv](https://direnv.net/) that natively integrates with Nix and Flakes. Every time the developer enters the folder, all dependencies and environment variables will be deployed without running the `nix shell` command. You will need to create a file .envrc where you store your `flake.nix`, or `shell.nix`. ```sh # for shell.vin ~> cat .envrc usen nix # for flake.nix ~> cat .envrc use flake ``` To be aware, learning Nix will be a very steep curve. In my opinion, the effort is worth it as it solves so many problems the development community faces daily working in large teams. ## Conclusions Nix-shell supports almost all programming languages under the sky. you can find an extensive repository with examples in [Git repository](https://github.com/BenSchZA/nix-shells) Check as well an excellent blog post by [Mattia Gheda's An introduction to nix-shell](https://ghedam.at/15978/an-introduction-to-nix-shell) and his [A nix-shell for developing Elixir](https://ghedam.at/15443/a-nix-shell-for-developing-elixir) We scratch the surface on potential use cases for `nix-shell`, but this can help with information and use cases to try and make the 1st step into the Nix ecosystem.

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.