or
or
By clicking below, you agree to our terms of service.
Sign in with Wallet
New to HackMD? Sign up
| Syntax | Example | Reference | |
|---|---|---|---|
| # Header | Header | 基本排版 | |
| - Unordered List |
|
||
| 1. Ordered List |
|
||
| - [ ] Todo List |
|
||
| > Blockquote | Blockquote |
||
| **Bold font** | Bold font | ||
| *Italics font* | Italics font | ||
| ~~Strikethrough~~ | |||
| 19^th^ | 19th | ||
| H~2~O | H2O | ||
| ++Inserted text++ | Inserted text | ||
| ==Marked text== | Marked text | ||
| [link text](https:// "title") | Link | ||
|  | Image | ||
| `Code` | Code |
在筆記中貼入程式碼 | |
| ```javascript var i = 0; ``` |
|
||
| :smile: |  |
Emoji list | |
| {%youtube youtube_id %} | Externals | ||
| $L^aT_eX$ | LaTeX | ||
| :::info This is a alert area. ::: |
This is a alert area. |
On a scale of 0-10, how likely is it that you would recommend HackMD to your friends, family or business associates?
Please give us some advice and help us improve HackMD.
Syncing
xxxxxxxxxx-
Any changes
Be notified of any changes
-
Mention me
Be notified of mention me
-
Unsubscribe
SubscribeBuilding reproducible Development environment
One of the development team's biggest challenges is having a consistent and reproducible environment between team members.
The bigger the team or more teams working on the same project codebase, the issue is amplified.
Different languages try to manage dependencies using specifically inbuilt language-specific package manager.
Ex. Python it is using by de-facto
pipas Python package installer that is using Python Package Index repository. All dependencies can be declared in the requirements.txt file and installed withpip install -r requirements.txtPypi is limited in managing consistent Python packages and versioning. New Python package managers try to solve the limitation of the pip like Data Science oriented Anaconda or Poetry that bring multiple improvements over pip with dependency management and packaging. Poetry is arguably Python's most advanced dependency management option today by using a configuration file intomlformat and generating project scaffolding, locking dependency in the*.lockfile.Ex. Node.js is another big contender in package management hell. Npm is a default Node.js package manager. Historically,
npmlacked a dependency locking mechanism. New projects sprong out to help rectify the lack of functionality of npm like Npmp and Yarn. As I am more familiar with, Yarn was initially developed by Facebook and included lock file support and monorepo support. Npmjs community, in the past few years since the npm version 5 release, got feature parity with yarn, when it comes to lock file support.Many other languages try to cope with dependency management by introducing specialized tolls.
Context
In my career as development lead, I encountered the same pattern of problems again and again on maintaining stable, repeatable, and consistent codebase and dependency management.
We can categorize the issue in two distinct buckets.
The development team working on the same code base may encounter issues once the dependency version is changed by one of the developers that may break the app that may rely on a specific version only.
You can mitigate this behavior primarily with test coverage or enforcing team practices on who can change the package version and how change is communicated to the rest of the group.
The dependency tree and example will be if the primary package is changed and the new package is dependent on multiple other packages. As a consequence, the new dependencies version may break partially or fully application code.
The only possible mitigation is to have a comprehensive test code coverage, which is not always possible.
The Problem
Let's define our problem.
We need to have declarative, reliable, and reproducible management development environment managed in the source control system.
Development environment codebase should be independent of the underlying system configuration or environment variable used.
The development environment should be easy to maintain and configure without additional management overhead or limitations on development teams.
The development environment configuration management tools should support monorepo practice and trunk-based development.
The solution should be easily integrated with the CICD pipeline.
Solution
In time I tried multiple options and solutions to maintain a consistent and reproducible environment between development teams. Most of the solutions tried were language-specific, and I could not replicate them on different development languages.
A few years back, I started using NixOS as my primary OS for Laptop and Desktop. The main attraction was the declarative nature of the underlying configuration management language Nix. One of Nix's tools is
nix-shellthat provided some ideas on how I could solve consistency and reproducibility issues for developers.Nix-shell is a big part of Nix, and one of the downsides is it can't be used stand-alone. As we already decided to use NixOs as our development environment, it was natural to use all the advanced development toolset provided by Nix.
The nix-shell official definition "nix-shell - start an interactive shell based on a Nix expression" is not helpful. Probably a better definition would be "nix-shell command is used to provide an isolated repeatable environment, and it uses Nix expressions for that."
If you are already running Nix on your Linux or using NixOS,
nix-shellis installed as part of your Nix env.You can use Nix-Shell in three scenarios.
nix-shell packages
As an ad-hoc replacement for package manager (you can run an application without resorting to permanent installation)
Ex. We need a Python interpreter to run a script. But Python is not installed in the system. We can use
nix-shellto run the environment and deploy Python only for the duration of thenix-shellrunning environment. Once we exit fromnix-shellPython will not be available anymore.We can specify the version of the python we want to run.
We can as well directly run, in our case, Python shell by;
We can run multiple packages, or in the below case, we run Python with the NumPy library already available for Python to consume. We need to specify only NumPy and Python interpreter will be deployed as a dependency.
nix-shell shebangs
Another use case is to embed
nix-shellas a shell script that will help run on any Nix compatible OS even if the software specified in the script is not installed on the system.We can use
nix-shell shebangstest.sh
shell.nix
The most valuable and essential capability of the
nix-shellis to have in your git repo code folder where you keep your sourcesshell.nixand where we can describe all dependencies and libraries needed for the project. Every time the developer is in the folder where the code resides and runsnix-shell, all project software dependencies will be deployed for the entire user session.Let's take the most basic example of the Node.js code base.
We need file shell.nix and populate with;
Now we can run
nix-shell.We also can specify the version.
shell.nix
Note: advantage is by developing in monorepo team may have microservices running with different Node.js versions, and this will allow the team members to support multiple version for the project
We can use the
shellHookfunction to extendnix-shellfunctionality.shell.nix
Let's take a look at the Python example with Pandas and NumPy libraries as dependencies.
shell.nix
Python's case can get tricky as Python provided virtualenv that can be managed by PyPI or poetry. nix-shell as well creates an isolated environment that can easily replace virtualenv.
The dilemma is mostly around how familiar the team is with nix expressions.
If you are starting your journey into Nix you can find a comprehensive guide in NixWiki Python page and use Mach-nix that makes it easy to create and share reproducible python environments or packages. Another option is to use poetry2nix to turn Poetry projects into Nix derivations without writing Nix expressions.
Nix-shell supports the latest nix flake option that will remove dependency on nix-channels and allow a better consistency between nix environments with direct integration with git.
Example from The blog post Practical Nix Flakes
Nix Wiki provides more in depth examples on NixWiki Flake Page
Our team implemented a native
nix shellwith flake integration and poetry to manage python package dependencies.flake.nix
To make it easy for the developers, we start using direnv that natively integrates with Nix and Flakes.
Every time the developer enters the folder, all dependencies and environment variables will be deployed without running the
nix shellcommand.You will need to create a file .envrc where you store your
flake.nix, orshell.nix.To be aware, learning Nix will be a very steep curve. In my opinion, the effort is worth it as it solves so many problems the development community faces daily working in large teams.
Conclusions
Nix-shell supports almost all programming languages under the sky.
you can find an extensive repository with examples in Git repository
Check as well an excellent blog post by Mattia Gheda's An introduction to nix-shell and his A nix-shell for developing Elixir
We scratch the surface on potential use cases for
nix-shell, but this can help with information and use cases to try and make the 1st step into the Nix ecosystem.