Fixing Dockerfile image build consistency

--- tags: Publication --- ![](https://i.imgur.com/8N2BupS.png) # Fixing Dockerfile image build consistency When Docker emerged in 2013 wasn't a new technology in containerization space. The container history started 1979 with Unix V7 Chroot and continued, 2000 FreeBSD Jails, 2001 Linux VServer, 2004 Solaris Containers, 2005 Open VZ, 2006 Process Containers, 2008 LXC, etc. dotCloud now Docker popularized containers by adding simplicity in packaging, managing, and layered dependency based on Linux Kernel technologies like cgroups, namespaces, and Linux Kernel Capabilities (cap_*) ## Context Docker had a significant impact on development, application packaging, and deployment. The most significant contribution, I believe, was in encapsulating all dependencies in deployable units that simplified the job of the DevOps/SRE teams significantly. No technology is perfect neither is Docker containers. Docker doesn't solve all problems in development and deployment. This article will cover a few potential options that can significantly reduce dependency, consistency, and management of the deployable units. ## The Problem Dockerfile is the leading contender for the problem to be solved. ### Dockerfile maintenance Due to the imperative nature of the Dockerfile, it is effortless to start, but once you start using it extensively, it creates more problems than it solves. ### Base image We always try to minimize the size of the base image, and we have multiple options to select from, starting with [Alpine Linux](https://www.alpinelinux.org/) that is base on [musl-libc](https://www.musl-libc.org/) and [BusyBox](https://www.busybox.net/) or [Ubuntu](https://hub.docker.com/_/ubuntu) and [Debian](https://hub.docker.com/_/debian/), [Fedora](https://hub.docker.com/_/fedora), etc. based distribution optimized for containers. All these Base Images bring package management dependency that adds to the storage. We can select minimal Base Images like `FROM scratch` or [Google distroless](https://github.com/GoogleContainerTools/distroless) best works with statically linked binaries or GO base binaries. Avoid as well building golden images as it is difficult to inspect and manage security. ## Manual layer management We all heard that you should minimize the docker image layers if you want to build an optimized image. But the tradeoff is by squishing the nr. of layers, and we remove the possibility of reusing the cash. Every line in Dockerfile like `RUN, CMD, COPY ADD, etc.` adds a new layer. You can combine the layers; From ```sh FROM alpine RUN apk update && apk upgrade RUN apk add nodejs ``` To ```sh FROM alpine RUN apk update && apk upgrade && apk add nodejs ``` But you will lose the possibility of reusing the cash. Even if you opt for a multilayer approach, you will not be able to reuse intermediate layers. Docker images are composable that stack on each other, and not an easy way to manage cached layers. _Note: You may consider utilizing the Multi-Stage build._ ### Image Test and inspection One of the problems with Dockerfile you can not test dependency efficiently and code before building the image. Details inspection of the components is also tough to achieve. You can test and inspect the image only once it is built. ## Solution We initially decided to use [Nix as a functional expression language](https://nixos.wiki/wiki/Nix_Expression_Language) and [NixOS](https://nixos.org/) as our development environment on GCP. The main reason for the decision was Nix excellent support for multiple programming languages and consistently reproducible and highly reliable dependencies management. Nix provides a reliable function `buildImage` and `buildLayeredImage` to build Docker images, and this offers an excellent opportunity to keep tooling consistent in the development environment. The below examples base on an excellent Video presentation by [Rok Garbas](https://twitter.com/garbas) from Tweag [Nix + Docker, a match made in heaven](https://www.youtube.com/watch?v=WP_oAmV6C2U) We will try to build [Redis](https://redis.io/) Docker container image. ### Single-layer Docker Images To build a single-layer Redis Docker Image, we will use the inbuilt nix function `buildImage` redis.nix ```nix { pkgs ? import <nixpkgs>{} }: let inherit (pkgs.dockerTools) buildImage; in buildImage { name = "nix-redis"; tag = "latest"; contents = with pkgs; [ redis ]; config = { Cmd = [ "/bin/redis-server" ]; WorkingDir = "/data"; Volumes = { "/data" = {}; }; }; created = "now"; } ``` As in above, you can see similarities with Dockerfile. Writing nix expression is straightforward and maps directly with Docker image build expressions. Nest step, we will build nix expression with ```sh ~> nix-build redis.nix /nix/store/j2i6lk1fgjb8f3ds5k508zjr7ymw3m88-docker-image-nix-redis.tar.gz ``` The above command will create a result symlink to *tar.gz file that contains all information needed to import into the Docker as an image. ```sh ~> ls -l result lrwxrwxrwx 73 mudrii 23 Sep 19:05 result -> /nix/store/j2i6lk1fgjb8f3ds5k508zjr7ymw3m88-docker-image-nix-redis.tar.gz ``` Now we can import *.tar.gz file into Docker. ```sh ~> docker load < result Loaded image: nix-redis:latest ~> docker images REPOSITORY TAG IMAGE ID CREATED SIZE nix-redis latest 13ea7dccae63 24 hours ago 156MB ``` We can run the container from nix-redis. ```sh ~> docker run --rm -ti nix-redis 1:C 23 Sep 2021 11:11:05.339 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 1:C 23 Sep 2021 11:11:05.339 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=1, just started 1:C 23 Sep 2021 11:11:05.339 # Warning: no config file specified, using the default config. In order to specify a config file use /bin/redis-server /path/to/redis.conf 1:M 23 Sep 2021 11:11:05.340 * monotonic clock: POSIX clock_gettime _._ _.-``__ ''-._ _.-`` `. `_. ''-._ Redis 6.2.5 (00000000/0) 64 bit .-`` .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in standalone mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379 | `-._ `._ / _.-' | PID: 1 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | https://redis.io `-._ `-._`-.__.-'_.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | `-._ `-._`-.__.-'_.-' _.-' `-._ `-.__.-' _.-' `-._ _.-' `-.__.-' 1:M 23 Sep 2021 11:11:05.340 # Server initialized 1:M 23 Sep 2021 11:11:05.340 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 1:M 23 Sep 2021 11:11:05.341 * Ready to accept connections ``` _Note: If you do not specify in redis.nix options for `tag = "latest";` Docker will generate and attach SHA256 build tag to the imported image._ _Note: If you omit from the file `created = "now";` Nix will use Docker Image created time as UNIX epoch (00:00:00, January 1st, 1970 in UTC) for the purpose of reproducibility._ ```sh ~> docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE nix-redis 798k6n4arzcdl82kz26v9xg5pk01554d f1a8b6a12f08 51 years ago 156MB ``` ### Multi-layer Docker Images For building a multilayered Redis Docker Image, we will use the inbuilt nix function `buildLayeredImage` Building a multilayered option is preferred. By using `buildLayeredImage` will allow us to cache every layer separately. We have granular control over layers in the image, and every single layer can be reused from the cache. redis-multi.nix ```nix { pkgs ? import <nixpkgs>{} , name ? "nix-redis-layered" , redis ? pkgs.redis }: let inherit (pkgs.dockerTools) buildLayeredImage; in buildLayeredImage { inherit name; tag = "latest"; contents = with pkgs; [ redis ]; config = { Cmd = [ "/bin/redis-server" ]; WorkingDir = "/data"; Volumes = { "/data" = {}; }; }; created = "now"; maxLayers = 100; } ``` As you can notice are minimal differences from a single layer. The only addon is to limit nr. of layers per Docker Image. _Note: Base on [Docker documentation](https://docs.docker.com/storage/storagedriver/overlayfs-driver/), allowed a maximum of 128 layers limits for overlay2 FS. From what I understood, this is due to a restriction of Linux Kernel not accepting more arguments length to a syscall, but is configurable._ Check your Docker Storage Driver by ```sh ~> docker info | grep "Storage Driver:" Storage Driver: overlay2 ``` Let's build out a multilayered container. ```sh ~> nix-build redis-multi.nix /nix/store/c37yd8b30sh5y9yr1900zlhrxmdimvfc-nix-redis-layered.tar.gz ``` Like in the 1st exercise, we should have a result symlink pointing to the nix store where the tar.gz archived image resides. We can have a pick inside the archive before importing an image into the Docker. ```sh ~> ls -la result .rw-r--r-- 431 mudrii users 23 Sep 19:58  redis-multi.nix .rw-r--r-- 320 mudrii users 23 Sep 19:58  redis.nix lrwxrwxrwx 68 mudrii users 23 Sep 19:58  result -> /nix/store/c37yd8b30sh5y9yr1900zlhrxmdimvfc-nix-redis-layered.tar.gz ~> tar --list --file result a412bdc048f52d573ad646355e118e7c237b59ea5cc7390f45a9ae932eaefc23/layer.tar c1c544388655af4ae8177c16f0c1b14c76a8c231094a29942666be76773bffc2/layer.tar c30d7ab5125d37157dc5826d9c2ae09890e22a5c8e298112e7461992397e3130/layer.tar 720d79ac643ac0faa43cbe0e5835233e67352e1f1bd17df124bd5b7d827cc73e/layer.tar ... .. . e9768307ff114da4db594040f966b4d9ff7b12373ccef7a4b40d4c1dc5b4bbd2/layer.tar 7b5c7d75aa7aea149ca73fe1d4aa33b65569395ccd2e7e69c3e82e4eb98389a6/layer.tar ad8e5c65f5bf76e14b1a1e0a2a1f3b5fdb217b1ef2fdc6aaa41c4fc85236174e/layer.tar 0c2b166ddb88e06bae5382b5dba592a7ffded79f5f502141fc46a76230f86816/layer.tar 9b2bc95625a01f8de32f62dc0eb4357bd5782909d714fbdb03a9a87caa7e9c71/layer.tar 492a73edeac89ee536058f505aef22a0b0a42a11af73e8988b5304d02fa5c619.json manifest.json ~> tar --list --file result | wc -l 75 ``` The newly created multilayered image gives us 74 layers. Now we are ready to import the archive into Docker as an image. ```sh ~> docker load < result a412bdc048f5: Loading layer [==================================================>] 1.649MB/1.649MB c1c544388655: Loading layer [==================================================>] 276.5kB/276.5kB c30d7ab5125d: Loading layer [==================================================>] 31.6MB/31.6MB 720d79ac643a: Loading layer [==================================================>] 1.362MB/1.362MB 461d9b12152c: Loading layer [==================================================>] 6.031MB/6.031MB c871541b5aae: Loading layer [==================================================>] 112.6kB/112.6kB fdb9fbdf000e: Loading layer [==================================================>] 3.543MB/3.543MB b1cae4033a7f: Loading layer [==================================================>] 133.1kB/133.1kB d3333d590067: Loading layer [==================================================>] 2.038MB/2.038MB 3c49a3a82d73: Loading layer [==================================================>] 133.1kB/133.1kB eadc6d642670: Loading layer [==================================================>] 512kB/512kB 0905d3dfdcae: Loading layer [==================================================>] 4.915MB/4.915MB a4c0f2c42da2: Loading layer [==================================================>] 532.5kB/532.5kB b9462ca7218c: Loading layer [==================================================>] 92.16kB/92.16kB ... .. . 8cb49edd4f95: Loading layer [==================================================>] 1.106MB/1.106MB 6d53ab07cf25: Loading layer [==================================================>] 2.97MB/2.97MB 7fc9f06b9563: Loading layer [==================================================>] 2.519MB/2.519MB 52830d6f69e1: Loading layer [==================================================>] 491.5kB/491.5kB 62f481182e43: Loading layer [==================================================>] 174.1kB/174.1kB e9768307ff11: Loading layer [==================================================>] 225.3kB/225.3kB 7b5c7d75aa7a: Loading layer [==================================================>] 1.812MB/1.812MB ad8e5c65f5bf: Loading layer [==================================================>] 28.44MB/28.44MB 0c2b166ddb88: Loading layer [==================================================>] 4.26MB/4.26MB 9b2bc95625a0: Loading layer [==================================================>] 10.24kB/10.24kB Loaded image: nix-redis-layered:latest ``` Review imported layers ```sh ~> docker history nix-redis-layered:latest IMAGE CREATED CREATED BY SIZE COMMENT 492a73edeac8 25 hours ago 252B store paths: ['/nix/store/vbsalcy8rvgcvlgdc4zfip4hvrbivf4y-nix-redis-layered-customisation-layer'] <missing> 25 hours ago 4.25MB store paths: ['/nix/store/a81n9nqr8imc36rf34kk9qjps7i62c90-redis-6.2.5'] <missing> 25 hours ago 27.7MB store paths: ['/nix/store/n5j5fjn60nhck658j9ab84k8n9z24n1r-systemd-247.6'] <missing> 25 hours ago 1.8MB store paths: ['/nix/store/0di899y5p0j8qx07pfvd3wb0iblkhr1b-pcre2-10.36'] <missing> 25 hours ago 215kB store paths: ['/nix/store/0b62nmnadx1xw2wz6hib3p6gzlylbdjc-lz4-1.9.3'] <missing> 25 hours ago 167kB store paths: ['/nix/store/rzmy26qy8afia865dbn06g6hfrrqiwvg-libmicrohttpd-0.9.71'] <missing> 25 hours ago 447kB store paths: ['/nix/store/002d6xvw29my0a8ayj8wb77wzvyy7vgv-libapparmor-3.0.1'] <missing> 25 hours ago 2.38MB store paths: ['/nix/store/d7fx1xwjdsvrypivnmp0pd2s2invphz8-iptables-1.8.7'] <missing> 25 hours ago 2.88MB store paths: ['/nix/store/n2dprmax869lb76gwpc2ygaqwbdgq7r0-gnutar-1.34'] <missing> 25 hours ago 1.09MB store paths: ['/nix/store/wi87l1pm7dfy9gn5kqh2ddzid7pn1x5s-gnupg-2.2.27'] <missing> 25 hours ago 744kB store paths: ['/nix/store/hh5ai1v2hhizkv5n4fr4hg72l1a5z1iw-curl-7.76.1'] <missing> 25 hours ago 2.54MB store paths: ['/nix/store/jrkrc4v0anvgi6axszy6lwqqcnlzjkq2-cryptsetup-2.3.5'] <missing> 25 hours ago 153kB store paths: ['/nix/store/1mai0bv7ch11nljavsyckcxlpixy6cg2-popt-1.18'] <missing> 25 hours ago 213kB store paths: ['/nix/store/n9va4cqy8r026jcjkxbnzhqbd33kl3nm-nghttp2-1.43.0-lib'] <missing> 25 hours ago 412kB store paths: ['/nix/store/1a9gbnqccqxjg86n5q8556ggi7f43psz-lvm2-2.03.12-lib'] <missing> 25 hours ago 293kB store paths: ['/nix/store/5r0a92rsppim1dljzr0rhpxwdsmnab9l-libssh2-1.9.0'] <missing> 25 hours ago 1.09MB store paths: ['/nix/store/5i7z7iidkhiwdb760p18b8px8g686rha-libpcap-1.10.0'] ... .. . <missing> 25 hours ago 5.98MB store paths: ['/nix/store/50msfhkz5wbyk8i78pjv3y9lxdrp7dlm-gcc-10.3.0-lib'] <missing> 25 hours ago 1.33MB store paths: ['/nix/store/wv35g5lff84rray15zlzarcqi9fxzz84-bash-4.4-p23'] <missing> 25 hours ago 30.9MB store paths: ['/nix/store/jsp3h3wpzc842j0rz61m5ly71ak6qgdn-glibc-2.32-54'] <missing> 25 hours ago 227kB store paths: ['/nix/store/ckb0qa2yrxrpp0piffgjq9id38gc5z9v-libidn2-2.3.1'] <missing> 25 hours ago 1.63MB store paths: ['/nix/store/5d821pjgzb90lw4zbg6xwxs7llm335wr-libunistring-0.9.10'] ``` Another excellent tool that gives us insight into image details and can analyze the image from an optimization standpoint is [dive](https://github.com/wagoodman/dive) ![](https://i.imgur.com/jRk50dV.gif) You can also configure dive and integrate with your CI tools. ```sh ~> CI=true dive nix-redis-layered Using default CI config Image Source: docker://nix-redis-layered Fetching image... (this can take a while for large images) Analyzing image... efficiency: 100.0000 % wastedBytes: 0 bytes (0 B) userWastedPercent: 0.0000 % Inefficient Files: Count Wasted Space File Path None Results: PASS: highestUserWastedPercent SKIP: highestWastedBytes: rule disabled PASS: lowestEfficiency Result:PASS [Total:3] [Passed:2] [Failed:0] [Warn:0] [Skipped:1] ``` ### Nix Docker Image optimization Let's compare the nix build images with available official Redis images. ```sh ~> docker pull redis Using default tag: latest latest: Pulling from library/redis a330b6cecb98: Pull complete 14bfbab96d75: Pull complete 8b3e2d14a955: Pull complete 5da5e1b21a2f: Pull complete 6af3a5ca4596: Pull complete 4f9efe5b47a5: Pull complete Digest: sha256:e595e79c05c7690f50ef0136acc9d932d65d8b2ce7915d26a68ca3fb41a7db61 Status: Downloaded newer image for redis:latest docker.io/library/redis:latest ~> docker pull redis:alpine alpine: Pulling from library/redis a0d0a0d46f8b: Already exists a04b0375051e: Pull complete cdc2bb0f9590: Pull complete 8f19735ec10c: Pull complete ac5156a4c6ca: Pull complete 7b7e1b3fdb00: Pull complete Digest: sha256:fa785f9bd167b94a6b30210ae32422469f4b0f805f4df12733c2f177f500d1ba Status: Downloaded newer image for redis:alpine docker.io/library/redis:alpine ``` Let's compare ```sh ~> docker images REPOSITORY TAG IMAGE ID CREATED SIZE nix-redis-layered latest 492a73edeac8 27 hours ago 156MB nix-redis latest 13ea7dccae63 27 hours ago 156MB redis latest 02c7f2054405 2 weeks ago 105MB redis alpine f6f2296798e9 3 weeks ago 32.3MB ``` At the 1st glance, we can see nix build is significantly bigger than official images, nix-redis:latest and nix-redis-layered:latest images is 156MB compare with redis:latest based on `debian:buster-slim` 105MB and redis:alpine based on alpine 32.3MB. Official Images are highly optimized with statically linked libraries. Now we can try and see if we can optimize our nix Redis image. We will use a two-stage build that will rely on the previous redis-multi.nix, and we need to have one more nix file redis-multi-mini.nix to optimize the image further. redis-multi-mini.nix ```nix { pkgs ? import <nixpkgs> {} }: import ./redis-multi.nix { inherit pkgs; name = "nix-redis-layered-minimal"; redis = pkgs.redis.overrideAttrs (old: { makeFlags = old.makeFlags ++ ["USE_SYSTEMD=no"]; preBuild = '' set -x makeFlagsArray=(PREFIX="$out" CC="${pkgs.musl.dev}/bin/musl-gcc" CFLAGS="-I${pkgs.musl.dev}/include" LDFLAGS="-L${pkgs.musl.dev}/lib") ''; postInstall = "rm -f $out/bin/redis-{benchmark,check-*,cli}"; }); } ``` Let's Build a new image. ```sh ~> nix-build redis-multi-mini.nix /nix/store/k2q3nk68gvr25v825nh46s8mfdryca4y-nix-redis-layered-minimal.tar.gz ``` The new image is optimized with fewer layers. ```sh ~> tar --list --file result 2a894fdfe28bd1ceee40ddeea14823d89d52c50f4d5bb005c7eac74244e8331e/layer.tar 4c40801bc9f0d6275846f9d6768701ee4a87c3cd199c21fbd27d28529d9d7fb0/layer.tar 0d9152ec11246d30cc6f6935e793a28f46db2cd5c7e1ef8bda1125e228526d2f/layer.tar ee626d23550b2811c38be9c7a5cf6eeb3a62425f1bf5363758b75ed0f8b55c7f/layer.tar 8023da80f76ff1c01ed438b5581533c4b4584b430218c15ee77f51bd388faa8c/layer.tar 1c5ba5caa619e901adde8f8062a324370f5a11e12fdfe325e04684293b3a112c/layer.tar 36bf402fa650e105532a17d08191e8b489dba14141df3c784a0b5db641bd3cb1.json ``` Importing archive into the Docker as a new image ```sh ~> docker load < result 2a894fdfe28b: Loading layer [==================================================>] 1.649MB/1.649MB 4c40801bc9f0: Loading layer [==================================================>] 276.5kB/276.5kB 0d9152ec1124: Loading layer [==================================================>] 31.6MB/31.6MB ee626d23550b: Loading layer [==================================================>] 4.198MB/4.198MB 8023da80f76f: Loading layer [==================================================>] 2.304MB/2.304MB 1c5ba5caa619: Loading layer [==================================================>] 10.24kB/10.24kB Loaded image: nix-redis-layered-minimal:latest ``` Let's check the size. ```sh ~> docker images REPOSITORY TAG IMAGE ID CREATED SIZE nix-redis-layered-minimal latest 36bf402fa650 25 hours ago 39.2MB nix-redis-layered latest 492a73edeac8 28 hours ago 156MB nix-redis latest 13ea7dccae63 28 hours ago 156MB redis latest 02c7f2054405 2 weeks ago 105MB redis alpine f6f2296798e9 3 weeks ago 32.3MB ``` We Managed to squeeze the image from 156MB to 39.2MB. We can statically link with `CC="${pkgs.musl.dev}/bin/musl-gcc -static"`(currently is broken in Redis build) that can reduce our image to merely 2MB ## Fin Building Docker images with nix expressions solve many inherited issue presented with Dockerfile. This solution is not for everybody as we already selected Nix as our configuration management language was easy to choose. We can find an extensive list of [ Nix expressions examples](https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/docker/examples.nix) The code used in the article you can find on my [GitHub repository](https://github.com/mudrii/docker-nix-build)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.