Why you shouldn't use the docker CLI for your daily tasks

# Why you shouldn't use the docker CLI for your daily tasks Avoid using `docker` CLI. ### Moving containers #### What you want to do We have an image registry.biz/dev/app:a1b2c3, and we want to promote it to prod. Because we care about these things, we use a separate repository for dev images that anybody can push to and for images intended for prod that only your CI/CD pipeline can push to. The way a lot of people has done it accomplish this is to have the pipeline do roughly: docker pull registry.biz/dev/app:v0.52.1 docker tag registry.biz/dev/app:v0.52.1 registry.biz/prod/app:v0.52.1 docker push registry.biz/prod/app:v0.52.1 But, by using Docker to accomplish this, you've used a much heavier tool than you needed, and in doing so, may have made your pipeline less secure. #### What happened The Docker CLI is just a client for the Docker daemon, which runs in the background all the time, and does all the actual work. When you invoke docker pull, your client sends a request to the daemon and tells it to pull the image from the registry. The daemon pulls the image contents from the registry and stores it in some path on your computer, where it stores these things. Next, your pipeline invokes the docker tag, which tells the daemon to refer to those layers as the new image reference. When you invoke docker push, the daemon pushes the layers to the registry, and when your pipeline completes the worker, all its attached storage disappears forever. A new worker starts fresh the next time. It works, except it is slow The registry API protocol is smart enough to avoid pushes for contents it already has elsewhere, so when you docker push that image, the registry might say, "oh, I already have that layer, thanks". If you hadn't buffered it, you could have avoided ever pulling it. The pull/tag/push dance has already made your pipeline worse by making it take longer. But wait, there's more. It Loses Information Because the Docker CLI is mainly focused on running containers, docker pull doesn't bother pulling image data for other architectures it can't run. So if your image happens to be a Docker manifest list or OCI image index (e.g., a multi-architecture image, like most base images), docker pull will only pull the image suitable for the platform you're running on. This means that when you docker tag and docker push that image, you'll only push the image specific to your computer's architecture. This can lead to confusion when comparing the digest of alpine on Dockerhub to your copy that you've Docker pull/tag/pushed to your own repository since your copy will only include one image. Even if you're not dealing with multi-arch manifest lists, information can get lost in the pull/tag/push dance due to compression differences and legacy formats support. Here is an excellent write up about this https://github.com/google/go-containerregistry/issues/895#issuecomment-753521526 All of this is understandable because the `docker` CLI is not mainly a tool for moving images around. It's a tool for pulling and running them. #### It's Less Secure Finally, by involving the Docker daemon in the process, you've accepted its Faustian bargain. The Docker daemon requires escalated privileges to run on your computer*. It needs this to run containers, which is what the Docker CLI is intended for. But in this case, your pipeline doesn't want to run containers. It just wants to move images around some registries. It doesn't need privilege, and it should have as little of it as possible. ### What to do instead Fundamentally the Docker CLI is not primarily a registry API client. It's a container runtime client focused on running images. So let's not use that. Several tools aim to make dealing with container images easier. Two of these are [Skopeo](https://github.com/containers/skopeo) and [crane](https://github.com/google/go-containerregistry/blob/main/cmd/crane/doc/crane.md). crane and Skopeo are great tools. What you were trying to do was move an image from registry.biz/dev/app:v0.52.1 to registry.biz/prod/app:v0.52.1. So let's just do that: `crane cp registry.biz/dev/app:v0.52.1 registry.biz/prod/app:v0.52.1` `crane cp` will make all the necessary registry API calls to copy the image from A to B by efficiently streaming layer blobs to your computer and streaming that data to the target registry, taking cross-repository mounting into account to avoid pulling anything it can. Not a single byte of data hits your disk. If the target registry says, "got it, thanks", that layer data is never pulled from the source registry. And, because you're not involving the Docker daemon, this can be done in a containerized build environment without unlocking doors to some bottomless Lovecraftian abyss. Faster. More Secure. Nice. ### Building Container Images This one's a bit more complicated. Practically speaking, you might have no option but to use the Docker CLI and Dockerfiles to build your image. It might just be unavoidable. But, it's worth considering whether it's unavoidable and what kinds of trade-offs you're making. Don't just use Dockerfiles because that's what the tutorial you read in 2017 told you to use. Understand what you're doing. I'm going to focus on building container images for Go, because that's what many people do these days. But the same spirit of curiosity applies to any language or framework. #### What you want to do You have a Go application, and you'd like to put it in a container image, and push it to a registry. Seems easy. Like any good software engineer, you google dockerfile golang, and the first result is this page on Docker's site. You're even excellent and thoughtful and scroll all the way down to the part about multi-stage builds for smaller, faster images! It tells you to use this Dockerfile: ``` # syntax=docker/dockerfile:1 ## ## Build ## FROM golang:1.16-buster AS build WORKDIR /app COPY go.mod ./ COPY go.sum ./ RUN go mod download COPY *.go ./ RUN go build -o /docker-gs-ping ## ## Deploy ## FROM gcr.io/distroless/base-debian10 WORKDIR / COPY --from=build /docker-gs-ping /docker-gs-ping EXPOSE 8080 USER nonroot:nonroot ENTRYPOINT ["/docker-gs-ping"] ``` You take that, modify it to fit your codebase, docker build it, docker push, it works, you go back to writing code. But, by using docker to accomplish this, you've used a much heavier tool than you needed, and in doing so, may have made your pipeline less secure. #### What happened When docker build runs through that Dockerfile, it builds up the resulting container image on the base image (FROM), by either copying some files into it (COPY, ADD), or for RUN, by running the specified command in the image its built up, and capturing any changes. There are also some metadata changes like EXPOSE, USER, and ENTRYPOINT. But the real workhorse of this build is the go build line. Every other line in the ## Build section is carefully setting up the build environment so that it can run go build at the end. The ## Deploy section is just taking the built Go binary out of the build image, putting it on top of the Distroless image (yay distroless!), and telling the image to run that binary on startup. But why does this need to be done in a container? For Node and Python apps I get it, setting up isolated, repeatable, efficient build environments for those languages can be really hard. Again, using docker build and Dockerfiles might be the best answer for you, but make that decision consciously. Anyway, back to my Go example. What's wrong there? #### It's Slow Go takes care to make builds isolated and repeatable. In general, you don't have to worry about go build, starting other random processes, or getting tricked into mining bitcoin. Setting up an isolated build environment in a container using the Dockerfile above means you lose out on one of the best tools for making a build run faster: caching. In all that setup to ADD and COPY files into the container environment, we never mounted the Go build cache, so every time you docker build it's doing a go build as if it's never seen a line of Go before. Because this container hasn't. Even if you did mount in the Go build cache, you're still unnecessarily copying files around from your dev environment to this containerized build environment, for what? So it can run go build without side effects? go build already doesn't have side effects. #### It's Less Secure As mentioned, docker build necessarily runs a container every time it encounters a RUN directive. As with moving images, this involves giving the build process privileges, which in your delivery pipeline can lead to compromises. For many languages (Python, Node, etc.), this is going to be largely unavoidable. The value of being able to repeatably build your application in isolation makes it worth dealing with the headache of making that work in a containerized build environment. But for Go, it's usually avoidable. So let's avoid it. #### What to do instead go build my app, put it in a container image. For a long time, "just put this file into a container image" was like some kind of unattainable dark art. The only way to do that was involved FROM and ADD and our old friend docker build. And again, because it works, a lot of people just stop there. But it's not that hard at its core. The crane tool has a crane append command that takes a tarball and an image reference, and adds the tarball as a new top layer on the base image, and pushes that to a registry. That's it. As with crane cp, it does it entirely using regisry API operations, streaming and deduping blob uploads, without having to run any containers on your computer. With crane append, you could go build your binary, tar it up, and append it to a base image. There's even a recipe to do it in a few lines of Bash. That operation -- go build && crane append -- is so powerful, which led to creating a tool ✨[ko](https://github.com/google/ko)✨. ko publish (soon to be renamed ko build) takes a Go importpath, builds it, and pushes it to a registry. If you go build ./cmd/app to get a Go binary, you can ko build ./cmd/app to get a container image that runs that Go binary. Because it's just go build, tarring and registry operations to append a layer, ko doesn't require that pesky Docker daemon and its pesky privileges. Because it's running go build in your regular development environment, it takes advantage of your regular build cache. If the code hasn't changed, ko build won't push any new layers. ko has a lot more than just ko build, like dead simple multi-arch images, YAML templating integration, and more. #### But What About Non-Go? That's all well and good for Go, but for a Java developer, or a Node developer, or one of those Rust folks I keep hearing about. For Java, there's [Jib](https://github.com/GoogleContainerTools/jib), which integrates with Maven and Gradle and does basically what ko does -- it builds an executable jar outside in your stable build environment, puts it on top of a base image, and pushes it to a registry. The same sort of thing should be possible with Rust. You could build the Rust executable with cargo build, and crane appends it to a base image that provides glibc.