Docker 社課

Containerization and its benefits

Consistency Across Environments:

Containers are consistent across different environments

Efficiency

Containers share the machine’s OS system kernel and therefore do not require an OS per application, driving higher server efficiencies and reducing server and licensing costs.

Speed

Containers can reduce the time it takes to spin up an app's operating system and begin running the application to mere seconds.

Isolation

Containers isolate application processes from the rest of the system, reducing the risk of system conflicts.

Modularity and Scalability

The container model encourages dividing the application into its functional components which can be independently scaled and managed.

Rapid Deployment and Scaling

Containers include the application and all of its dependencies.

Docker's role in the development

Development

Docker provides a consistent environment for application development and testing, reducing discrepancies and bugs.

Integration and Testing

Automated Docker containers can build and test environments quickly and scale to simulate production environments.

Deployment

Docker ensures that the application operates in any environment by bundling it with all its dependencies.

Maintenance and Update

Containers can be easily stopped, modified, and restarted for maintenance and updates, thus simplifying the process and minimizing downtime.

Underlying Technology

Resource Efficiency

Startup Time

Application Packaging

Isolation

Portability

Density

Performance

Security

Docker 為什麼會比傳統 VM 輕量？

因為少跑了 \(N\) 個完整的 Guest OS

Docker Architecture

docker is a client-server model architecture

Docker Daemon

The server-side service that manages Docker objects such as images, containers, networks, and volumes. It’s the engine that runs on the host machine and responds to requests from the Docker client.

Docker Client

The command-line interface (CLI) that users interact with. The Docker client sends commands to the Docker daemon, which carries them out.

Docker Objects

Images

A Docker image is built up from a series of layers. Each layer represents an instruction in the image's Dockerfile. Each layer except the very last one is read-only.

• image：多個 R/O layer (只讀層，read-only)，它們重疊在一起。除了最下面一層，其它層都會有一個指標指向下一層

what is layer?

Each layer contains two main components:

• Files: contains the files and directories that were created, modified, or deleted in that layer.

• Metadata: contains information such as the command that was run to create the layer, environmental changes, user changes, exposed ports, and volume information.

# syntax=docker/dockerfile:1
 
FROM ubuntu:22.04
LABEL org.opencontainers.image.authors="org@example.com"
COPY . /app
RUN make /app
RUN rm -r $HOME/.cache
CMD python /app/app.py

• Commands that create a layer:
  1. FROM ubuntu:22.04 (line 3) : Starts with the Ubuntu image, creating a base layer.
  2. COPY . /app (line 5) : Copies local files into the image, creating a new layer.
  3. RUN make /app (line 6) : Builds the application, creating a new layer.
  4. RUN rm -r $HOME/.cache (line 7) : Removes the cache directory, creating a new layer.

• Commands that only modify metadata:
  1. LABEL <key>=<value> (line 4) : Sets the image's author label, no new layer.
  2. CMD python /app/app.py (line 8) : Sets the default command to run in the container, no new layer.

• Optmizing build

Each layer is only a set of differences from the layer before it. Note that both adding, and removing files will result in a new layer. In the example above, the $HOME/.cache directory is removed, but will still be available in the previous layer and add up to the image's total size. Refer to the Best practices for writing Dockerfiles and use multi-stage builds sections to learn how to optimize your Dockerfiles for efficient images.

The copy-on-write (CoW) strategy

$ docker image history acme/my-base-image:1.0

IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
da3cf8df55ee   5 minutes ago   RUN /bin/sh -c apk add --no-cache bash # bui⦠  2.15MB    buildkit.dockerfile.v0
<missing>      7 weeks ago     /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      7 weeks ago     /bin/sh -c #(nop) ADD file:f278386b0cef68136⦠  5.6MB

$ docker image history  acme/my-final-image:1.0

IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
8bd85c42fa7f   3 minutes ago   CMD ["/bin/sh" "-c" "/app/hello.sh"]            0B        buildkit.dockerfile.v0
<missing>      3 minutes ago   RUN /bin/sh -c chmod +x /app/hello.sh # buil⦠  39B       buildkit.dockerfile.v0
<missing>      3 minutes ago   COPY . /app # buildkit                          222B      buildkit.dockerfile.v0
<missing>      4 minutes ago   RUN /bin/sh -c apk add --no-cache bash # bui⦠  2.15MB    buildkit.dockerfile.v0
<missing>      7 weeks ago     /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      7 weeks ago     /bin/sh -c #(nop) ADD file:f278386b0cef68136⦠  5.6MB

BuildKit

• Detect and skip executing unused build stages
• Parallelize building independent build stages
• Incrementally transfer only the changed files in your build context between builds
• Detect and skip transferring unused files in your build context
• Use Dockerfile frontend implementations with many new features
• Avoid side effects with rest of the API (intermediate images and containers)
• Prioritize your build cache for automatic pruning

Containers

• container：也是分層的，當執行一個 container 時，會在 image 的最上面加一層 R/W layer (可讀寫層)
  container = image (layer) + R/W layer (container layer)
  對正在執行的 container 所做的所有更改 (如寫入新，修改和刪除檔案) 都會寫入此可寫的 container layer
  當 container 被刪除時，可寫層也會被刪除，但底層 image 保持不變

Networks

1. Default Bridge Network

Containers on the default bridge network can only access each other by IP addresses, unless you use the --link option

docker 會新增一個 software bridge 作為 container 網路對外的出口，預設名稱為 docker0
docker0 會與 host 中的對外網卡(上圖為 eth0)相連，藉此取得對外連線的能力
每個 container 會使用一個 veth device 與 docker0 相連，因此具備連外能力

2. user-defined Bridge Network

more isolated and flexible

$ docker network create --driver bridge my-bridge

$ docker run -d --name app3 --network my-bridge nginx:alpine
$ docker run -d --name app4 --network my-bridge nginx:alpine
$ docker run -d --name app2 nginx:alpine

Try to reach app4 from app3 using the hostname app4:

$ docker exec app3 curl app4 # succeess

Try to reach app3 from app4:

$ docker exec app4 curl app3 # succeess

Will we be able to reach app2 from app3 using DNS?

$ docker exec app3 curl app2 # failed

3. Host Network

此模式的網路有以下特點：

• 不使用獨立的網路，而是與 Docker Host 使用相同的網路

• container 如果沒有對外開 port 提供服務，其實設定為 host mode 並沒有意義

網路架構如下圖：

4. more but I didnt research

• overlay
• IPvlan
• Macblan
• None

Volumes

Volumes is managed by Docker
(located at /var/lib/docker/volumes/ on linux)

1. volumes

• Create a volume:

$ docker volume create my_volume

• List volumes:

$ docker volume ls

• Run a container with a volume:
  This command runs a container and mounts the volume my_volume to the container at the path /data.

$ docker run -d --name my_container -v my_volume:/data my_image

• Inspect a volume:

$ docker volume inspect my_volume

• Remove a volume:

$ docker volume rm my_volume

2. bind mounts

Bind mounts may be stored anywhere on the host system. They may even be important system files or directories.

• Run a container with a bind mount:
  This command mounts the host directory /path/on/host to the container at the path /path/in/container.

$ docker run -d --name my_container -v /path/on/host:/path/in/container my_image

rw vs. ro in docker volume

• Read-Only (ro)

# volume:
$ docker run -d --name my_container -v my_volume:/data:ro my_image

# bind mounts:
$ docker run -d \
--name my_container \
-v /path/on/host:/path/in/container:ro \
my_image

• Read-Write (rw)

# volume:
$ docker run -d --name my_container -v my_volume:/data:rw my_image

# bind mounts:
$ docker run -d \
--name my_container \
-v /path/on/host:/path/in/container:rw \
my_image

Docker(Container) Registries

more registries…

refer to: https://docs.docker.com/engine/install/

Play with Docker

如果你不想安裝 Docker，有另一種方式可以練習 Docker，那就是 Docker 官方提供的 Play with Docker (PWD)

Configuring Docker for your environment

After installing Docker, when you first run docker info to check Docker system information, you might encounter a permission error because the Docker daemon binds to a Unix socket owned by root. By default, non-root users need to prepend sudo to Docker commands.

Reason

The permission issue arises because the Docker daemon uses a Unix socket, not a TCP port, and by default, it's owned by root. To use Docker commands without sudo, you need to be part of the docker Unix group that has permissions to read/write to the Unix socket.

Solution

To avoid using sudo with every Docker command, create a docker group and add your user to it. This allows your user to run Docker commands without sudo. However, be aware that being part of the docker group gives privileges equivalent to the root user, which can affect system security.

Steps to add your user to the docker group:

1. Create the docker group.

$ sudo groupadd docker

2. Add your user to the docker group.

$ sudo usermod -aG docker $USER

3. Log out and log back in, or restart your VM to apply these changes.

4. Verify your user is added to the docker group.

$ id -nG

For users not logged in, use:

$ sudo usermod -a -G docker <username>

Docker Desktop

Compose is an orchestration tool that makes spinning up multi-container distributed applications with Docker an effortless task. It is easy to define a multi-container application and accompanying services in one file, then spin it up in a single command that will start and run your entire app.

history

Introduction to multi-container applications

• Dont want to create each container one by one?
• Dont want to configure network, volumes… with cli line by line?

Writing and understanding docker-compose files

todo list continue…

compose.yaml

services:
  app:
    image: node:18-alpine
    command: sh -c "yarn install && yarn run dev"
    ports:
      - 127.0.0.1:3000:3000
    working_dir: /app
    volumes:
      - ./:/app
    environment:
      MYSQL_HOST: mysql
      MYSQL_USER: root
      MYSQL_PASSWORD: secret
      MYSQL_DB: todos

  mysql:
    image: mysql:8.0
    volumes:
      - todo-mysql-data:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: secret
      MYSQL_DATABASE: todos

volumes:
  todo-mysql-data:

Running and managing composed applications

check this out: https://github.com/docker/awesome-compose

my codespace to demo (you cant access): https://github.com/codespaces/curly-spoon-vr6g6jp6qrr2p9vg

I will show the /flask-redis example

the website at port \(8000\) should look like this:

and after you reload the page the number would increase

Strategies for database and stateful application handling

use docker to compose a primary-replica postgres SQL

https://github.com/eremeykin/pg-primary-replica
https://github.com/peterxcli/pg-primary-replica

version: '3.8'
x-postgres-common:
  &postgres-common
  image: postgres:14-alpine
  user: postgres
  restart: always
  healthcheck:
    test: 'pg_isready -U user --dbname=postgres'
    interval: 10s
    timeout: 5s
    retries: 5

services:
  postgres_primary:
    <<: *postgres-common
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: user
      POSTGRES_DB: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_HOST_AUTH_METHOD: "scram-sha-256\nhost replication all 0.0.0.0/0 md5"
      POSTGRES_INITDB_ARGS: "--auth-host=scram-sha-256"
    command: |
      postgres 
      -c wal_level=replica 
      -c hot_standby=on 
      -c max_wal_senders=10 
      -c max_replication_slots=10 
      -c hot_standby_feedback=on
    volumes:
      - ./00_init.sql:/docker-entrypoint-initdb.d/00_init.sql
      - pgdata_primary:/var/lib/postgresql/data

  postgres_replica:
    <<: *postgres-common
    ports:
      - 5433:5432
    environment:
      PGUSER: replicator
      PGPASSWORD: replicator_password
    command: |
      bash -c "
      until pg_basebackup --pgdata=/var/lib/postgresql/data -R --slot=replication_slot --host=postgres_primary --port=5432
      do
      echo 'Waiting for primary to connect...'
      sleep 1s
      done
      echo 'Backup done, starting replica...'
      chmod 0700 /var/lib/postgresql/data
      postgres
      "
    depends_on:
      - postgres_primary
    volumes:
      - pgdata_replica:/var/lib/postgresql/data

volumes:
  pgdata_primary:
    driver: local
  pgdata_replica:
    driver: local

test if we run successfully

$ psql postgres://user:password@localhost:5432/postgres -xc \
'select * from pg_replication_slots;'

-[ RECORD 1 ]-------+-----------------
slot_name           | replication_slot
plugin              | 
slot_type           | physical
datoid              | 
database            | 
temporary           | f
active              | t
active_pid          | 60
xmin                | 
catalog_xmin        | 
restart_lsn         | 0/3000148
confirmed_flush_lsn | 
wal_status          | reserved
safe_wal_size       | 
two_phase           | f

Github action build image and push to docker hub public container registry

dockerfile

https://github.com/peterxcli/basic-auth-gin/blob/main/Dockerfile

#build stage
FROM golang:alpine AS builder
RUN apk add --no-cache git
WORKDIR /app
COPY . /app
RUN go build -o /app/main -v

#final stage
FROM alpine:latest
RUN apk --no-cache add ca-certificates
COPY --from=builder /app/main /app/main
COPY --from=builder /app/public /app/public
WORKDIR /app
ENTRYPOINT /app/main
LABEL Name=mongogo Version=0.0.1
EXPOSE 9000

github action

https://github.com/peterxcli/basic-auth-gin/blob/main/.github/workflows/ci.yml

# This workflow will build a golang project
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go

name: ci

on: [push, pull_request]

env:
  IMAGE_TAG: ${{ github.sha }}

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        mongodb-version: ["6.0"]
    steps:
      - uses: actions/checkout@v3

      - name: Set up Go
        uses: actions/setup-go@v3
        with:
          go-version: 1.19

      - name: Start MongoDB
        uses: supercharge/mongodb-github-action@1.8.0
        with:
          mongodb-version: ${{ matrix.mongodb-version }}

      - name: Build
        run: go build -v -buildvcs=false ./...

      - name: Test
        env:
          EMAIL: ${{ secrets.EMAIL }}
          SENDGRID_API_KEY: ${{ secrets.SENDGRID_API_KEY }}
        run: make test
        
  build:
    needs: [test]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up QEMU
        uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # v2.1.0

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          push: false
          context: .
          file: ./Dockerfile
          tags: peter0814/basic-auth-gin:${{ github.sha }}
          outputs: type=docker, dest=docker.tar

      - name: docker login
        env:
          DOCKER_USER: ${{ secrets.DOCKER_USER }}
          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
        run: |
          docker login -u $DOCKER_USER -p $DOCKER_PASSWORD

      - name: cache
        uses: actions/upload-artifact@83fd05a356d7e2593de66fc9913b3002723633cb # v3.1.1
        with:
          name: docker.tar
          path: docker.tar

  push:
    if: ${{ github.ref == 'refs/heads/release' }}
    needs : [build]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up QEMU
        uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # v2.1.0

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: docker login
        env:
          DOCKER_USER: ${{ secrets.DOCKER_USER }}
          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
        run: |
          docker login -u $DOCKER_USER -p $DOCKER_PASSWORD

      - name: Download artifact
        uses: actions/download-artifact@v2
        with:
          name: docker.tar
          path: ./

      - name: Load Docker image
        run: |
          docker load --input docker.tar
          docker image ls -a

      - name: Docker Push
        run: docker push peter0814/basic-auth-gin:$IMAGE_TAG

Dev Container

a truely isolated and consisent development environment

https://github.com/peterxcli/basic-auth-gin/tree/main/.devcontainer

some integrated features in vscode

1. vscode extension as Code (VSCEAC)