# Docker 社課 [TOC] --- ## 自我介紹 - 名字: 李緒成 - 動機: 對 docker 不熟, 所以想準備 docker - 聯絡資訊： - https://www.linkedin.com/in/peterxcli - https://github.com/peterxcli - https://www.facebook.com/peterxcli ----  --- how can we handle this?  --- ## Introduction to Docker ---- ### Containerization and its benefits ---- #### Consistency Across Environments: Containers are consistent across different environments  ---- #### Efficiency Containers share the machine’s OS system kernel and therefore do not require an OS per application, driving higher server efficiencies and reducing server and licensing costs. ---- #### Speed Containers can reduce the time it takes to spin up an app's operating system and begin running the application to mere seconds. ---- #### Isolation Containers isolate application processes from the rest of the system, reducing the risk of system conflicts. ---- #### Modularity and Scalability The container model encourages dividing the application into its functional components which can be independently scaled and managed. ---- #### Rapid Deployment and Scaling Containers include the application and all of its dependencies.  ---- ### Docker's role in the development ---- #### Development Docker provides a consistent environment for application development and testing, reducing discrepancies and bugs. ---- #### Integration and Testing Automated Docker containers can build and test environments quickly and scale to simulate production environments. ---- #### Deployment Docker ensures that the application operates in any environment by bundling it with all its dependencies. ---- #### Maintenance and Update Containers can be easily stopped, modified, and restarted for maintenance and updates, thus simplifying the process and minimizing downtime. --- ### Comparison with traditional virtualization ----  ---- #### Underlying Technology | Aspect | Container | Traditional Virtualization | |--------|-------------------------|----------------------------| | Technology | Container runtime on host OS kernel. | Hypervisors managing full virtual machines with separate kernels. | ---- #### Resource Efficiency | Aspect | Container | Traditional Virtualization | |--------|-------------------------|----------------------------| | Efficiency | Shares OS kernel, minimal overhead. | Requires full guest OS, more resource-intensive. | ---- #### Startup Time | Aspect | Container | Traditional Virtualization | |--------|-------------------------|----------------------------| | Startup Speed | Starts in seconds. | Takes minutes to boot a full OS. | ---- #### Application Packaging | Aspect | Container | Traditional Virtualization | |--------|-------------------------|----------------------------| | Packaging | Packages applications with dependencies in images. | VMs include OS, application, and all dependencies. | ---- #### Isolation | Aspect | Container | Traditional Virtualization | |--------|-------------------------|----------------------------| | Level of Isolation | Process-level, using namespaces and cgroups. | Hardware-level, each VM operates independently. | ---- #### Portability | Aspect | Container | Traditional Virtualization | |--------|-------------------------|----------------------------| | Portability | Highly portable across different environments. | Less portable, often tied to hypervisor formats. | ---- #### Density | Aspect | Container | Traditional Virtualization | |--------|-------------------------|----------------------------| | Workload Density | Higher density due to lower overhead. | Lower density due to overhead of multiple OS instances. | ---- #### Performance | Aspect | Container | Traditional Virtualization | |--------|-------------------------|----------------------------| | Performance | Near-native, direct access to host kernel. | Reduced by hypervisor overhead. | ---- #### Security | Aspect | Container | Traditional Virtualization | |--------|-------------------------|----------------------------| | Security | Less isolated, depends on host OS security. | More isolated, advantageous for sensitive applications. | ---- #### Docker 為什麼會比傳統 VM 輕量？ <!-- 因為少跑了 N 個完整的 Guest OS，下面就是 Virtualization 和 Containerization 的比較表： --> | | Virtualization | Containerization | | ------------- | --------------- | ---------------- | | 啟動 | 慢 (最快也要分鐘) | 快 (秒開) | | 容量 | 大 (GB) | 小 (MB) | | 效能 | 慢 | 快 | | host 可支撐數量 | 數個～數十個 | 數個～數百個 | | 複製相同環境 | 超慢 | 快 | ----  因為少跑了 $N$ 個完整的 Guest OS --- ## Docker Fundamentals ---- ### Docker Architecture docker is a client-server model architecture  ----  ---- #### **Docker Daemon** The server-side service that manages Docker objects such as `images`, `containers`, `networks`, and `volumes`. It’s the engine that runs on the host machine and responds to requests from the Docker client. ---- #### **Docker Client** The command-line interface (CLI) that users interact with. The Docker client sends commands to the Docker daemon, which carries them out. ---- #### **Docker Objects** ---- ##### **Images** A Docker image is built up from a series of layers. Each layer represents an instruction in the image's Dockerfile. Each layer except the very last one is read-only. - image：多個 R/O layer (只讀層，read-only)，它們重疊在一起。除了最下面一層，其它層都會有一個指標指向下一層 ----  ---- ###### what is layer? Each layer contains two main components: - **Files**: contains the files and directories that were created, modified, or deleted in that layer. - **Metadata**: contains information such as the command that was run to create the layer, environmental changes, user changes, exposed ports, and volume information. ---- ```dockerfile= # syntax=docker/dockerfile:1 FROM ubuntu:22.04 LABEL org.opencontainers.image.authors="org@example.com" COPY . /app RUN make /app RUN rm -r $HOME/.cache CMD python /app/app.py ``` ---- - Commands that create a layer: 1. `FROM ubuntu:22.04` (line 3) : Starts with the Ubuntu image, creating a base layer. 2. `COPY . /app` (line 5) : Copies local files into the image, creating a new layer. 3. `RUN make /app` (line 6) : Builds the application, creating a new layer. 4. `RUN rm -r $HOME/.cache` (line 7) : Removes the cache directory, creating a new layer. ---- - Commands that only modify metadata: 1. `LABEL <key>=<value>` (line 4) : Sets the image's author label, no new layer. 2. `CMD python /app/app.py` (line 8) : Sets the default command to run in the container, no new layer. ---- - Optmizing build Each layer is only a set of differences from the layer before it. **Note that both adding, and removing files will result in a new layer**. In the example above, the `$HOME/.cache` directory is removed, but will still be available in the previous layer and add up to the image's total size. Refer to the [Best practices for writing Dockerfiles](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) and use [multi-stage builds](https://docs.docker.com/build/building/multi-stage/) sections to learn how to optimize your Dockerfiles for efficient images. ---- ##### The copy-on-write (CoW) strategy ```shell $ docker image history acme/my-base-image:1.0 IMAGE CREATED CREATED BY SIZE COMMENT da3cf8df55ee 5 minutes ago RUN /bin/sh -c apk add --no-cache bash # bui… 2.15MB buildkit.dockerfile.v0 <missing> 7 weeks ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B <missing> 7 weeks ago /bin/sh -c #(nop) ADD file:f278386b0cef68136… 5.6MB ``` ```shell $ docker image history acme/my-final-image:1.0 IMAGE CREATED CREATED BY SIZE COMMENT 8bd85c42fa7f 3 minutes ago CMD ["/bin/sh" "-c" "/app/hello.sh"] 0B buildkit.dockerfile.v0 <missing> 3 minutes ago RUN /bin/sh -c chmod +x /app/hello.sh # buil… 39B buildkit.dockerfile.v0 <missing> 3 minutes ago COPY . /app # buildkit 222B buildkit.dockerfile.v0 <missing> 4 minutes ago RUN /bin/sh -c apk add --no-cache bash # bui… 2.15MB buildkit.dockerfile.v0 <missing> 7 weeks ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B <missing> 7 weeks ago /bin/sh -c #(nop) ADD file:f278386b0cef68136… 5.6MB ``` ---- ##### [BuildKit](https://docs.docker.com/build/buildkit/) - Detect and skip executing unused build stages - Parallelize building independent build stages - Incrementally transfer only the changed files in your build context between builds - Detect and skip transferring unused files in your build context - Use Dockerfile frontend implementations with many new features - Avoid side effects with rest of the API (intermediate images and containers) - Prioritize your build cache for automatic pruning ---- ##### **Containers**  ---- - container：也是分層的，當執行一個 container 時，會在 image 的最上面加一層 R/W layer (可讀寫層) container = image (layer) + R/W layer (container layer) 對正在執行的 container 所做的所有更改 (如寫入新，修改和刪除檔案) 都會寫入此可寫的 container layer 當 container 被刪除時，可寫層也會被刪除，但底層 image 保持不變 ---- ##### **Networks** ---- ###### 1. Default Bridge Network Containers on the default bridge network can only access each other by IP addresses, unless you use the `--link` option ----  ---- docker 會新增一個 software bridge 作為 container 網路對外的出口，預設名稱為 docker0 docker0 會與 host 中的對外網卡(上圖為 eth0)相連，藉此取得對外連線的能力 每個 container 會使用一個 veth device 與 docker0 相連，因此具備連外能力 ---- ###### 2. user-defined Bridge Network more isolated and flexible ---- ```shell $ docker network create --driver bridge my-bridge ``` ```shell $ docker run -d --name app3 --network my-bridge nginx:alpine $ docker run -d --name app4 --network my-bridge nginx:alpine $ docker run -d --name app2 nginx:alpine ``` ---- Try to reach app4 from app3 using the hostname app4: ```shell $ docker exec app3 curl app4 # succeess ``` Try to reach app3 from app4: ```shell $ docker exec app4 curl app3 # succeess ``` Will we be able to reach app2 from app3 using DNS? ```shell $ docker exec app3 curl app2 # failed ``` ---- ###### 3. Host Network 此模式的網路有以下特點： - 不使用獨立的網路，而是與 Docker Host 使用相同的網路 - container 如果沒有對外開 port 提供服務，其實設定為 host mode 並沒有意義 ---- 網路架構如下圖：  ---- ##### 4. more but I didnt research - overlay - IPvlan - Macblan - None ---- ##### **Volumes** ----  Volumes is managed by Docker (located at `/var/lib/docker/volumes/` on linux) ---- ###### 1. volumes - **Create a volume:** ```shell $ docker volume create my_volume ``` - **List volumes:** ```shell $ docker volume ls ``` ---- - **Run a container with a volume:** This command runs a container and mounts the volume `my_volume` to the container at the path `/data`. ```shell $ docker run -d --name my_container -v my_volume:/data my_image ``` - **Inspect a volume:** ```shell $ docker volume inspect my_volume ``` - **Remove a volume:** ```shell $ docker volume rm my_volume ``` ---- ###### 2. bind mounts Bind mounts may be stored anywhere on the host system. They may even be important system files or directories. ---- - **Run a container with a bind mount:** This command mounts the host directory `/path/on/host` to the container at the path `/path/in/container`. ```shell $ docker run -d --name my_container -v /path/on/host:/path/in/container my_image ``` ---- ##### **`rw` vs. `ro` in docker volume** - Read-Only (`ro`) ```shell # volume: $ docker run -d --name my_container -v my_volume:/data:ro my_image # bind mounts: $ docker run -d \ --name my_container \ -v /path/on/host:/path/in/container:ro \ my_image ``` - Read-Write (`rw`) ```shell # volume: $ docker run -d --name my_container -v my_volume:/data:rw my_image # bind mounts: $ docker run -d \ --name my_container \ -v /path/on/host:/path/in/container:rw \ my_image ``` ---- #### **Docker(Container) Registries**  ---- more registries...  --- ## Installation and Configuration ---- refer to: https://docs.docker.com/engine/install/ ----  ----  ---- ### Play with Docker 如果你不想安裝 Docker，有另一種方式可以練習 Docker，那就是 Docker 官方提供的 [Play with Docker (PWD)](https://labs.play-with-docker.com/)  ---- ### Configuring Docker for your environment After installing Docker, when you first run `docker info` to check Docker system information, you might encounter a permission error because the Docker daemon binds to a Unix socket owned by `root`. By default, non-root users need to prepend `sudo` to Docker commands. ---- #### Reason The permission issue arises because the Docker daemon uses a Unix socket, not a TCP port, and by default, it's owned by `root`. To use Docker commands without `sudo`, you need to be part of the `docker` Unix group that has permissions to read/write to the Unix socket. ---- #### Solution To avoid using `sudo` with every Docker command, create a `docker` group and add your user to it. This allows your user to run Docker commands without `sudo`. However, be aware that being part of the `docker` group gives privileges equivalent to the `root` user, which can affect system security. ---- #### Steps to add your user to the `docker` group: ---- 1. Create the `docker` group. ```shell $ sudo groupadd docker ``` 2. Add your user to the `docker` group. ```shell $ sudo usermod -aG docker $USER ``` 3. Log out and log back in, or restart your VM to apply these changes. 4. Verify your user is added to the `docker` group. ```shell $ id -nG ``` For users not logged in, use: ```shell $ sudo usermod -a -G docker <username> ``` ---- ### Docker Desktop  --- ## Basic Docker Operations see: https://hackmd.io/@titangene/rk3zjKVIz --- ## Docker Compose  ---- Compose is an orchestration tool that makes spinning up ==multi-container== distributed applications with Docker an effortless task. It is easy to define a multi-container application and accompanying services in one file, then spin it up in a ==single command== that will start and run your entire app. ----  ---- ### history  ---- ### Introduction to multi-container applications  ---- - Dont want to create each container one by one? - Dont want to configure network, volumes... with cli line by line? ---- ### Writing and understanding docker-compose files  ---- todo list continue...  ---- `compose.yaml` ```yaml services: app: image: node:18-alpine command: sh -c "yarn install && yarn run dev" ports: - 127.0.0.1:3000:3000 working_dir: /app volumes: - ./:/app environment: MYSQL_HOST: mysql MYSQL_USER: root MYSQL_PASSWORD: secret MYSQL_DB: todos mysql: image: mysql:8.0 volumes: - todo-mysql-data:/var/lib/mysql environment: MYSQL_ROOT_PASSWORD: secret MYSQL_DATABASE: todos volumes: todo-mysql-data: ``` ---- ### Running and managing composed applications check this out: https://github.com/docker/awesome-compose my codespace to demo (you cant access): https://github.com/codespaces/curly-spoon-vr6g6jp6qrr2p9vg I will show the /flask-redis example ---- the website at port $8000$ should look like this:  and after you reload the page the number would increase ---- ### Strategies for database and stateful application handling ---- #### use docker to compose a primary-replica postgres SQL  ---- https://github.com/eremeykin/pg-primary-replica https://github.com/peterxcli/pg-primary-replica ```yaml! version: '3.8' x-postgres-common: &postgres-common image: postgres:14-alpine user: postgres restart: always healthcheck: test: 'pg_isready -U user --dbname=postgres' interval: 10s timeout: 5s retries: 5 services: postgres_primary: <<: *postgres-common ports: - 5432:5432 environment: POSTGRES_USER: user POSTGRES_DB: postgres POSTGRES_PASSWORD: password POSTGRES_HOST_AUTH_METHOD: "scram-sha-256\nhost replication all 0.0.0.0/0 md5" POSTGRES_INITDB_ARGS: "--auth-host=scram-sha-256" command: | postgres -c wal_level=replica -c hot_standby=on -c max_wal_senders=10 -c max_replication_slots=10 -c hot_standby_feedback=on volumes: - ./00_init.sql:/docker-entrypoint-initdb.d/00_init.sql - pgdata_primary:/var/lib/postgresql/data postgres_replica: <<: *postgres-common ports: - 5433:5432 environment: PGUSER: replicator PGPASSWORD: replicator_password command: | bash -c " until pg_basebackup --pgdata=/var/lib/postgresql/data -R --slot=replication_slot --host=postgres_primary --port=5432 do echo 'Waiting for primary to connect...' sleep 1s done echo 'Backup done, starting replica...' chmod 0700 /var/lib/postgresql/data postgres " depends_on: - postgres_primary volumes: - pgdata_replica:/var/lib/postgresql/data volumes: pgdata_primary: driver: local pgdata_replica: driver: local ``` ---- test if we run successfully ```shell $ psql postgres://user:password@localhost:5432/postgres -xc \ 'select * from pg_replication_slots;' -[ RECORD 1 ]-------+----------------- slot_name | replication_slot plugin | slot_type | physical datoid | database | temporary | f active | t active_pid | 60 xmin | catalog_xmin | restart_lsn | 0/3000148 confirmed_flush_lsn | wal_status | reserved safe_wal_size | two_phase | f ``` --- ## ~~Advanced~~ More Use Cases ---- ### Github action build image and push to docker hub public container registry  ---- dockerfile https://github.com/peterxcli/basic-auth-gin/blob/main/Dockerfile ```dockerfile #build stage FROM golang:alpine AS builder RUN apk add --no-cache git WORKDIR /app COPY . /app RUN go build -o /app/main -v #final stage FROM alpine:latest RUN apk --no-cache add ca-certificates COPY --from=builder /app/main /app/main COPY --from=builder /app/public /app/public WORKDIR /app ENTRYPOINT /app/main LABEL Name=mongogo Version=0.0.1 EXPOSE 9000 ``` ---- docker compose file https://github.com/peterxcli/basic-auth-gin/blob/main/.github/workflows/ci.yml ```yaml # This workflow will build a golang project # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go name: ci on: [push, pull_request] env: IMAGE_TAG: ${{ github.sha }} jobs: test: runs-on: ubuntu-latest strategy: matrix: mongodb-version: ["6.0"] steps: - uses: actions/checkout@v3 - name: Set up Go uses: actions/setup-go@v3 with: go-version: 1.19 - name: Start MongoDB uses: supercharge/mongodb-github-action@1.8.0 with: mongodb-version: ${{ matrix.mongodb-version }} - name: Build run: go build -v -buildvcs=false ./... - name: Test env: EMAIL: ${{ secrets.EMAIL }} SENDGRID_API_KEY: ${{ secrets.SENDGRID_API_KEY }} run: make test build: needs: [test] runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up QEMU uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # v2.1.0 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v2 - name: Build and push uses: docker/build-push-action@v4 with: push: false context: . file: ./Dockerfile tags: peter0814/basic-auth-gin:${{ github.sha }} outputs: type=docker, dest=docker.tar - name: docker login env: DOCKER_USER: ${{ secrets.DOCKER_USER }} DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }} run: | docker login -u $DOCKER_USER -p $DOCKER_PASSWORD - name: cache uses: actions/upload-artifact@83fd05a356d7e2593de66fc9913b3002723633cb # v3.1.1 with: name: docker.tar path: docker.tar push: if: ${{ github.ref == 'refs/heads/release' }} needs : [build] runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up QEMU uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # v2.1.0 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v2 - name: docker login env: DOCKER_USER: ${{ secrets.DOCKER_USER }} DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }} run: | docker login -u $DOCKER_USER -p $DOCKER_PASSWORD - name: Download artifact uses: actions/download-artifact@v2 with: name: docker.tar path: ./ - name: Load Docker image run: | docker load --input docker.tar docker image ls -a - name: Docker Push run: docker push peter0814/basic-auth-gin:$IMAGE_TAG ``` ---- ### Dev Container a truely isolated and consisent development environment https://github.com/peterxcli/basic-auth-gin/tree/main/.devcontainer ---- #### some integrated features in vscode 1. vscode extension as Code (VSCEAC)  --- ## reference ### [Docker 系列筆記](https://hackmd.io/@titangene/docker-collection/)