owned this note
owned this note
Published
Linked with GitHub
---
type: slide
slideOptions:
spotlight:
enabled: false
transition: 'slide'
parallaxBackgroundImage: 'https://s3.amazonaws.com/hakim-static/reveal-js/reveal-parallax-1.jpg'
---
# Docker 社課
[TOC]
---
## 自我介紹
- 名字: 李緒成
- 動機: 對 docker 不熟, 所以想準備 docker
- 聯絡資訊：
- https://www.linkedin.com/in/peterxcli
- https://github.com/peterxcli
- https://www.facebook.com/peterxcli
----
---
how can we handle this?
---
## Introduction to Docker
----
### Containerization and its benefits
----
#### Consistency Across Environments:
Containers are consistent across different environments
----
#### Efficiency
Containers share the machine’s OS system kernel and therefore do not require an OS per application, driving higher server efficiencies and reducing server and licensing costs.
----
#### Speed
Containers can reduce the time it takes to spin up an app's operating system and begin running the application to mere seconds.
----
#### Isolation
Containers isolate application processes from the rest of the system, reducing the risk of system conflicts.
----
#### Modularity and Scalability
The container model encourages dividing the application into its functional components which can be independently scaled and managed.
----
#### Rapid Deployment and Scaling
Containers include the application and all of its dependencies.
----
### Docker's role in the development
----
#### Development
Docker provides a consistent environment for application development and testing, reducing discrepancies and bugs.
----
#### Integration and Testing
Automated Docker containers can build and test environments quickly and scale to simulate production environments.
----
#### Deployment
Docker ensures that the application operates in any environment by bundling it with all its dependencies.
----
#### Maintenance and Update
Containers can be easily stopped, modified, and restarted for maintenance and updates, thus simplifying the process and minimizing downtime.
---
### Comparison with traditional virtualization
----
----
#### Underlying Technology
| Aspect | Container | Traditional Virtualization |
|--------|-------------------------|----------------------------|
| Technology | Container runtime on host OS kernel. | Hypervisors managing full virtual machines with separate kernels. |
----
#### Resource Efficiency
| Aspect | Container | Traditional Virtualization |
|--------|-------------------------|----------------------------|
| Efficiency | Shares OS kernel, minimal overhead. | Requires full guest OS, more resource-intensive. |
----
#### Startup Time
| Aspect | Container | Traditional Virtualization |
|--------|-------------------------|----------------------------|
| Startup Speed | Starts in seconds. | Takes minutes to boot a full OS. |
----
#### Application Packaging
| Aspect | Container | Traditional Virtualization |
|--------|-------------------------|----------------------------|
| Packaging | Packages applications with dependencies in images. | VMs include OS, application, and all dependencies. |
----
#### Isolation
| Aspect | Container | Traditional Virtualization |
|--------|-------------------------|----------------------------|
| Level of Isolation | Process-level, using namespaces and cgroups. | Hardware-level, each VM operates independently. |
----
#### Portability
| Aspect | Container | Traditional Virtualization |
|--------|-------------------------|----------------------------|
| Portability | Highly portable across different environments. | Less portable, often tied to hypervisor formats. |
----
#### Density
| Aspect | Container | Traditional Virtualization |
|--------|-------------------------|----------------------------|
| Workload Density | Higher density due to lower overhead. | Lower density due to overhead of multiple OS instances. |
----
#### Performance
| Aspect | Container | Traditional Virtualization |
|--------|-------------------------|----------------------------|
| Performance | Near-native, direct access to host kernel. | Reduced by hypervisor overhead. |
----
#### Security
| Aspect | Container | Traditional Virtualization |
|--------|-------------------------|----------------------------|
| Security | Less isolated, depends on host OS security. | More isolated, advantageous for sensitive applications. |
----
#### Docker 為什麼會比傳統 VM 輕量？
<!--
因為少跑了 N 個完整的 Guest OS，下面就是 Virtualization 和 Containerization 的比較表：
-->
| | Virtualization | Containerization |
| ------------- | --------------- | ---------------- |
| 啟動 | 慢 (最快也要分鐘) | 快 (秒開) |
| 容量 | 大 (GB) | 小 (MB) |
| 效能 | 慢 | 快 |
| host 可支撐數量 | 數個～數十個 | 數個～數百個 |
| 複製相同環境 | 超慢 | 快 |
----
因為少跑了 $N$ 個完整的 Guest OS
---
## Docker Fundamentals
----
### Docker Architecture
docker is a client-server model architecture
----
----
#### **Docker Daemon**
The server-side service that manages Docker objects such as `images`, `containers`, `networks`, and `volumes`. It’s the engine that runs on the host machine and responds to requests from the Docker client.
----
#### **Docker Client**
The command-line interface (CLI) that users interact with. The Docker client sends commands to the Docker daemon, which carries them out.
----
#### **Docker Objects**
----
##### **Images**
A Docker image is built up from a series of layers. Each layer represents an instruction in the image's Dockerfile. Each layer except the very last one is read-only.
- image：多個 R/O layer (只讀層，read-only)，它們重疊在一起。除了最下面一層，其它層都會有一個指標指向下一層
----
----
###### what is layer?
Each layer contains two main components:
- **Files**: contains the files and directories that were created, modified, or deleted in that layer.
- **Metadata**: contains information such as the command that was run to create the layer, environmental changes, user changes, exposed ports, and volume information.
----
```dockerfile=
# syntax=docker/dockerfile:1
FROM ubuntu:22.04
LABEL org.opencontainers.image.authors="org@example.com"
COPY . /app
RUN make /app
RUN rm -r $HOME/.cache
CMD python /app/app.py
```
----
- Commands that create a layer:
1. `FROM ubuntu:22.04` (line 3) : Starts with the Ubuntu image, creating a base layer.
2. `COPY . /app` (line 5) : Copies local files into the image, creating a new layer.
3. `RUN make /app` (line 6) : Builds the application, creating a new layer.
4. `RUN rm -r $HOME/.cache` (line 7) : Removes the cache directory, creating a new layer.
----
- Commands that only modify metadata:
1. `LABEL <key>=<value>` (line 4) : Sets the image's author label, no new layer.
2. `CMD python /app/app.py` (line 8) : Sets the default command to run in the container, no new layer.
----
- Optmizing build
Each layer is only a set of differences from the layer before it. **Note that both adding, and removing files will result in a new layer**. In the example above, the `$HOME/.cache` directory is removed, but will still be available in the previous layer and add up to the image's total size. Refer to the [Best practices for writing Dockerfiles](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) and use [multi-stage builds](https://docs.docker.com/build/building/multi-stage/) sections to learn how to optimize your Dockerfiles for efficient images.
----
##### The copy-on-write (CoW) strategy
```shell
$ docker image history acme/my-base-image:1.0
IMAGE CREATED CREATED BY SIZE COMMENT
da3cf8df55ee 5 minutes ago RUN /bin/sh -c apk add --no-cache bash # bui⦠2.15MB buildkit.dockerfile.v0
<missing> 7 weeks ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B
<missing> 7 weeks ago /bin/sh -c #(nop) ADD file:f278386b0cef68136⦠5.6MB
```
```shell
$ docker image history acme/my-final-image:1.0
IMAGE CREATED CREATED BY SIZE COMMENT
8bd85c42fa7f 3 minutes ago CMD ["/bin/sh" "-c" "/app/hello.sh"] 0B buildkit.dockerfile.v0
<missing> 3 minutes ago RUN /bin/sh -c chmod +x /app/hello.sh # buil⦠39B buildkit.dockerfile.v0
<missing> 3 minutes ago COPY . /app # buildkit 222B buildkit.dockerfile.v0
<missing> 4 minutes ago RUN /bin/sh -c apk add --no-cache bash # bui⦠2.15MB buildkit.dockerfile.v0
<missing> 7 weeks ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B
<missing> 7 weeks ago /bin/sh -c #(nop) ADD file:f278386b0cef68136⦠5.6MB
```
----
##### [BuildKit](https://docs.docker.com/build/buildkit/)
- Detect and skip executing unused build stages
- Parallelize building independent build stages
- Incrementally transfer only the changed files in your build context between builds
- Detect and skip transferring unused files in your build context
- Use Dockerfile frontend implementations with many new features
- Avoid side effects with rest of the API (intermediate images and containers)
- Prioritize your build cache for automatic pruning
----
##### **Containers**
----
- container：也是分層的，當執行一個 container 時，會在 image 的最上面加一層 R/W layer (可讀寫層)
container = image (layer) + R/W layer (container layer)
對正在執行的 container 所做的所有更改 (如寫入新，修改和刪除檔案) 都會寫入此可寫的 container layer
當 container 被刪除時，可寫層也會被刪除，但底層 image 保持不變
----
##### **Networks**
----
###### 1. Default Bridge Network
Containers on the default bridge network can only access each other by IP addresses, unless you use the `--link` option
----
----
docker 會新增一個 software bridge 作為 container 網路對外的出口，預設名稱為 docker0
docker0 會與 host 中的對外網卡(上圖為 eth0)相連，藉此取得對外連線的能力
每個 container 會使用一個 veth device 與 docker0 相連，因此具備連外能力
----
###### 2. user-defined Bridge Network
more isolated and flexible
----
```shell
$ docker network create --driver bridge my-bridge
```
```shell
$ docker run -d --name app3 --network my-bridge nginx:alpine
$ docker run -d --name app4 --network my-bridge nginx:alpine
$ docker run -d --name app2 nginx:alpine
```
----
Try to reach app4 from app3 using the hostname app4:
```shell
$ docker exec app3 curl app4 # succeess
```
Try to reach app3 from app4:
```shell
$ docker exec app4 curl app3 # succeess
```
Will we be able to reach app2 from app3 using DNS?
```shell
$ docker exec app3 curl app2 # failed
```
----
###### 3. Host Network
此模式的網路有以下特點：
- 不使用獨立的網路，而是與 Docker Host 使用相同的網路
- container 如果沒有對外開 port 提供服務，其實設定為 host mode 並沒有意義
----
網路架構如下圖：
----
##### 4. more but I didnt research
- overlay
- IPvlan
- Macblan
- None
----
##### **Volumes**
----
Volumes is managed by Docker
(located at `/var/lib/docker/volumes/` on linux)
----
###### 1. volumes
- **Create a volume:**
```shell
$ docker volume create my_volume
```
- **List volumes:**
```shell
$ docker volume ls
```
----
- **Run a container with a volume:**
This command runs a container and mounts the volume `my_volume` to the container at the path `/data`.
```shell
$ docker run -d --name my_container -v my_volume:/data my_image
```
- **Inspect a volume:**
```shell
$ docker volume inspect my_volume
```
- **Remove a volume:**
```shell
$ docker volume rm my_volume
```
----
###### 2. bind mounts
Bind mounts may be stored anywhere on the host system. They may even be important system files or directories.
----
- **Run a container with a bind mount:**
This command mounts the host directory `/path/on/host` to the container at the path `/path/in/container`.
```shell
$ docker run -d --name my_container -v /path/on/host:/path/in/container my_image
```
----
##### **`rw` vs. `ro` in docker volume**
- Read-Only (`ro`)
```shell
# volume:
$ docker run -d --name my_container -v my_volume:/data:ro my_image
# bind mounts:
$ docker run -d \
--name my_container \
-v /path/on/host:/path/in/container:ro \
my_image
```
- Read-Write (`rw`)
```shell
# volume:
$ docker run -d --name my_container -v my_volume:/data:rw my_image
# bind mounts:
$ docker run -d \
--name my_container \
-v /path/on/host:/path/in/container:rw \
my_image
```
----
#### **Docker(Container) Registries**
----
more registries...
---
## Installation and Configuration
----
refer to: https://docs.docker.com/engine/install/
----
----
----
### Play with Docker
如果你不想安裝 Docker，有另一種方式可以練習 Docker，那就是 Docker 官方提供的 [Play with Docker (PWD)](https://labs.play-with-docker.com/)
----
### Configuring Docker for your environment
After installing Docker, when you first run `docker info` to check Docker system information, you might encounter a permission error because the Docker daemon binds to a Unix socket owned by `root`. By default, non-root users need to prepend `sudo` to Docker commands.
----
#### Reason
The permission issue arises because the Docker daemon uses a Unix socket, not a TCP port, and by default, it's owned by `root`. To use Docker commands without `sudo`, you need to be part of the `docker` Unix group that has permissions to read/write to the Unix socket.
----
#### Solution
To avoid using `sudo` with every Docker command, create a `docker` group and add your user to it. This allows your user to run Docker commands without `sudo`. However, be aware that being part of the `docker` group gives privileges equivalent to the `root` user, which can affect system security.
----
#### Steps to add your user to the `docker` group:
----
1. Create the `docker` group.
```shell
$ sudo groupadd docker
```
2. Add your user to the `docker` group.
```shell
$ sudo usermod -aG docker $USER
```
3. Log out and log back in, or restart your VM to apply these changes.
4. Verify your user is added to the `docker` group.
```shell
$ id -nG
```
For users not logged in, use:
```shell
$ sudo usermod -a -G docker <username>
```
----
### Docker Desktop
---
## Basic Docker Operations
see: https://hackmd.io/@titangene/rk3zjKVIz
---
## Docker Compose
----
Compose is an orchestration tool that makes spinning up ==multi-container== distributed applications with Docker an effortless task. It is easy to define a multi-container application and accompanying services in one file, then spin it up in a ==single command== that will start and run your entire app.
----
----
### history
----
### Introduction to multi-container applications
----
- Dont want to create each container one by one?
- Dont want to configure network, volumes... with cli line by line?
----
### Writing and understanding docker-compose files
----
todo list continue...
----
`compose.yaml`
```yaml
services:
app:
image: node:18-alpine
command: sh -c "yarn install && yarn run dev"
ports:
- 127.0.0.1:3000:3000
working_dir: /app
volumes:
- ./:/app
environment:
MYSQL_HOST: mysql
MYSQL_USER: root
MYSQL_PASSWORD: secret
MYSQL_DB: todos
mysql:
image: mysql:8.0
volumes:
- todo-mysql-data:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: secret
MYSQL_DATABASE: todos
volumes:
todo-mysql-data:
```
----
### Running and managing composed applications
check this out: https://github.com/docker/awesome-compose
my codespace to demo (you cant access): https://github.com/codespaces/curly-spoon-vr6g6jp6qrr2p9vg
I will show the /flask-redis example
----
the website at port $8000$ should look like this:
and after you reload the page the number would increase
----
### Strategies for database and stateful application handling
----
#### use docker to compose a primary-replica postgres SQL
----
https://github.com/eremeykin/pg-primary-replica
https://github.com/peterxcli/pg-primary-replica
```yaml!
version: '3.8'
x-postgres-common:
&postgres-common
image: postgres:14-alpine
user: postgres
restart: always
healthcheck:
test: 'pg_isready -U user --dbname=postgres'
interval: 10s
timeout: 5s
retries: 5
services:
postgres_primary:
<<: *postgres-common
ports:
- 5432:5432
environment:
POSTGRES_USER: user
POSTGRES_DB: postgres
POSTGRES_PASSWORD: password
POSTGRES_HOST_AUTH_METHOD: "scram-sha-256\nhost replication all 0.0.0.0/0 md5"
POSTGRES_INITDB_ARGS: "--auth-host=scram-sha-256"
command: |
postgres
-c wal_level=replica
-c hot_standby=on
-c max_wal_senders=10
-c max_replication_slots=10
-c hot_standby_feedback=on
volumes:
- ./00_init.sql:/docker-entrypoint-initdb.d/00_init.sql
- pgdata_primary:/var/lib/postgresql/data
postgres_replica:
<<: *postgres-common
ports:
- 5433:5432
environment:
PGUSER: replicator
PGPASSWORD: replicator_password
command: |
bash -c "
until pg_basebackup --pgdata=/var/lib/postgresql/data -R --slot=replication_slot --host=postgres_primary --port=5432
do
echo 'Waiting for primary to connect...'
sleep 1s
done
echo 'Backup done, starting replica...'
chmod 0700 /var/lib/postgresql/data
postgres
"
depends_on:
- postgres_primary
volumes:
- pgdata_replica:/var/lib/postgresql/data
volumes:
pgdata_primary:
driver: local
pgdata_replica:
driver: local
```
----
test if we run successfully
```shell
$ psql postgres://user:password@localhost:5432/postgres -xc \
'select * from pg_replication_slots;'
-[ RECORD 1 ]-------+-----------------
slot_name | replication_slot
plugin |
slot_type | physical
datoid |
database |
temporary | f
active | t
active_pid | 60
xmin |
catalog_xmin |
restart_lsn | 0/3000148
confirmed_flush_lsn |
wal_status | reserved
safe_wal_size |
two_phase | f
```
---
## ~~Advanced~~ More Use Cases
----
### Github action build image and push to docker hub public container registry
----
dockerfile
https://github.com/peterxcli/basic-auth-gin/blob/main/Dockerfile
```dockerfile
#build stage
FROM golang:alpine AS builder
RUN apk add --no-cache git
WORKDIR /app
COPY . /app
RUN go build -o /app/main -v
#final stage
FROM alpine:latest
RUN apk --no-cache add ca-certificates
COPY --from=builder /app/main /app/main
COPY --from=builder /app/public /app/public
WORKDIR /app
ENTRYPOINT /app/main
LABEL Name=mongogo Version=0.0.1
EXPOSE 9000
```
----
github action
https://github.com/peterxcli/basic-auth-gin/blob/main/.github/workflows/ci.yml
```yaml
# This workflow will build a golang project
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go
name: ci
on: [push, pull_request]
env:
IMAGE_TAG: ${{ github.sha }}
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
mongodb-version: ["6.0"]
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: 1.19
- name: Start MongoDB
uses: supercharge/mongodb-github-action@1.8.0
with:
mongodb-version: ${{ matrix.mongodb-version }}
- name: Build
run: go build -v -buildvcs=false ./...
- name: Test
env:
EMAIL: ${{ secrets.EMAIL }}
SENDGRID_API_KEY: ${{ secrets.SENDGRID_API_KEY }}
run: make test
build:
needs: [test]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up QEMU
uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # v2.1.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Build and push
uses: docker/build-push-action@v4
with:
push: false
context: .
file: ./Dockerfile
tags: peter0814/basic-auth-gin:${{ github.sha }}
outputs: type=docker, dest=docker.tar
- name: docker login
env:
DOCKER_USER: ${{ secrets.DOCKER_USER }}
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
run: |
docker login -u $DOCKER_USER -p $DOCKER_PASSWORD
- name: cache
uses: actions/upload-artifact@83fd05a356d7e2593de66fc9913b3002723633cb # v3.1.1
with:
name: docker.tar
path: docker.tar
push:
if: ${{ github.ref == 'refs/heads/release' }}
needs : [build]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up QEMU
uses: docker/setup-qemu-action@e81a89b1732b9c48d79cd809d8d81d79c4647a18 # v2.1.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: docker login
env:
DOCKER_USER: ${{ secrets.DOCKER_USER }}
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
run: |
docker login -u $DOCKER_USER -p $DOCKER_PASSWORD
- name: Download artifact
uses: actions/download-artifact@v2
with:
name: docker.tar
path: ./
- name: Load Docker image
run: |
docker load --input docker.tar
docker image ls -a
- name: Docker Push
run: docker push peter0814/basic-auth-gin:$IMAGE_TAG
```
----
### Dev Container
a truely isolated and consisent development environment
https://github.com/peterxcli/basic-auth-gin/tree/main/.devcontainer
----
#### some integrated features in vscode
1. vscode extension as Code (VSCEAC)
---
## reference
### [Docker 系列筆記](https://hackmd.io/@titangene/docker-collection/)
Google Drive
Gist
Clipboard
Sign in with Wallet
