# 從無到有建立Docker swarm運行自有服務 ###### tags: `Tag(ubuntu)` `Tag(Swarm)` `Tag(docker)` ## 本篇架構 將會依照以下流程建立服務 1. 建立三個node (1 Manage node, 2 Worker nodes) 2. 安裝docker 3. 建立swarm manager node 4. 建立swarm worker node 5. 在swarm內建立private registry (insecure registry) 6. push服務到private registry 7. 部署stack服務到swarm --- ## 1. 建立三個node (1 Manage node, 2 Worker nodes) 使用multipass建立三個VM作為node，並將主機分別命名為manager1, worker1, worker2 ``` dancyu@DancMacbook% multipass launch --name manager1 dancyu@DancMacbook% multipass launch --name worker1 dancyu@DancMacbook% multipass launch --name worker2 dancyu@DancMacbook% multipass list Name State IPv4 Image primary Stopped -- Ubuntu 20.04 LTS manager1 Running 192.168.64.6 Ubuntu 20.04 LTS 172.17.0.1 worker1 Running 192.168.64.11 Ubuntu 20.04 LTS 172.17.0.1 worker2 Running 192.168.64.12 Ubuntu 20.04 LTS ``` --- ## 2. 安裝docker - 參考[docker官網在ubuntu的安裝步驟](https://docs.docker.com/engine/install/ubuntu/),在三個VM上執行以下指令 ``` sudo apt upgrade sudo apt-get update sudo apt-get install -y ca-certificates curl gnupg lsb-release curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update sudo apt-get install -y docker-ce docker-ce-cli containerd.io sudo docker run hello-world ``` - unbuntu安裝docker之後，執行docker 指令需要透過sudo才能執行，不然會報錯，如下 ``` ubuntu@manager1$ docker run hello-world docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create": dial unix /var/run/docker.sock: connect: permission denied. See 'docker run --help'. ubuntu@manager1$ ls -l /var/run/docker.sock srw-rw---- 1 root docker 0 Nov 19 15:22 /var/run/docker.sock ``` * 這是因為The Docker daemon binds to a Unix socket instead of a TCP port. Unix Socket的擁有者是root, 因此得用sudo才能執行docker命令。 依照[Post-installation steps for Linux](https://docs.docker.com/engine/install/linux-postinstall/)的說明是Unix socket檔案的group 是docker, 把你執行的user加入docker group，但這種作法有風險性：可詳見[Docker Daemon Attack Surface.](https://docs.docker.com/engine/security/#docker-daemon-attack-surface) 因為docker群組的權限幾乎接近root，若隨意把帳號加入docker, 那可能做了不恰當的行為會充滿風險，不如透過sudo docker, 由docker指令控制可操作的範圍。 - 若不改使用者的權限，網路一般建議的解法是修改檔案權限 ``` sudo chmod 666 /var/run/docker.sock ``` - 我決定維持預設方式，透過sudo docker的方式操作。 --- ## 3. 建立swarm manager node 透過multipass shell manager1 ``` dancyu@DancMacbook ~ % multipass shell manager1 ubuntu@manager1:~% docker swarm init --advertise-addr 192.168.64.6 Swarm initialized: current node (o1jhsl5j50m8gjw0h4dkmq5ed) is now a manager. To add a worker to this swarm, run the following command: docker swarm join --token SWMTKN-1-2y0dgo8n3wqaxhhxv295a2qszx7o935606jfszhosmlejlmx1u-1isd58ymxwvnbrew3051jhnpq 192.168.64.6:2377 To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. ``` 已初始化swarm,透過回應可知增加worker 要執行 docker swarm join --token SWMTKN-1-2y0dgo8n3wqaxhhxv295a2qszx7o935606jfszhosmlejlmx1u-1isd58ymxwvnbrew3051jhnpq 192.168.64.6:2377 要增加manager 可執行 docker swarm join-token manager 用docker info看一下 ``` ubuntu@manager1:~$ sudo docker info Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., v0.9.1-beta3) buildx: Build with BuildKit (Docker Inc., v0.6.3-docker) scan: Docker Scan (Docker Inc., v0.9.0) Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 20.10.11 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: active NodeID: o1jhsl5j50m8gjw0h4dkmq5ed Is Manager: true ClusterID: 4hop4hfcubsj5fij9lljcf3lc Managers: 1 Nodes: 1 Default Address Pool: 10.0.0.0/8 SubnetSize: 24 Data Path Port: 4789 Orchestration: Task History Retention Limit: 5 Raft: Snapshot Interval: 10000 Number of Old Snapshots to Retain: 0 Heartbeat Tick: 1 Election Tick: 10 Dispatcher: Heartbeat Period: 5 seconds CA Configuration: Expiry Duration: 3 months Force Rotate: 0 Autolock Managers: false Root Rotation In Progress: false Node Address: 192.168.64.6 Manager Addresses: 192.168.64.6:2377 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc version: v1.0.2-0-g52b36a2 init version: de40ad0 Security Options: apparmor seccomp Profile: default ... ``` 使用docker node指令 ``` ubuntu@manager1:~$ sudo docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION o1jhsl5j50m8gjw0h4dkmq5ed * manager1 Ready Active Leader 20.10.11 ``` --- ## 4. 建立swarm worker node 登入worker1 ``` ubuntu@worker1:~$ sudo docker swarm join --token SWMTKN-1-2y0dgo8n3wqaxhhxv295a2qszx7o935606jfszhosmlejlmx1u-1isd58ymxwvnbrew3051jhnpq 192.168.64.6:2377 This node joined a swarm as a worker. ``` 到manager1來用docker node ls可以看到有產生第二個 ``` ubuntu@manager1:~$ sudo docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION o1jhsl5j50m8gjw0h4dkmq5ed * manager1 Ready Active Leader 20.10.11 z8k2c4rhpykernnwuvua08j1p worker1 Ready Active 20.10.11 ``` 一樣的指令，登入worker2再將worker2也加入swarm，透過manager1可以看到所有節點情形，MANAGER STATUS=Leader 就是manager ![](https://i.imgur.com/njYr52C.png) --- ## 5. 在swarm內建立registry 這裡以自建registry server的方式來部署服務。 5.1 先在mac本機建立目錄，把這個目錄mount到各個節點(每個VM)的/registrydisk ``` dancyu@DancMacbook% moultipass ~/opt/volume/swarm/registry manager1:/registrydisk dancyu@DancMacbook% moultipass ~/opt/volume/swarm/registry worker1:/registrydisk dancyu@DancMacbook% moultipass ~/opt/volume/swarm/registry worker2:/registrydisk ``` 5.2 在manager1上新增registry服務 ``` sudo docker service create \ --name registry \ --mount type=bind,src=/registrydisk,dst=/var/lib/registry \ --publish published=5000,target=5000 \ --replicas 1 \ registry:2 ``` - 説明：replicas 1 表示registry服務只會啟動一個container來執行，但不管這個container跑在哪一個node，它都能對應到/registrydisk使用同一份檔案空間維護registry資料。（不要起多個registry service) 5.3 透過docker service檢查服務 ![](https://i.imgur.com/SOSC359.png) 5.4 在本機上可指定三個ip來確認registry server有服務 ![](https://i.imgur.com/hfelBC3.png) --- ## 6. Push服務到private registry 延伸[這篇建立的SpringBootImage](https://hackmd.io/@dancyu/BJki-NzdF) image已先建立在本機 ![](https://i.imgur.com/YxSe6wM.png) 6.1 透過以下指令把image放到自建registry server ``` docker tag springboot_app_image:1 192.168.64.6:5000/springboot_app_image docker push 192.168.64.6:5000/springboot_app_image ``` 6.2 push時發現報錯，這是因為docker 預設用https做push ![](https://i.imgur.com/Tx3C6gf.png) 6.3 透過Docker Destop Preferences->Docker Engine修改設定，增加"insecure-registries":["192.168.64.6:5000"]後按Apply & Restart ![](https://i.imgur.com/nOASCZB.png) 6.4 本機的docker重啟完成後，再次執行指令終於能push成功了 ![](https://i.imgur.com/Z8tkcTC.png) --- ## 7. 本機驗證image 部署服務到swarm時，都吃docker-compose.yml檔 7.1 在部署上swarm前先測試，調整docker-compose.yml內的鏡像位置 ``` version: '3.7' services: web: #container_name: springboot_app build: . image: 192.168.64.6:5000/springboot_app_image ports: - 8080:8080 ``` 7.2 在本機執行docker-compose up -d啟動後，用docker ps可以看到IMAGE如預期的使用private registry server ![](https://i.imgur.com/IidP16r.png) 7.3 停止本機服務docker-compose down -volumes (順帶刪除container與有用到的) ## 8. 部署stack服務到swarm 8.1 登入到manager1 8.2 建立目錄deploystack/springdocker 並編輯docker-compose.yml ``` % mkdir -p deploystack/springdocker/ % cd deploystack/springdocker/ % vi docker-compose.yml (內容同上 docker-compose.yml) % sudo docker stack deploy --compose-file docker-compose.yml springdocker ``` ![](https://i.imgur.com/GV5NjKq.png) 8.3 用docker statck ls 檢查服務 ![](https://i.imgur.com/iH5UkD0.png) 8.3 用docker service檢查服務 ![](https://i.imgur.com/GOtyOJq.png) 8.4 用postman測試 - GET http://192.168.64.11:8080/products ![](https://i.imgur.com/bQ3pPSK.png) - POST http://192.168.64.12:8080/products ![](https://i.imgur.com/FjsDiKz.png) - GET http://192.168.64.11:8080/products?keywork=abc ![](https://i.imgur.com/a2PreUj.png) 任何一個node都可以查詢到這個服務，這是因為在create service時，docker預設配置overlay network的關係。 ![](https://i.imgur.com/ffO9ut8.png) - 在步驟8.3 docker service list中發現 worker2 無法找到鏡像 這是因為當docker把啟動container的工作分派給worker2時，worker2去拉鏡像還是預設走https，因此我們需要在manager1,worker1,worker2 上/etc/docker/daemon.json定義以下內容，若daemon.json不存在則新增一個。 > { "insecure-registries": ["192.168.64.6:5000"] } 編輯daemon.json後並重啟當台docker服務 ``` ubuntu@worker1:/etc/docker$ sudo vi daemon.json ubuntu@worker1:/etc/docker$ sudo systemctl restart docker ``` --- ## 9.擴展服務的數量 ``` sudo docker service scale springdocker_web=2 ``` ![](https://i.imgur.com/nbbwx6V.png) --- ## 10. 停止stack服務 停止 stack 的方式 ``` sudo docker stack rm springdocker ``` ![](https://i.imgur.com/LGb0TTV.png) --- ## 待深入研究 - private registry 權限控管 > [發現這篇](https://mileschou.github.io/ironman/12th/docker-newbie/day19/) ，後續再來試 - 為什麼 private registry 內的image沒有TAG? ![](https://i.imgur.com/wrhKqTG.png) > 上面的問題是錯的，我在manager1上操作docker image list 看到的是曾在manager1上下載的鏡像，而不是private registry的鏡像，若要看到private registry server上所建立的image，要使用API 去查找。也可以透過registry-web這個opensource方便用瀏覽器查看， ``` sudo docker service create \ --name registry-web \ --env REGISTRY_URL=http://192.168.64.6:5000/v2 \ --env REGISTRY_NAME=192.168.64.6:5000 \ --publish published=8090,target=8080 \ hyper/docker-registry-web ``` > 執行結果如下 ![](https://i.imgur.com/AkZmb91.png) > 用 http://192.168.64.6:8090開啟網頁，畫面如下，共有三個repository ![](https://i.imgur.com/Ru2gEQf.png) 點選第一個dancyu/springboot_app_image可看到這個repo內的各個Tag ![](https://i.imgur.com/8fL5VCG.png)