# 雲端通訊整合實務(12/1)
###### tags: `docker`
## 環境準備
虛擬機：3台
處理器：2核心
記憶體：2G
| IP Address | Hostname | CPU | Ram | OS |
|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
| 192.168.102.135 | vm1(master) | 2 | 4 | ubuntu(16.04) |
| 192.168.102.139 | vm2(worker) | 2 | 4 | ubuntu(16.04) |
| 192.168.102.140 | vm3(worker) | 2 | 4 | ubuntu(16.04) |
## kubernetes安裝(ubuntu)
**若是centos的用戶可以參照這一篇安裝k8s：**
https://blog.tomy168.com/2019/08/centos-76-kubernetes.html
安裝net-tools工具
```
apt-get update
apt-get install wget net-tools nano -y
```
更改虛擬機名稱
```
hostnamectl set-hostname vm1
hostnamectl set-hostname vm2
hostnamectl set-hostname vm3
```
編輯`/etc/hosts`，在最後一行增加三台虛擬機的IP以及名稱：
```
192.168.102.140 vm3
192.168.102.135 vm1
192.168.102.139 vm2
```
編輯完後，重啟一下網路並且測試用名稱互相ping：
```
systemctl restart NetworkManager
```
```
root@vm1:/home/user# ping vm1
PING vm1 (192.168.102.135) 56(84) bytes of data.
64 bytes from vm1 (192.168.102.135): icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from vm1 (192.168.102.135): icmp_seq=2 ttl=64 time=0.081 ms
^C
--- vm1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.063/0.072/0.081/0.009 ms
root@vm1:/home/user# ping vm2
PING vm2 (192.168.102.139) 56(84) bytes of data.
64 bytes from vm2 (192.168.102.139): icmp_seq=1 ttl=64 time=0.836 ms
64 bytes from vm2 (192.168.102.139): icmp_seq=2 ttl=64 time=0.885 ms
^C
--- vm2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.836/0.860/0.885/0.038 ms
root@vm1:/home/user# ping vm3
PING vm3 (192.168.102.140) 56(84) bytes of data.
64 bytes from vm3 (192.168.102.140): icmp_seq=1 ttl=64 time=0.866 ms
64 bytes from vm3 (192.168.102.140): icmp_seq=2 ttl=64 time=1.10 ms
^C
--- vm3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.866/0.983/1.100/0.117 ms
root@vm1:/home/user#
```
檢查firewalld以及SELinux是否關閉
```
systemctl stop firewalld
systemctl status firewalld
```
```
getenforce
sed -i 's@SELINUX=enforcing@SELINUX=disabled@' /etc/sysconfig/selinux
```
編輯`/etc/fstab`，並將swap那行註解掉
```
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda1 during installation
UUID=14bff259-e1db-4b35-b0a0-b79fb9c847a7 / ext4 errors=remount-ro 0 1
# swap was on /dev/sda5 during installation
#UUID=a3684131-294c-46ce-b9d0-a0a999665ad1 none swap sw 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
```
編輯完後重啟電腦`reboot`，重啟後使用`free`檢查swap的值是否都為0。
```
root@vm1:/home/user# free
total used free shared buff/cache available
Mem: 4028472 1445624 256292 59080 2326556 2116432
Swap: 0 0 0
```
將iptables相關功能或模組的啟用與停用
```
echo 1 > /proc/sys/net/ipv4/ip_forward
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
```
透過`sysctl -p`檢查上面指令是否成功輸入
```
root@vm1:/home/user# sysctl -p
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
```
將 br_netfilter模組特別寫入`/etc/modules-load.d`的目錄中，讓重開機後的作業系統能夠自動載入
```
modprobe br_netfilter
echo "br_netfilter" > /etc/modules-load.d/br_netfilter.conf
```
利用`lsmod`檢查br_netfilter模組是否成功載入
```
root@vm1:/home/user# lsmod | grep br_netfilter
br_netfilter 24576 0
bridge 122880 1 br_netfilter
```
若未安裝docker的用戶，可以使用以下指令安裝
```
apt-get update && apt-get install -y docker.io
```
安裝完後記得開啟docker，並且設定每次開機後自動開啟此功能
```
systemctl start docker
systemctl enable docker
```
安裝k8s相關套件(kubectl、kubeadm、kubelet)
```
apt-get update && apt-get isntall -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/
apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubectl kubeadm kubelet
```
初始化master
```
kubeadm init --apiserver-advertise-address=192.168.102.146 --pod-network-cidr=10.244.0.0/16
```
成功會出現這這樣的畫面：
```
(省略)
...
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.102.135:6443 --token 2hhif4.iarj1b27mtpmgu1f \
--discovery-token-ca-cert-hash sha256:0ced03c22bbf329282db8a301ee3f7c0e80720536cc35318ca7f26637a0d0e4a
```
其中需要先將k8s的安裝路徑設定好
```
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
安裝通用的 flannel 容器網路介面CNI介面
```
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```
接著就可以將token複製下來，並且貼到其他`VM(vm2、vm3)`上來建立worker
若成功後可以到master端進行查看，其結果會長這樣：
```
root@vm1:/etc/systemd/system/kubelet.service.d# kubectl get nodes
NAME STATUS ROLES AGE VERSION
vm1 Ready master 2d v1.19.4
vm2 Ready <none> 2d v1.19.4
vm3 Ready <none> 2d v1.19.4
```
> kubectl get nodes:查看節點狀況
接下來可以做個測驗來測試k8s是否可執行以及成功建立起網路的部署
```
kubectl create deployment myweb --image=httpd
kubectl expose deployment myweb --type="NodePort" --port=80
```
**創建一個部署，透過httpd的images，名稱叫做myweb**
> kubectl create deployment myweb --image=httpd
**為myweb的httpd部署提供80埠**
> kubectl expose deployment myweb --type="NodePort" --port=80
建立完後可以透過`kubectl get pods`和`kubectl get svc`查看pod和service
```
root@vm1:/etc/systemd/system/kubelet.service.d# kubectl get pods
NAME READY STATUS RESTARTS AGE
myweb-6f795ffc84-dccnx 1/1 Running 0 45h
```
```
root@vm1:/etc/systemd/system/kubelet.service.d# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d1h
myweb NodePort 10.110.178.159 <none> 80:31262/TCP 44h
```
> 你可以從svc看到，myweb的埠號為31262端口
因此你就可以進行httpd測試，去測試vm1～vm3的ip是否都能顯示`it works`
* 初始化 k8s：`kubeadm reset`
## k8s 基本指令
查詢節點`vm1`詳細資料
```
kubectl describe node [ master ]
```
```
root@vm1:~# kubectl describe node vm1
Name: vm1
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=vm1
kubernetes.io/os=linux
node-role.kubernetes.io/master=
Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"0e:42:91:4b:3d:a7"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 192.168.102.146
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 29 Dec 2020 00:53:09 -0800
Taints: node-role.kubernetes.io/master:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: vm1
AcquireTime: <unset>
RenewTime: Thu, 31 Dec 2020 08:24:44 -0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Tue, 29 Dec 2020 00:55:33 -0800 Tue, 29 Dec 2020 00:55:33 -0800 FlannelIsUp Flannel is running on this node
MemoryPressure False Thu, 31 Dec 2020 08:20:54 -0800 Tue, 29 Dec 2020 00:53:06 -0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 31 Dec 2020 08:20:54 -0800 Tue, 29 Dec 2020 00:53:06 -0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 31 Dec 2020 08:20:54 -0800 Tue, 29 Dec 2020 00:53:06 -0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 31 Dec 2020 08:20:54 -0800 Tue, 29 Dec 2020 00:53:28 -0800 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.102.146
Hostname: vm1
Capacity:
cpu: 2
ephemeral-storage: 101016992Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 4028476Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 93097259674
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3926076Ki
pods: 110
System Info:
Machine ID: 953160eec3fd473e8359d677ec90af53
System UUID: D3194D56-5150-0493-73C0-A9ECA9169305
Boot ID: 377ad6c1-abfb-4430-86cb-04f045faf5a5
Kernel Version: 4.4.0-197-generic
OS Image: Ubuntu 16.04.6 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.7
Kubelet Version: v1.19.4
Kube-Proxy Version: v1.19.4
PodCIDR: 10.244.0.0/24
PodCIDRs: 10.244.0.0/24
Non-terminated Pods: (8 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system etcd-vm1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2d7h
kube-system kube-apiserver-vm1 250m (12%) 0 (0%) 0 (0%) 0 (0%) 2d7h
kube-system kube-controller-manager-vm1 200m (10%) 0 (0%) 0 (0%) 0 (0%) 2d7h
kube-system kube-flannel-ds-fgxj7 100m (5%) 100m (5%) 50Mi (1%) 50Mi (1%) 2d7h
kube-system kube-proxy-wk2mm 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2d7h
kube-system kube-scheduler-vm1 100m (5%) 0 (0%) 0 (0%) 0 (0%) 2d7h
kubernetes-dashboard dashboard-metrics-scraper-7445d59dfd-8ktwd 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2d1h
kubernetes-dashboard kubernetes-dashboard-5bc88ff8f9-t8v25 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2d1h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 650m (32%) 100m (5%)
memory 50Mi (1%) 50Mi (1%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events: <none>
```
查詢`deployment`資訊
```
kubectl get deployment
```
若想要刪除`deployment`
```
root@vm1:~# kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
mysql-1609235295 0/1 1 0 2d6h
web1 1/1 1 1 2d6h
root@vm1:~# kubectl delete deployment mysql-1609235295
deployment.apps "mysql-1609235295" deleted
root@vm1:~# kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
web1 1/1 1 1 2d6h
```
若要更改副本數，可以使用`scale`
```
kubectl scale deployment web1 --replicas 2
```
```
root@vm1:~# kubectl scale deployment web1 --replicas 2
deployment. extenions/web1 scaled
root@vm1:~# kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
web1 2/2 2 2 2d6h
```
想查詢`web1`更多資訊可使用`describe`
```
kubectl describe deployment web1
```
```
root@vm1:~# kubectl describe deployment web1
Name: web1
Namespace: default
CreationTimestamp: Tue, 29 Dec 2020 01:50:56 -0800
Labels: app=web1
app.kubernetes.io/managed-by=Helm
Annotations: deployment.kubernetes.io/revision: 1
meta.helm.sh/release-name: abc-1609235456
meta.helm.sh/release-namespace: default
Selector: app=web1
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=web1
Containers:
httpd:
Image: httpd:2.4.46
Port: <none>
Host Port: <none>
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Progressing True NewReplicaSetAvailable
Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: web1-7469c97b99 (1/1 replicas created)
Events: <none>
```
查看`pods`資訊
```
root@vm1:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
web1-7469c97b99-d97gd 1/1 Running 0 174m
```
使用`describe`查看更多資訊
```
root@vm1:~# kubectl describe pods web1-7469c97b99-d97gd
Name: web1-7469c97b99-d97gd
Namespace: default
Priority: 0
Node: vm2/192.168.102.139
Start Time: Thu, 31 Dec 2020 06:04:41 -0800
Labels: app=web1
pod-template-hash=7469c97b99
Annotations: <none>
Status: Running
IP: 10.244.1.4
IPs:
IP: 10.244.1.4
Controlled By: ReplicaSet/web1-7469c97b99
Containers:
httpd:
Container ID: docker://35f137ca6577ebc6f4477e10d99f7692022bca067781e09d38064a80f30cb397
Image: httpd:2.4.46
Image ID: docker-pullable://httpd@sha256:a3a2886ec250194804974932eaf4a4ba2b77c4e7d551ddb63b01068bf70f4120
Port: <none>
Host Port: <none>
State: Running
Started: Thu, 31 Dec 2020 06:04:43 -0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-f9h8d (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-f9h8d:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-f9h8d
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
```
查看`web1`的`yaml`格式
```
root@vm1:~# kubectl get deployment web1 -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
meta.helm.sh/release-name: abc-1609235456
meta.helm.sh/release-namespace: default
creationTimestamp: "2020-12-29T09:50:56Z"
generation: 1
labels:
app: web1
app.kubernetes.io/managed-by: Helm
managedFields:
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:meta.helm.sh/release-name: {}
f:meta.helm.sh/release-namespace: {}
f:labels:
.: {}
f:app: {}
f:app.kubernetes.io/managed-by: {}
f:spec:
f:progressDeadlineSeconds: {}
f:replicas: {}
f:revisionHistoryLimit: {}
f:selector:
f:matchLabels:
.: {}
f:app: {}
f:strategy:
f:rollingUpdate:
.: {}
f:maxSurge: {}
f:maxUnavailable: {}
f:type: {}
f:template:
f:metadata:
f:labels:
.: {}
f:app: {}
f:spec:
f:containers:
k:{"name":"httpd"}:
.: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:dnsPolicy: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:terminationGracePeriodSeconds: {}
manager: Go-http-client
operation: Update
time: "2020-12-29T09:50:56Z"
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:deployment.kubernetes.io/revision: {}
f:status:
f:availableReplicas: {}
f:conditions:
.: {}
k:{"type":"Available"}:
.: {}
f:lastTransitionTime: {}
f:lastUpdateTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"Progressing"}:
.: {}
f:lastTransitionTime: {}
f:lastUpdateTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
f:observedGeneration: {}
f:readyReplicas: {}
f:replicas: {}
f:updatedReplicas: {}
manager: kube-controller-manager
operation: Update
time: "2020-12-31T14:04:44Z"
name: web1
namespace: default
resourceVersion: "48095"
selfLink: /apis/apps/v1/namespaces/default/deployments/web1
uid: d1133f74-8c6f-471c-899e-197e7905be39
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: web1
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: web1
spec:
containers:
- image: httpd:2.4.46
imagePullPolicy: IfNotPresent
name: httpd
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2020-12-29T09:50:56Z"
lastUpdateTime: "2020-12-29T09:50:58Z"
message: ReplicaSet "web1-7469c97b99" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: "2020-12-31T14:04:44Z"
lastUpdateTime: "2020-12-31T14:04:44Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
```
將`web1`輸出成`yaml`檔，並且編輯成`web2`
```
kubectl get deployment web1 -o yaml > myweb.yml
```
> 輸出格式：-o yaml
透過`yaml`腳本創建
```
root@vm1:~# kubectl apply -f myweb.yaml
deployment.apps/web2 created
```
當你創建好之後，可以查看`deployment`和`pods`是否存在新的名叫`web2`
```
root@vm1:~# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
web1 1/1 1 1 2d7h
web2 1/1 1 1 7s
```
查看`pods`是否運行在哪個節點上
```
root@vm1:~# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web1-7469c97b99-d97gd 1/1 Running 0 3h43m 10.244.1.4 vm2 <none> <none>
web2-58b88d6994-d58bt 1/1 Running 0 2m20s 10.244.2.5 vm3 <none> <none>
```
可使用`curl`測試你的服務是否成功執行
```
root@vm1:~# curl 127.0.0.1
<html><body><h1>It works!</h1></body></html>
```
## Reference
1. [上課影片](https://drive.google.com/drive/folders/1oM_ejAeSIhGbDGAVBDnVRv4DuoXaDwYw?usp=sharing)
2. https://blog.tomy168.com/2019/08/centos-76-kubernetes.html
3. https://kubernetes.io/docs/tutorials/kubernetes-basics/
Sign in with Wallet
Connect another wallet