[雲端] K8s / env / Minikube === ###### tags: `雲端 / K8s` ###### tags: `雲端`, `K8s`, `Minikube` <br> ![](https://i.imgur.com/vsCp0RX.png =20%x) ![](https://i.imgur.com/ADBIuT4.png =50%x) <br><br> [TOC] <br> ## 安裝 Minikube - Minikube 官網 https://minikube.sigs.k8s.io/docs/start/ ```bash= curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 sudo install minikube-linux-amd64 /usr/local/bin/minikube minikube start # minikube start --driver=docker ``` - 使用 1.20.2 (2021/05/20) - K8s 官網 https://v1-18.docs.kubernetes.io/docs/tasks/tools/install-minikube/ <br> ## [Driver: kvm](https://minikube.sigs.k8s.io/docs/drivers/kvm2/) - [How to fix KVM permission denied error on Ubuntu 18.04](https://blog.chirathr.com/android/ubuntu/2018/08/13/fix-avd-error-ubuntu-18-04/) ```bash $ virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking if device /dev/kvm exists : PASS QEMU: Checking if device /dev/kvm is accessible : FAIL (Check /dev/kvm is world writable or you are in a group that is allowed to access it) ... QEMU: Checking for device assignment IOMMU support : PASS QEMU: Checking if IOMMU is enabled by kernel : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments) ... ``` 查看 group permission，並添加 `$USER` 到 kvm group ``` $ cat /etc/group | grep -i kvm kvm:x:130: $ sudo adduser $USER kvm Adding user `diatango_lin' to group `kvm' ... Adding user diatango_lin to group kvm Done. $ cat /etc/group | grep -i kvm kvm:x:130:diatango_lin $ virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking if device /dev/kvm exists : PASS QEMU: Checking if device /dev/kvm is accessible : FAIL (Check /dev/kvm is world writable or you are in a group that is allowed to access it) ``` ```bash # logout, than login again $ logout ``` ``` $ cat /etc/group | grep -i kvm kvm:x:130:diatango_lin $ virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking if device /dev/kvm exists : PASS QEMU: Checking if device /dev/kvm is accessible : PASS <--- ``` <br> - [[minikube] NVIDIA GPU Support](https://minikube.sigs.k8s.io/docs/tutorials/nvidia_gpu/) ``` $ minikube start --ports="10.78.26.241::30080" --ports="10.78.26.241::30443" --driver=kvm2 --kvm-gpu 😄 minikube v1.16.0 on Ubuntu 16.04 ✨ Using the kvm2 driver based on user configuration 👍 Starting control plane node minikube in cluster minikube 🔥 Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ... 🤦 StartHost failed, but will try again: creating host: create: Error creating machine: Error in driver during machine creation: creating devices: couldn't generate devices XML: no IOMMU groups found at "/sys/kernel/iommu_groups/". Make sure your host supports IOMMU. See instructions at https://minikube.sigs.k8s.io/docs/tutorials/nvidia_gpu/ 🔄 Restarting existing kvm2 VM for "minikube" ... 😿 Failed to start kvm2 VM. Running "minikube delete" may fix it: driver start: getting connection: looking up domain: virError(Code=42, Domain=10, Message='Domain not found: no domain with matching name 'minikube'') ❌ Exiting due to GUEST_KVM2_NO_DOMAIN: Failed to start host: driver start: getting connection: looking up domain: virError(Code=42, Domain=10, Message='Domain not found: no domain with matching name 'minikube'') 💡 Suggestion: The VM that minikube is configured for no longer exists. Run 'minikube delete' 🍿 Related issue: https://github.com/kubernetes/minikube/issues/3636 ``` - 😿 Failed to start kvm2 VM. - no IOMMU groups found at "/sys/kernel/iommu_groups/". - **Make sure your host supports IOMMU.** - [How do I enable KVM device passthrough in Linux?](https://serverfault.com/questions/633183/how-do-i-enable-kvm-device-passthrough-in-linux) ```bash $ nano /etc/default/grub ... GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on" ... ``` - [Enable IOMMU or VT-d in your motherboard BIOS](https://us.informatiweb.net/tutorials/it/bios/enable-iommu-or-vt-d-in-your-bios.html) ![](https://i.imgur.com/M5v7DEH.png) - If you have an Intel CPU, the IOMMU technology is called "Intel VT-d" - 導向式I/O 的Intel® 虛擬化技術I/O (VT-d) (Intel® Virtualization Technology for Directed I/O (VT-d)) <br> ## 指令用法 ### minikube addons - [Using a Private Registry ](https://minikube.sigs.k8s.io/docs/handbook/registry/) - [[K8s] Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) - [[minikube] Documentation / Handbook / Registries](https://minikube.sigs.k8s.io/docs/handbook/registry/) <br> ### minikube ip ```bash $ minikube ip 192.168.49.2 ``` <br> ### minikube profile list ```bash $ minikube profile list |----------|-----------|---------|--------------|------|---------|---------|-------| | Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes | |----------|-----------|---------|--------------|------|---------|---------|-------| | minikube | docker | docker | 192.168.49.2 | 8443 | v1.20.0 | Running | 1 | |----------|-----------|---------|--------------|------|---------|---------|-------| ``` <br> ### minikube ssh ```bash $ minikube ssh docker@minikube:~$ ``` <br> ### minikube start #### 指定底層的 VM - [driver: docker](https://minikube.sigs.k8s.io/docs/drivers/docker/) ``` $ minikube start --driver=docker ``` #### minikube 對外開放 30080 與 30090 - minikube start ``` $ minikube start --ports="10.78.26.241:30080:30080" --ports="10.78.26.241::30090" ``` - ```10.78.26.241:30080:30080``` - host: 30080 - minikube: 30080 - ```10.78.26.241::30090``` - host: 隨機分配 - minikube: 30090 - 產生的 docker run 指令 ``` docker run -d -t \ --privileged \ --security-opt seccomp=unconfined \ --tmpfs /tmp \ --tmpfs /run \ -v /lib/modules:/lib/modules:ro \ --hostname minikube \ --name minikube \ --label created_by.minikube.sigs.k8s.io=true \ --label name.minikube.sigs.k8s.io=minikube \ --label role.minikube.sigs.k8s.io= \ --label mode.minikube.sigs.k8s.io=minikube \ --network minikube \ --ip 192.168.49.2 \ --volume minikube:/var \ --security-opt apparmor=unconfined \ --memory=3900mb \ --memory-swap=3900mb \ --cpus=2 \ -e container=docker \ --expose 8443 \ -p 5000:5000 \ --publish=127.0.0.1::8443 \ --publish=127.0.0.1::22 \ --publish=127.0.0.1::2376 \ --publish=127.0.0.1::5000 \ gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4@sha256:ef1f485b5a1cfa4c989bc05e153f0a8525968ec999e242efff871cbb31649c16: ``` - port 22: ssh - port 2376: docker - port 5000: registry - port 8443: kubernetes - 跑一個 NodePort service 測試 ```tj-bluewhale.yaml``` ```yaml # tj-bluewhale-pod.yaml apiVersion: v1 kind: Pod metadata: name: tj-bluewhale-pod labels: app: webserver spec: containers: - name: tj-bluewhale-container image: hcwxd/blue-whale ports: - containerPort: 3000 --- # tj-bluewhale-service.yaml apiVersion: v1 kind: Service metadata: name: tj-bluewhale-service spec: type: NodePort ports: - port: 80 nodePort: 30080 targetPort: 3000 selector: app: webserver ``` 佈署到 K8s ``` $ kubectl apply -f tj-bluewhale.yaml ``` 連線測試 ```bash $ kubectl get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR tj-bluewhale-service NodePort 10.97.150.85 <none> 80:30080/TCP 19s app=webserver $ minikube ip 192.168.49.2 ``` ```bash $ curl 192.168.49.2:30080 ``` ``` $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5a0154b8f0f4 gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4 "/usr/local/bin/entr…" 2 minutes ago Up 2 minutes 10.78.26.241:30080->30080/tcp, 127.0.0.1:32775->22/tcp, 127.0.0.1:32774->2376/tcp, 127.0.0.1:32773->5000/tcp, 127.0.0.1:32772->8443/tcp, 10.78.26.241:32772->30090/tcp minikube ``` - ```10.78.26.241:30080->30080/tcp``` - 30080 為指定的主機 port - ```10.78.26.241:32772->30090/tcp``` - 32772 隨機配得 - 主機的 port 32772，映射到 minikube 的 port 30090 - 測試主機 port 30080 http://10.78.26.241:30080/ #### minikube 對外開放 5000 (registry) - [寫在 stackoverflow](https://stackoverflow.com/a/65589218/4359712) #### 常用啟動指令(可以work?) ```bash= IP=127.0.0.1 minikube start --driver=docker \ --ports=[ \ "$IP:5000:5000", \ "$IP:8080:8080", \ "$IP:8888:8888", \ "$IP:9090:9090", \ "$IP:30080:30080", \ "$IP:38080:38080"] ``` #### 參考資料 - [Minikube docker driver: customize exposed ports #8398](https://github.com/kubernetes/minikube/issues/8398) <br> ### docker cp #### 複製檔案到 minikube ```bash $ docker ps -a CONTAINER ID IMAGE ... de5e5b2bb78b gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4 ... $ docker cp your_file de5e5b2bb78b:/home/docker/ ``` <br> ## config ### 產生 kubeconfig 給 container 使用 - ### 手動導出 kubeconfig 並帶入 container： ```bash # 重新導出 kubeconfig（指向有效的 cert 路徑） minikube update-context minikube config view minikube config get profile minikube config set embed-certs true # 導出完整的 config minikube kubeconfig > /tmp/mykubeconfig ``` - ### 接著啟動 container 時掛進去： ```bash docker run -it \ -v /tmp/mykubeconfig:/root/.kube/config \ my-kubectl-image ```