# :a: 新版 建議 ## 0. sudo 記得初次或重開機執行 都要先執行這一行 ```bash sudo ls ``` ## 1. 安裝 NIVIDA 驅動程式 (GPU) ```bash # 更新系統 sudo apt update && sudo apt upgrade -y # 移除所有nvidia套件 sudo apt autoremove nvidia* --purge -y # 安裝ubuntu nvudia 套件 sudo apt install ubuntu-drivers-common -y # 列出nvidia可安裝DRIVER版本 ubuntu-drivers devices ubuntu-drivers list # 並動更新, 若移除以下第二行之# 指定版本 sudo ubuntu-drivers autoinstall #sudo apt install nvidia-driver-525 # 列出顯卡硬體內容 sudo lshw -C display # 重開機 sudo reboot now ``` ## 2. 移除 Docker (GPU/CPU) (OPTION) ```bash! sudo apt-get remove docker docker-engine docker.io containerd runc sudo apt-get update docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 docker ps -q -a -f volume={} | xargs -r docker rm -f sudo apt-get purge -y nvidia-docker ``` ## 3. 安裝 Docker (GPU/CPU) ```bash sudo apt-get update -y sudo apt-get install php-cli zip jq apt-transport-https ca-certificates curl gnupg-agent software-properties-common -y curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable" -y sudo apt-get update -y sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y ``` ## 3. 安裝 Docker (GPU/CPU) docker 25 ``` sudo apt-get remove docker docker-engine docker.io containerd runc sudo apt-get update sudo apt-get install \ ca-certificates \ curl \ gnupg \ lsb-release sudo mkdir -p /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg \ | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \ https://download.docker.com/linux/ubuntu jammy stable" \ | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get install docker-ce=5:25.0.5-1~ubuntu.22.04~jammy \ docker-ce-cli=5:25.0.5-1~ubuntu.22.04~jammy \ containerd.io docker-buildx-plugin docker-compose-plugin ``` - 20.04 ``` echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu focal stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt update sudo apt install docker-ce=5:25.0.5-1~ubuntu.20.04~focal \ docker-ce-cli=5:25.0.5-1~ubuntu.20.04~focal \ containerd.io docker-buildx-plugin docker-compose-plugin ``` ### 4. Install the NVIDIA Container Toolkit. (GPU) - Configure the repository ```bash curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt-get update -y ``` - Install the NVIDIA Container Toolkit packages ```bash sudo apt-get install -y nvidia-container-toolkit ``` ### 5. Configure Docker to use Nvidia driver (GPU) ```bash sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker ``` ### 6. 測試是否成功 (GPU) ```bash sudo docker run --rm --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi ``` ### 7. 加入 Docker 群組, 記得你的身分為ubuntu (GPU/CPU) ``` sudo groupadd docker sudo usermod -aG docker $USER sudo systemctl enable docker # 啟動 Docker 服務 docker --version ``` ### 8. 測試安裝結果, 記得你的身分為ubuntu (GPU/CPU) ``` sudo su su ubuntu docker run hello-world ``` ### 9. Start the container for ollama (GPU), CPU 請移除 --gpus=all ```bash docker run -d --gpus=all -v ollama:/root/.ollama -p 443:11434 --name ollama ollama/ollama ``` ```bash docker run -d --gpus=all -v ollama:/root/.ollama -p 443:11434 -e OLLAMA_MAX_LOADED_MODELS=4 -e OLLAMA_NUM_PARALLEL=16 --name ollama ollama/ollama ``` 啟動 ollama 前面加上這個 OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4 ollama xxxxxxxxxxxxx ### 10. 模型下載 ```bash docker exec ollama ollama pull jcai/llama3-taide-lx-8b-chat-alpha1:Q4_K_M docker exec ollama ollama pull jcai/llama3-taide-lx-8b-chat-alpha1:f16 docker exec ollama ollama pull jcai/breeze-7b-32k-instruct-v1_0:f16 ``` ### 11. 驗測 ```bash curl http://localhost:443/api/generate -d '{"model": "jcai/breeze-7b-32k-instruct-v1_0:f16","prompt":"Why is the sky blue?"}' curl http://localhost:443/api/generate -d '{"model": "jcai/llama3-taide-lx-8b-chat-alpha1:f16","prompt":"Why is the sky blue?"}' curl http://localhost:443/api/generate -d '{"model": "jcai/llama3-taide-lx-8b-chat-alpha1:Q4_K_M","prompt":"Why is the sky blue?"}' curl http://localhost:443/api/chat -d '{"model": "jcai/breeze-7b-32k-instruct-v1_0:f16","messages": [{"role": "user", "content": "why is the sky blue?" }]}' curl http://localhost:443/api/chat -d '{"model": "jcai/llama3-taide-lx-8b-chat-alpha1:f16","messages": [{"role": "user", "content": "why is the sky blue?" }]}' curl http://localhost:443/api/chat -d '{"model": "jcai/llama3-taide-lx-8b-chat-alpha1:Q4_K_M","messages": [{"role": "user", "content": "why is the sky blue?" }]}' ``` ```bash curl http://203.145.222.186:443/api/generate -d '{"model": "jcai/breeze-7b-32k-instruct-v1_0:f16","prompt":"Why is the sky blue?"}' curl http://203.145.222.186:443/api/generate -d '{"model": "jcai/llama3-taide-lx-8b-chat-alpha1:f16","prompt":"Why is the sky blue?"}' curl http://203.145.222.186:443/api/generate -d '{"model": "jcai/llama3-taide-lx-8b-chat-alpha1:Q4_K_M","prompt":"Why is the sky blue?"}' curl http://203.145.222.186:443/api/chat -d '{"model": "jcai/breeze-7b-32k-instruct-v1_0:f16","messages": [{"role": "user", "content": "why is the sky blue?" }]}' curl http://203.145.222.186:443/api/chat -d '{"model": "jcai/llama3-taide-lx-8b-chat-alpha1:f16","messages": [{"role": "user", "content": "why is the sky blue?" }]}' curl http://203.145.222.186:443/api/chat -d '{"model": "jcai/llama3-taide-lx-8b-chat-alpha1:Q4_K_M","messages": [{"role": "user", "content": "why is the sky blue?" }]}' ```