雲端 HW5
===
https://hackmd.io/s/rk-RmWwez
https://hackmd.io/IwVgHAnATCDMCmBaEAjAZiRAWeB2TAhsDIvGivBKGgMbkpA=?view
###### tags: `cloud`

## Demo 時會確認三部份
- swarm-manager 上看到佈署至 worker 的 services
- 能開啟 jupyter 執行測試是否使用 GPU的程式
- 透過 REST API 得到回覆

# 理解 docker swarm

# 實驗建議步驟
## 2. 在上述兩機器上安裝 docker and docker swarm
> Done

1. Install docker:

$ curl -sSL https://get.docker.com/ | sh
$ sudo usermod -aG docker username
$ docker version

2. Install docker Compose:

$ sudo curl -L https://github.com/docker/compose/releases/download/1.17.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ docker-compose --version

ref: https://docs.docker.com/compose/install/#install-compose
3. Install docker-machine

$ curl -L https://github.com/docker/machine/releases/download/v0.13.0/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && \
 chmod +x /tmp/docker-machine && \
 sudo cp /tmp/docker-machine /usr/local/bin/docker-machine

## 3. 先於本機端建立 worker 將使用的 service image (CUDA、Nvidia driver、Deep learning framework、Jupyter).
> Done

在 docker-swarm-node 安裝
ref: https://medium.com/google-cloud/jupyter-tensorflow-nvidia-gpu-docker-google-compute-engine-4a146f085f17
1. sh cuda.sh

#!/bin/bash
echo "Checking for CUDA and installing."
# Check for CUDA and try to install.
if ! dpkg-query -W cuda-8-0; then
  # The 16.04 installer works with 16.10.
  curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
  dpkg -i ./cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
  apt-get update
  apt-get install cuda-8-0 -y
fi

2. check via

$ nvidia-smi
Thu Nov 30 13:17:24 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90                 Driver Version: 384.90                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   31C    P0    65W / 149W |      0MiB / 11439MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

3. Install Docker(-Engine)

#/bin/bash
# install packages to allow apt to use a repository over HTTPS:
sudo apt-get -y install \
apt-transport-https ca-certificates curl software-properties-common
# add Docker’s official GPG key:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - 
# set up the Docker stable repository.
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable-17.09"
# update the apt package index:
sudo apt-get -y update
# finally, install docker
sudo apt-get -y install docker-ce=17.09.0~ce-0~ubuntu

4. install nvidia-docker

# Add the package repositories
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \
  sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/nvidia-docker.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update

# Install nvidia-docker2 and reload the Docker daemon configuration
sudo apt-get install -y nvidia-docker2
sudo pkill -SIGHUP dockerd

5. Test nvidia-smi with the latest official CUDA image

$ sudo docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi

Unable to find image 'nvidia/cuda:latest' locally
latest: Pulling from nvidia/cuda
660c48dd555d: Pull complete 
4c7380416e78: Pull complete 
421e436b5f80: Pull complete 
e4ce6c3651b3: Pull complete 
be588e74bd34: Pull complete 
f597507b3c37: Pull complete 
9c5d4127a23d: Pull complete 
398bf259fcdc: Pull complete 
4f4092762618: Pull complete 
94130a21e154: Pull complete 
Digest: sha256:954c82d2d060f38de13b3d7933b7e1549b25330cc6412008dc1253f3c148448d
Status: Downloaded newer image for nvidia/cuda:latest
Thu Nov 30 14:04:25 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90                 Driver Version: 384.90                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   32C    P0    60W / 149W |      0MiB / 11439MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

## 4. 作業將進行 image classification ，可使用任何 deep learning framework，範例採用 Keras 與 Tensorflow.
> Done

1. Launch Jupyter and Tensorboard

sudo nvidia-docker run --rm --name tf1 -p 8888:8888 -p 6006:6006 gcr.io/tensorflow/tensorflow:latest-gpu jupyter notebook --allow-root

2. enter worker

docker exec -it tf1 bash

3. `pip install keras`
4. `vim ~/.keras/keras.json`
5. add `"image_dim_ordering": "tf"`

## 5. 可使用任何 model,範例採用 RestNet 50.
> Done

https://zhuanlan.zhihu.com/p/31502877

## 6. 利用 docker swarm manager，將 image deploy 至 worker 上，記得打開對外的 port 給 jupyter 與 REST API server 使用.

## 7. 確認可以透過 REST API 傳送照片至 worker container進行辨識並得到回覆。
> Done

curl -X POST -H "Content-Type: multipart/form-data" -F file=@$PWD/test.jpg http://yourip:port/upload

pip install Flask

sudo nvidia-docker run --rm --name tf1 -p 8888:8888 -p 6006:6006 -p 5000:5000 sufuf3/cloud2017:v1 jupyter notebook --allow-root

docker exec -it tf1 bash

cd APIserver
pip install Flask
pip install Flask-API
cd ~/APIserver && FLASK_APP=app.py flask run --host=0.0.0.0

先啟動 Jupi

## 8. 確認可以透過 Jupyter 運作程式，並請撰寫一支能顯示 framework 所使用硬體的程式，要能看到 GPU，範例採用 tensorflow。
> Done

HW5.ipynb

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

sudo nvidia-docker run --rm --name tf2 -p 8888:8888 -p 6006:6006 -p 5000:5000 sufuf3/cloud2017:v4 jupyter notebook --allow-root

## 剩下的 docker swarm manager
ref: https://kairen.github.io/2016/11/16/container/docker-swarm/
### admin VM

$ docker swarm init --advertise-addr 10.140.0.3
Swarm initialized: current node (xh68ponh5jksu89k1ncq6pgn0) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-4naxypl0675r84xmy7pcy75um0axd64179k95fby0nj6fnwdfe-6y9a5l637z0s8o1ofpn33zhie 10.140.0.3:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

$ docker network create --driver=overlay --attachable cnblogs
mb2f6vjhc8e18sd2giw2sg4z3

$ docker network create --driver overlay proxy
q3o144gqh6703b4bmtyew3eoo

### Node VM

$ docker swarm join --token SWMTKN-1-4naxypl0675r84xmy7pcy75um0axd64179k95fby0nj6fnwdfe-6y9a5l637z0s8o1ofpn33zhie 10.140.0.3:2377                                                                
This node joined a swarm as a worker.

### admin VM

$ docker node ls
ID                            HOSTNAME             STATUS              AVAILABILITY        MANAGER STATUS                                               
xh68ponh5jksu89k1ncq6pgn0 *   docker-swarm-admin   Ready               Active              Leader                                                       
jm4eq7bv141zyrglcslmg40ry     docker-swarm-node    Ready               Active       

docker service create --name tf3 -p 8888:8888 -p 6006:6006 -p 5000:5000 sufuf3/cloud2017:v4 jupyter notebook --allow-root                    

## 目前狀況
Admin: `docker service create --name tf1 -p 8888:8888 -p 6006:6006 -p 5000:5000 sufuf3/cloud2017:v4 jupyter notebook --allow-root`
Node:

$ docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                     
     NAMES
7c293fa922ef        sufuf3/cloud2017:v4   "jupyter notebook ..."   51 seconds ago      Up 47 seconds       5000/tcp, 6006/tcp, 8888/tcp   tf3.1.jf892bntvgddpk82nx2hiio7w

但如果是原本 node 自己執行

$ docker ps
CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS              PORTS                                            NAMES
99c701d403af        gcr.io/tensorflow/tensorflow:latest-gpu   "jupyter notebook ..."   22 seconds ago      Up 19 seconds       0.0.0.0:6006->6006/tcp, 0.0.0.0:8888->8888/tcp   tf1

port 的地方不一樣～～

待參考：https://rominirani.com/docker-swarm-on-google-compute-engine-364765b400ed

https://docs.docker.com/engine/swarm/services/#connect-the-service-to-an-overlay-network
https://blog.codeship.com/running-services-within-docker-swarm/

correct one!
docker service create --name tf1 --publish mode=host,target=8888,published=8888 --publish mode=host,target=5000,published=5000 --publish mode=host,target=6006,published=6006 sufuf3/cloud2017:v4 jupyter notebook --allow-root
from http://columns.chicken-house.net/2017/07/25/wc-swarm-labs/