---
# System prepended metadata

title: Docker & k8s
tags: [Software Engineering]

---

# Docker
## Installation
[Docker](https://docs.docker.com/desktop/install/mac-install)



## Creating the Container

### Check Version & Status
``` bash!
docker version
```

``` bash!
docker info
```

### Image vs. Container
![](https://hackmd.io/_uploads/SywbMOLj3.png)


### Starting a Nginx Web Server
#### Run Container
```bash!
docker container run --publish 80:80 nginx
```

#### Show Container
```bash!
docker container ls
```
![](https://hackmd.io/_uploads/ByTYwO8in.png)

#### Stop Container
```bash!
docker container stop 2020 # container id
```

#### List Container

```bash!
docker ps
```

#### What's Going on in Container
```bash!
docker container top
docker container stats
docker container inspect
```

### Getting a Shell Inside Containers


```bash!
docker container run -it
docker container exec -it
```

### Instance
Installation
```bash!
sudo apt-get update
sudo apt-get install docker.io -y
```

Ignore sudo
``` bash!
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker 
```

Build Image
```bash!
sudo docker build -t ass1:kao .
```

Run Container
```bash!
sudo docker run -p 80:80 --name ass1_kao -d ass1:kao
```

## Docker Networks
### Default
- Each container connected to a private virtual network "bridge"
- Each virtual network routes through NAT firewall on host IP
- All containers on a cirtual network can talk to each other without '-p'
- Best practice is to create a new virtual network for each app:
    - Network "my_web_app" for mysql and php/apache containers
    - Network "my_api" for mango and nodeks containers

```bash!
docker container run -p 80:80 --name webhost -d nginx
docker container port webhost
docker container inspect --format '{{.NetworkSettings.IPAddress}}' webhost
```

### CLI Management
```bash!
docker network docker network ls # show networks
docker network inspect # inspect a network
docker network create --drive # create a network
docker network connecct # attach a network to container
docker network disconnecct # detach a network from container
```

### DNS
- Containers shouldn't rely on IP's for inter-communication
- DNS for friendly names is built-in if you use custom networks

## Docker Image
### What's In An Image
- Metadata about image data and how to run the image
- Not a complete OS. No Kernel, Kernel modules
- Small as one file (your app binary)
- Big as a Ubuntu


### Dockerfile
```bash!
# use this empty Dockerfile to build your assignment

# NOTE: The instructions are now in the README.md file in this directory.


FROM node:6-alpine

EXPOSE 3000

RUN apk add --update tini
RUN mkdir -p /usr/src/app

WORKDIR /usr/src/app
COPY package.json package.json 

RUN npm install && npm cache clean

COPY . .

CMD [ "tini", "--", "node", "./bin/www" ]
```

## Persistent Data
### Container Lifetime & Persistent Data
- Containers are immutable: Only re-deploy the containers, never change
- This is an ideal scenario, but what about database??
- Docker give us features to ensure this concern
- This is known as 'persistent data'
- Two ways: Volumes and Bind Mounts
    - Volumes: make special location outside of container
    - Bind Mounts: link container path to host path


### Volume
> Voulme would not be removed even if we remove the container!!!

Install & deploy mysql

```bash!
docker pull mysql
docker inspect mysql
docker container run -d --name mysql -e MYSQL_ALLOW_EMPTY_PASSWORD=True mysql
```
![](https://hackmd.io/_uploads/SksrhFr62.png)

Check Volume


```bash!
docker volume ls
```

Name Volume
```bash!
docker container run -d --name mysql -e MYSQL_ALLOW_EMPTY_PASSWORD=True -v mysql_db:/var/lib/mysql mysql
docker volume ls
```
![](https://hackmd.io/_uploads/HkyA3KSTn.png)

### Bind Mounting
- Maps a host file or directory to a container file or directory
- Two locations point to the same files
- Can't use in Dockerfile, must be at container run

## Docker Compose
### Why
- Configure relationships between containers
- Save our docker container run settings in easy-to-read file
- Create one-liner developer environment startups

### Installation
```bash!
sudo curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

sudo chmod +x /usr/local/bin/docker-compose

docker-compose version

# example of running
docker-compose -f docker-compose.yml up -d
```


### Run Sample
```bash!
services:
  drupal:
    image: drupal
    ports:
      - "8080:80"
    volumes:
      - drupal-modules:/var/www/html/modules 
      - drupal-profiles:/var/www/html/profiles 
      - drupal-sites:/var/www/html/sites 
      - drupal-themes:/var/www/html/themes 
      
  postgres:
    image: postgres
    environment:
      - POSTGRES_PASSWORD=mypasswd

volumes:
  drupal-modules:
  drupal-profiles:
  drupal-sites:
  drupal-themes:

```

### Using Compose to Build
- Compose can also build your custom images
- Will build them with docker-compose up if not found in cache
- Also rebuild with docker-compose build
- Great for complex builds that have lots of vars or build args



```bash!
services:
  proxy:
    build:
      context: .
      dockerfile: nginx.Dockerfile
    ports:
      - '80:80'
  web:
    image: httpd
    volumes:
      - ./html:/usr/local/apache2/htdocs/
```

### Remove compose 

``` bash!
docker-compose down --rmi all --volumes

docker-compose -f docker-compose-prac7.yml down --volumes

```


## Swarm



![](https://hackmd.io/_uploads/ByfCMUBJa.png)

### What Just Happened?

``` bash!
docker swarm init
docker node ls
```

![](https://hackmd.io/_uploads/rJhm7LBJa.png)

### Docker Service Command

Service in Swarn **replace** Docker run
``` bash!
docker service create alpine ping 8.8.8.8
```
Show the running service & services in the service

``` bash!
docker service ls
docker service ps musing_lalande
```

![](https://hackmd.io/_uploads/rJJevIBka.png)

```bash!
docker service update dfoohcboqep6 --replicas 3
```


![](https://hackmd.io/_uploads/rk8etISyp.png)
![](https://hackmd.io/_uploads/S1u-Y8Hka.png)

```bash!
docker service rm musing_lalande
```

![](https://hackmd.io/_uploads/rk69YISJa.png)



## 3-Node Swarm

### Connect with other nodes
On master node run following command to obtain token.
```bash!
docker swarm init

docker swarm join-token manager
```
![](https://hackmd.io/_uploads/BknUDjDZ6.png)

Put the token into another node. The other nodes cannot access the node information because they are not the `Leader` node.


```bash!
docker swarm join --token SWMTKN-1 0942yqaq8avwbq66ae4o1eqair32z7bg96hcw9ktkkl9cp7tos-5qnsc882i6xuoubt0hkcaeidt 10.152.0.15:2377
```
![](https://hackmd.io/_uploads/SyucPiwZa.png)

### Assign leader node

```bash!
docker node update --role manager kao-2
```

![](https://hackmd.io/_uploads/rkl4_jDba.png)

### Run node

![](https://hackmd.io/_uploads/BJf3Kjv-6.png)

## Overlay Multi-Host Networking
1. Just choose `--driver overlay` when creating network.
2. Only for container-to-container inside a **single** swarm.
3. Each service can be connected to multiple networks.

```bash!
docker network create --driver overlay mydrupal
```
![](https://hackmd.io/_uploads/SkCp3ovbp.png)

### Create service

```bash!
docker service create --name psql --network mydrupal -e POSTGRES_PASSWORD=mypass postgres

docker service create --name drupal --network mydrupal -p 80:80 drupal

watch docker service ls
```

## Stacks: production grade compose
Use `docker stack deploy` to deploy rather than `docker create`.

[Code](https://github.com/BretFisher/udemy-docker-mastery/tree/main/swarm-stack-1)

```yaml!
# this file is meant for Docker Swarm stacks only
# trying it in compose will fail because of multiple replicas trying to bind to the same port
# Swarm currently does not support Compose Spec, so we'll pin to the older version 3.9

version: "3.9"

services:

  redis:
    image: redis:alpine
    networks:
      - frontend

  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_USER: "postgres"
      POSTGRES_PASSWORD: "postgres"
    volumes:
      - db-data:/var/lib/postgresql/data
    networks:
      - backend

  vote:
    image: dockersamples/examplevotingapp_vote
    ports:
      - 5000:80
    networks:
      - frontend
    deploy:
      replicas: 2

  result:
    image: dockersamples/examplevotingapp_result
    ports:
      - 5001:80
    networks:
      - backend

  worker:
    image: dockersamples/examplevotingapp_worker
    networks:
      - frontend
      - backend
    deploy:
      replicas: 2

networks:
  frontend:
  backend:

volumes:
  db-data:
```
### Deploy
Use `docker stack deploy` to spin it up!

``` bash!
docker stack deploy -c example-voting-app-stack.yml voteapp
```

### Get information

``` bash!
docker stack ps voteapp
```

![](https://hackmd.io/_uploads/SJiYxOt-a.png)

``` bash!
docker container ls
```
![](https://hackmd.io/_uploads/B1TogdtZp.png)

``` bash!
docker stack services voteapp
```

![](https://hackmd.io/_uploads/SJcCgutZT.png)

## Visualisation Tool

``` bash!
visualizer:
    image: bretfisher/visualizer
    ports:
      - 8080:8080
    stop_grace_period: 1m30s
    networks:
      - frontend
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      placement:
        constraints: [node.role == manager]
```

![](https://hackmd.io/_uploads/Hy5-dGc-6.png)





## Assignment

![](https://hackmd.io/_uploads/HJQvj3Pba.png)

1. voting-app : Front-end of the application written in python flask framework, which allows the users to cast their votes.
2. redis : An in-memory data structure store, used as a temporary database to store the votes casted by the users.
3. worker : service written in .NET, that retrieves the votes data from redis and stores it into PostgreSQL DB service.
4. postgreSQL DB : PostgreSQL DB used as persistent storage database.
5. result-app : service written in node js, displays the voting results to the user.

### Non-compose version

#### Create internet 

``` bash!
docker network create --driver overlay frontend
docker network create --driver overlay backend
```
#### Create service 
``` bash!
docker service create --name vote --replicas 2 --network frontend -p 80:80 dockersamples/examplevotingapp_vote:before

docker service create --name redis --replicas 2 --network frontend redis:3.2

docker service create --name worker --replicas 1 --network backend dockersamples/examplevotingapp_worker

docker service create --name db --network backend -e POSTGRES_HOST_AUTH_METHOD=trust --mount type=volume,source=db-data,target=/var/lib/postgresql/data postgres:9.4

docker service create --name result --network backend -p 5001:80 dockersamples/examplevotingapp_result:before

```
[Failed to build db](https://stackoverflow.com/questions/67770835/detected-task-failure-docker-service-create-name-db-network-backend-mount)

### Debug

```bash!
docker service logs worker
```


# Hadoop
## Commands

![](https://hackmd.io/_uploads/SJFwmiqba.png)

Run `docker exec -it cc4f bash` to log in the namenode container

``` bash!
docker exec -it cc4f bash

echo "Hello, HDFS!" > $HOME/hdfs_demo.txt

hdfs dfs -put $HOME/hdfs_demo.txt /hdfs_demo/hdfs_demo.txt

hdfs dfs -copyFromLocal /path 1 /path 2 .... /path n /destination

```



# Kubernetes
## Architecture
### Borg
Borg use `cell` to define a series of machine.
`Cluster` includes multiple `cells`.
![](https://hackmd.io/_uploads/SyI5NOGga.png)


### Kubernetes
k8s is consisted of masters and nodes.
![](https://hackmd.io/_uploads/rk1brdzx6.png)
#### Master
1. kube-apiserver
Entry of API. Conducting the operations for database and nodes.

2. kube-controller-manager
Maintaining the kubernetes clusters, such as updating or creating.

3. kube-scheduler
Linsting the unused PODS.

#### Nodes
1. Creating and maintaining image.
2. Maintaining POD lifecycle

## Basic Terms
![](https://hackmd.io/_uploads/HkiQCE-xp.png)

## k8s run, create, and apply

### Check version
``` bash!
kubectl version
kubectl version --short
```
![](https://hackmd.io/_uploads/SJFkQr-x6.png)


### Run
Pod is not really running, k8s only store the request in this step.
``` bash!
kubectl run [pod-name] --image [image-name]
kubectl run my-nginx --image nginx
```
### Inpect
``` bash!
kubectl get pods
kubectl get all
```
![](https://hackmd.io/_uploads/rJLnXHZl6.png)


### Why Pods exist in the first place?
Pods is the idea of resources type

- k8s tell the container rumtime to create containers for you

### Create 
``` bash!
kubectl create deployment my-nginx --image nginx
kubectl delete deployment my-nginx
```
![](https://hackmd.io/_uploads/rJNG8Sblp.png)


### Scaling ReplicaSets
What happend when we scaled?
- kubectl scale will `change` the deployment record
- CM will see that only replica has changed
- It will change the number of pods in RS
- Scheduler sees a new pod is requested, assigns a node
- Kubelet sees a new pod

``` bash!
kubectl create deployment my-apache --image httpd
kubectl scale deployment my-apache --replicas 2
```
![](https://hackmd.io/_uploads/Hy5WAufxa.png)


## Inspecting K8S Resources
### Get
``` bash!
kubectl get all -o wide
kubectl get deploy my-apache -o wide
kubectl get deploy my-apache -o yaml
```
### Describe
``` bash!
kubectl describe deploy my-apache
```

### Describe
Check out the log of pods
``` bash!
kubectl get pods -w
```


## K8S Services
### Services
- Cluster IP
    - Single, internal virtual IP
    - Only reachable from within cluster
    - Pods can reach service on apps port number
- Node Port
    - High port allocated on each node
    - Port is open on every node's IP 
- Load Balancer
    - Control a LB endpoint external to the cluster
- External Name
    - Add CNAME DNS record to coreDNS only


## Example


``` bash!
kubectl get pods -w

kubectl create deployment httpenv --image=bretfisher/httpenv
kubectl scale deployment/httpenv --replicas=5
kubectl expose deployment/httpenv --port 8888
kubectl expose deployment/httpenv --port 8888 --name httpenv-np --type NodePort
kubectl expose deployment/httpenv --port 8888 --name httpenv-lb --type LoadBalancer
```
