# Docker
## Installation
[Docker](https://docs.docker.com/desktop/install/mac-install)
## Creating the Container
### Check Version & Status
``` bash!
docker version
```
``` bash!
docker info
```
### Image vs. Container
### Starting a Nginx Web Server
#### Run Container
```bash!
docker container run --publish 80:80 nginx
```
#### Show Container
```bash!
docker container ls
```
#### Stop Container
```bash!
docker container stop 2020 # container id
```
#### List Container
```bash!
docker ps
```
#### What's Going on in Container
```bash!
docker container top
docker container stats
docker container inspect
```
### Getting a Shell Inside Containers
```bash!
docker container run -it
docker container exec -it
```
### Instance
Installation
```bash!
sudo apt-get update
sudo apt-get install docker.io -y
```
Ignore sudo
``` bash!
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker
```
Build Image
```bash!
sudo docker build -t ass1:kao .
```
Run Container
```bash!
sudo docker run -p 80:80 --name ass1_kao -d ass1:kao
```
## Docker Networks
### Default
- Each container connected to a private virtual network "bridge"
- Each virtual network routes through NAT firewall on host IP
- All containers on a cirtual network can talk to each other without '-p'
- Best practice is to create a new virtual network for each app:
- Network "my_web_app" for mysql and php/apache containers
- Network "my_api" for mango and nodeks containers
```bash!
docker container run -p 80:80 --name webhost -d nginx
docker container port webhost
docker container inspect --format '{{.NetworkSettings.IPAddress}}' webhost
```
### CLI Management
```bash!
docker network docker network ls # show networks
docker network inspect # inspect a network
docker network create --drive # create a network
docker network connecct # attach a network to container
docker network disconnecct # detach a network from container
```
### DNS
- Containers shouldn't rely on IP's for inter-communication
- DNS for friendly names is built-in if you use custom networks
## Docker Image
### What's In An Image
- Metadata about image data and how to run the image
- Not a complete OS. No Kernel, Kernel modules
- Small as one file (your app binary)
- Big as a Ubuntu
### Dockerfile
```bash!
# use this empty Dockerfile to build your assignment
# NOTE: The instructions are now in the README.md file in this directory.
FROM node:6-alpine
EXPOSE 3000
RUN apk add --update tini
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY package.json package.json
RUN npm install && npm cache clean
COPY . .
CMD [ "tini", "--", "node", "./bin/www" ]
```
## Persistent Data
### Container Lifetime & Persistent Data
- Containers are immutable: Only re-deploy the containers, never change
- This is an ideal scenario, but what about database??
- Docker give us features to ensure this concern
- This is known as 'persistent data'
- Two ways: Volumes and Bind Mounts
- Volumes: make special location outside of container
- Bind Mounts: link container path to host path
### Volume
> Voulme would not be removed even if we remove the container!!!
Install & deploy mysql
```bash!
docker pull mysql
docker inspect mysql
docker container run -d --name mysql -e MYSQL_ALLOW_EMPTY_PASSWORD=True mysql
```
Check Volume
```bash!
docker volume ls
```
Name Volume
```bash!
docker container run -d --name mysql -e MYSQL_ALLOW_EMPTY_PASSWORD=True -v mysql_db:/var/lib/mysql mysql
docker volume ls
```
### Bind Mounting
- Maps a host file or directory to a container file or directory
- Two locations point to the same files
- Can't use in Dockerfile, must be at container run
## Docker Compose
### Why
- Configure relationships between containers
- Save our docker container run settings in easy-to-read file
- Create one-liner developer environment startups
### Installation
```bash!
sudo curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose version
# example of running
docker-compose -f docker-compose.yml up -d
```
### Run Sample
```bash!
services:
drupal:
image: drupal
ports:
- "8080:80"
volumes:
- drupal-modules:/var/www/html/modules
- drupal-profiles:/var/www/html/profiles
- drupal-sites:/var/www/html/sites
- drupal-themes:/var/www/html/themes
postgres:
image: postgres
environment:
- POSTGRES_PASSWORD=mypasswd
volumes:
drupal-modules:
drupal-profiles:
drupal-sites:
drupal-themes:
```
### Using Compose to Build
- Compose can also build your custom images
- Will build them with docker-compose up if not found in cache
- Also rebuild with docker-compose build
- Great for complex builds that have lots of vars or build args
```bash!
services:
proxy:
build:
context: .
dockerfile: nginx.Dockerfile
ports:
- '80:80'
web:
image: httpd
volumes:
- ./html:/usr/local/apache2/htdocs/
```
### Remove compose
``` bash!
docker-compose down --rmi all --volumes
docker-compose -f docker-compose-prac7.yml down --volumes
```
## Swarm
### What Just Happened?
``` bash!
docker swarm init
docker node ls
```
### Docker Service Command
Service in Swarn **replace** Docker run
``` bash!
docker service create alpine ping 8.8.8.8
```
Show the running service & services in the service
``` bash!
docker service ls
docker service ps musing_lalande
```
```bash!
docker service update dfoohcboqep6 --replicas 3
```
```bash!
docker service rm musing_lalande
```
## 3-Node Swarm
### Connect with other nodes
On master node run following command to obtain token.
```bash!
docker swarm init
docker swarm join-token manager
```
Put the token into another node. The other nodes cannot access the node information because they are not the `Leader` node.
```bash!
docker swarm join --token SWMTKN-1 0942yqaq8avwbq66ae4o1eqair32z7bg96hcw9ktkkl9cp7tos-5qnsc882i6xuoubt0hkcaeidt 10.152.0.15:2377
```
### Assign leader node
```bash!
docker node update --role manager kao-2
```
### Run node
## Overlay Multi-Host Networking
1. Just choose `--driver overlay` when creating network.
2. Only for container-to-container inside a **single** swarm.
3. Each service can be connected to multiple networks.
```bash!
docker network create --driver overlay mydrupal
```
### Create service
```bash!
docker service create --name psql --network mydrupal -e POSTGRES_PASSWORD=mypass postgres
docker service create --name drupal --network mydrupal -p 80:80 drupal
watch docker service ls
```
## Stacks: production grade compose
Use `docker stack deploy` to deploy rather than `docker create`.
[Code](https://github.com/BretFisher/udemy-docker-mastery/tree/main/swarm-stack-1)
```yaml!
# this file is meant for Docker Swarm stacks only
# trying it in compose will fail because of multiple replicas trying to bind to the same port
# Swarm currently does not support Compose Spec, so we'll pin to the older version 3.9
version: "3.9"
services:
redis:
image: redis:alpine
networks:
- frontend
db:
image: postgres:15-alpine
environment:
POSTGRES_USER: "postgres"
POSTGRES_PASSWORD: "postgres"
volumes:
- db-data:/var/lib/postgresql/data
networks:
- backend
vote:
image: dockersamples/examplevotingapp_vote
ports:
- 5000:80
networks:
- frontend
deploy:
replicas: 2
result:
image: dockersamples/examplevotingapp_result
ports:
- 5001:80
networks:
- backend
worker:
image: dockersamples/examplevotingapp_worker
networks:
- frontend
- backend
deploy:
replicas: 2
networks:
frontend:
backend:
volumes:
db-data:
```
### Deploy
Use `docker stack deploy` to spin it up!
``` bash!
docker stack deploy -c example-voting-app-stack.yml voteapp
```
### Get information
``` bash!
docker stack ps voteapp
```
``` bash!
docker container ls
```
``` bash!
docker stack services voteapp
```
## Visualisation Tool
``` bash!
visualizer:
image: bretfisher/visualizer
ports:
- 8080:8080
stop_grace_period: 1m30s
networks:
- frontend
volumes:
- /var/run/docker.sock:/var/run/docker.sock
deploy:
placement:
constraints: [node.role == manager]
```
## Assignment
1. voting-app : Front-end of the application written in python flask framework, which allows the users to cast their votes.
2. redis : An in-memory data structure store, used as a temporary database to store the votes casted by the users.
3. worker : service written in .NET, that retrieves the votes data from redis and stores it into PostgreSQL DB service.
4. postgreSQL DB : PostgreSQL DB used as persistent storage database.
5. result-app : service written in node js, displays the voting results to the user.
### Non-compose version
#### Create internet
``` bash!
docker network create --driver overlay frontend
docker network create --driver overlay backend
```
#### Create service
``` bash!
docker service create --name vote --replicas 2 --network frontend -p 80:80 dockersamples/examplevotingapp_vote:before
docker service create --name redis --replicas 2 --network frontend redis:3.2
docker service create --name worker --replicas 1 --network backend dockersamples/examplevotingapp_worker
docker service create --name db --network backend -e POSTGRES_HOST_AUTH_METHOD=trust --mount type=volume,source=db-data,target=/var/lib/postgresql/data postgres:9.4
docker service create --name result --network backend -p 5001:80 dockersamples/examplevotingapp_result:before
```
[Failed to build db](https://stackoverflow.com/questions/67770835/detected-task-failure-docker-service-create-name-db-network-backend-mount)
### Debug
```bash!
docker service logs worker
```
# Hadoop
## Commands
Run `docker exec -it cc4f bash` to log in the namenode container
``` bash!
docker exec -it cc4f bash
echo "Hello, HDFS!" > $HOME/hdfs_demo.txt
hdfs dfs -put $HOME/hdfs_demo.txt /hdfs_demo/hdfs_demo.txt
hdfs dfs -copyFromLocal /path 1 /path 2 .... /path n /destination
```
# Kubernetes
## Architecture
### Borg
Borg use `cell` to define a series of machine.
`Cluster` includes multiple `cells`.
### Kubernetes
k8s is consisted of masters and nodes.
#### Master
1. kube-apiserver
Entry of API. Conducting the operations for database and nodes.
2. kube-controller-manager
Maintaining the kubernetes clusters, such as updating or creating.
3. kube-scheduler
Linsting the unused PODS.
#### Nodes
1. Creating and maintaining image.
2. Maintaining POD lifecycle
## Basic Terms
## k8s run, create, and apply
### Check version
``` bash!
kubectl version
kubectl version --short
```
### Run
Pod is not really running, k8s only store the request in this step.
``` bash!
kubectl run [pod-name] --image [image-name]
kubectl run my-nginx --image nginx
```
### Inpect
``` bash!
kubectl get pods
kubectl get all
```
### Why Pods exist in the first place?
Pods is the idea of resources type
- k8s tell the container rumtime to create containers for you
### Create
``` bash!
kubectl create deployment my-nginx --image nginx
kubectl delete deployment my-nginx
```
### Scaling ReplicaSets
What happend when we scaled?
- kubectl scale will `change` the deployment record
- CM will see that only replica has changed
- It will change the number of pods in RS
- Scheduler sees a new pod is requested, assigns a node
- Kubelet sees a new pod
``` bash!
kubectl create deployment my-apache --image httpd
kubectl scale deployment my-apache --replicas 2
```
## Inspecting K8S Resources
### Get
``` bash!
kubectl get all -o wide
kubectl get deploy my-apache -o wide
kubectl get deploy my-apache -o yaml
```
### Describe
``` bash!
kubectl describe deploy my-apache
```
### Describe
Check out the log of pods
``` bash!
kubectl get pods -w
```
## K8S Services
### Services
- Cluster IP
- Single, internal virtual IP
- Only reachable from within cluster
- Pods can reach service on apps port number
- Node Port
- High port allocated on each node
- Port is open on every node's IP
- Load Balancer
- Control a LB endpoint external to the cluster
- External Name
- Add CNAME DNS record to coreDNS only
## Example
``` bash!
kubectl get pods -w
kubectl create deployment httpenv --image=bretfisher/httpenv
kubectl scale deployment/httpenv --replicas=5
kubectl expose deployment/httpenv --port 8888
kubectl expose deployment/httpenv --port 8888 --name httpenv-np --type NodePort
kubectl expose deployment/httpenv --port 8888 --name httpenv-lb --type LoadBalancer
```
Sign in with Wallet
Connect another wallet