# Working session 2021-08-08

## Setting up the cluster

gcloud container clusters \
    get-credentials {{ cluster_name }} \
    --region {{ region-name }}

This should get the kubeconfig file for the cluster and setup kubectl as well.

kubectl apply -f osquery-deployment.yml

Things to note in the file

  - name: osquery-exporter
    image: kubeir/osquery-exporter
    ports:
      - containerPort: 5000
    livenessProbe:
      httpGet:
        path: /
        port: 5000
      initialDelaySeconds: 3

  - name: sleeper
    image: docker.io/prateeknischal/sleeper
    securityContext:
      capabilities:
        add: ["NET_RAW"]
      privileged: true
      runAsUser: 0
    volumeMounts:
      - name: dockersock
        mountPath: /app/run
  volumes:
  - name: dockersock
    hostPath:
      path: /run

Then get the pod name using `kubectl get pods` and then enter the sleeper container using

kubectl exec -it {{ pod_name }} --container sleeper -- sh

The pod should have a few things in the `/app/run` folder, example

total 24
drwxr-xr-x   25 root     root           680 Aug  7 18:13 .
drwxr-xr-x    3 root     root          4096 Aug  7 18:16 ..
-rw-------    1 root     root             0 Aug  7 17:40 agetty.reload
drwxr-xr-x    2 root     root            80 Aug  7 17:40 blkid
drwxr-x---    2 root     root            60 Aug  7 17:40 chrony
-rw-r--r--    1 root     root             4 Aug  7 17:40 chronyd.pid
drwxr-xr-x    2 root     root           240 Aug  7 17:41 cloud-init
drwx--x--x    7 root     root           180 Aug  7 17:41 containerd
drwxr-xr-x    2 root     root            60 Aug  7 17:40 crash_reporter
drwxr-xr-x    3 root     root            60 Aug  7 18:04 crio
drwxr-xr-x    2 root     root            40 Aug  7 18:13 crio.sock
drwxr-xr-x    2 root     root            60 Aug  7 17:40 dbus
drwx------    5 root     root           120 Aug  7 17:41 docker
-rw-r--r--    1 root     root             4 Aug  7 17:41 docker.pid
srw-rw----    1 root     412              0 Aug  7 17:40 docker.sock
drwxr-xr-x    2 root     root            60 Aug  7 17:40 fsck
drwxr-xr-x    3 root     root            60 Aug  7 17:42 google-fluentbit
prw-------    1 root     root             0 Aug  7 17:40 initctl
drwxrwxrwt    3 root     root           100 Aug  7 17:40 lock
drwxr-xr-x    2 root     root            60 Aug  7 17:40 lockbox
drwxr-xr-x    2 root     root            40 Aug  7 17:40 log
drwx------    2 root     root            40 Aug  7 17:40 lvm
-r--r--r--    1 root     root            33 Aug  7 17:40 machine-id
drwxrwx---    3 20140    20140           60 Aug  7 17:40 metrics
drwxr-xr-x    2 root     root            40 Aug  7 17:40 mount
drwxr-xr-x    2 root     root            80 Aug  7 18:16 netns
-rw-r--r--    1 root     root             4 Aug  7 17:40 sshd.pid
drwx--x--x    3 root     root            60 Aug  7 17:40 sudo
drwxr-xr-x   18 root     root           440 Aug  7 17:41 systemd
drwxr-xr-x    2 root     root            60 Aug  7 17:40 tmpfiles.d
drwxr-xr-x    6 root     root           140 Aug  7 18:16 udev
drwxr-xr-x    2 root     root            40 Aug  7 17:40 user
-rw-rw-r--    1 root     utmp           768 Aug  7 17:40 utmp
-rw-------    1 root     root             0 Aug  7 17:40 xtables.lock

## Things to note
* There are 3 container engines in the above folder, crio, containerd and docker. For the node pool in GCP, we were using a Container optimised OS with containerd runtime. We can change that to docker using [using-containerd#updating-image-type](https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd#updating-image-type).
* The `containerd.sock` seems to exclusively communicate via gRPC

Once the runtime is changed to docker, we can do the following things

# where we mounted /var directory on our privileged container
cd /app/run
curl --unix-sock docker.sock http:/v1.41/containers/json

And the following files appear

docker.sock
dockershim.sock

This should give the list of containers running on the host, we select the container we want and then

curl --unix-socket docker.sock \
    http:/v1.24/containers/53e2ca7959d3311b37ac50b12847cd505d6fe60af2b003e9c1f53f1ffbbf5ed5/checkpoints \
    -H "content-type: application/json" \
    -d '{"CheckpointID": "foobar", "CheckpointDir": "/run/foo", "Exit": false}'

and we get a response

{"message":"This experimental feature is disabled by default. Start the Docker daemon in experimental mode in order to enable it."}

After this point, docker breaks due to checkpoint being in experimental stage and we can't restart the docker daemon on GKE nodes like we did on minikube.