Debug your Oakestra deployments

--- title: "Debug your Oakestra deployments" tags: wikidocs teams: - maintainers participants: - Giovanni (GB) --- # How to access a running container? If you need to access a running container the proper way of doing it is via the `ctr` utility. `ctr` is the cli of `containerd`, the runtime used by oakestra to deploy containers. Oakestra uses a custom namespace `oakestra` where all the containers are. In `containerd` we distinguish between tasks and containers. A container is the environment, the filesystem, and all the metadata. A task is a process running in such an environment. Therefore, to check all running containers on a given worker node you can use: ``` sudo ctr -n oakestra container ls ``` To check all running tasks you can use the following: ``` sudo ctr -n oakestra task ls ``` If you wish to attach and execute commands inside a running container, you can use: ``` sudo ctr -n oakestra task exec --exec-id tty <your task name here> <the command you wish to execute> ``` >E.g., to use a shell inside the container x.y.z we can use > ``` > sudo ctr -n oakestra task exec --exec-id tty x.y.z /bin/sh > ``` >![Screenshot 2024-06-05 at 09.47.05](https://hackmd.io/_uploads/HJScu5TN0.png) # Where are the logfiles stored? The `stdout` and `stderr` of each container or unikernel is stored under `/tmp/<appname>.<appns>.<servicename>.<servicens>.<instancenumber>` of the worker node running the instance. > E.g., to access the latest logs of instance 0 of `x.y.z.y` in my worker node I can run `tail /tmp/x.y.z.y.0` # How to access the network namespace of a container When `containerd` generates the network namespace for your container it does not show up out of the box in the network namespace list. In fact, when running: ``` sudo ip netns list ``` You'll get an empty list, even if using ``` sudo ctr -n oakestra container ls ``` Will give you a list of running containers. ### So... what do we do? The issue is simple, when a container is deployed and a network namespace is created, the symlink of the net namespace is not automatically created inside the `/var/run/netns` directory. So we just need to do that! First, let's retrieve the PID of the container ``` sudo ctr -n oakestra task ls ``` ![image](https://hackmd.io/_uploads/BkJlZauz0.png) In my case, the PID is `222432` (Optional) If not there already, create the /var/run/netns/ directory with mkdir -p /var/run/netns/ Then create the symlink using the command ``` ln -sfT /proc/<container PID>/ns/net /var/run/netns/<container name> ``` In my case, the command will look like ``` ln -sfT /proc/222432/ns/net /var/run/netns/test.test.nginx.test.instance.0 ``` now if I run the command ``` sudo ip netns list ``` I'll see the `test.test.nginx.test.instance.0` namespace. # How to debug the network Once the namespace is accessible via the `netns` command, you can enter inside it and debug your environment using the classic utilities such as `tcpdump`/`tshark`, you can check the interfaces using the `ip` utils or execute any command you fancy. You can even create new interfaces inside and play around with them. Simply use ``` ip netns exec <containername> <your command> ``` E.g., to show the interfaces inside the namespace of the previous example I can run ``` sudo ip netns exec test.test.nginx.test.instance.0 sudo ip a s` ``` # What about Unikernels network If you're running unikernels directly using Oakestra native unikernel virtualization, you'll automatically find the namepsace in `ip netns list`. If you're using `runu` runtime attached to containerd, you can still use the procedure described above for containerd.