direct routing lab.

⬢ Cilium Routing Options
There are a variety of routing options in Cilium:

By default cilium will deploy in an overlay mode and encapsulate all traffic between nodes.

We also support Direct Routing. Where we route all traffic between nodes directly using the underlying network

In this lab, we will explore what happens to packets that are transmitted between pods when using the Direct Routing option.

We don't know that the nodes are part of 10.0.0.0/8 in fact docker exec -ti clab-bgp-cplane-demo-control-plane ip addr shows that they are part of a /24

I suspect that the cilium config is probably faulty in this case.

This means the Cilium Operator will assign CIDRs to each Kubernetes node using the Node resource type.

should be:

This means that cilium will look at the node resource and use the podCIDR allocated to it for the pod addresses.

Here is a great way to do the netshoot thing in a script.

https://gist.github.com/mauilion/79ca44b583e52faa33113f2f52cf46d2

Host to Pod.

Whenever considering a router we have to consider two one way routes. How do we route traffic into the container and how do we route that traffic back out.

When traffic is destined for a pod ip address we route that traffic through the cilium_host interface.

using the ip route get command we can query the linux routing table about how traffic will be routed.

From the host:

ip route get 10.1.0.82
10.1.0.82 dev cilium_host src 10.1.0.188 uid 0 
    cache 

From the pod:

ip route get 4.2.2.1
4.2.2.1 via 10.1.0.188 dev eth0 src 10.1.0.82 uid 0 
    cache mtu 9500 

Pod to Pod (Intra Host)

When considering how traffic is routed out of a pod. It's interesting to look at the subnet mask associated with the pod network interface.

6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default qlen 1000
    link/ether 9e:d5:d9:92:0e:d7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.1.0.82/32 scope global eth0

In most traditional networks a scenarion like this would describe traffic that is passed between hosts using the link layer or the mac addresses of both pods. Both hosts would have an address in the same CIDR and they would be configured to use the CIDR subnetmask. Then each host would maintain a list of know mac addresses for their neighbors and send traffic to the neighbor without having to resort to routing.

In the case of cilium and many sdn based approaches. We want to ensure this traffic is routed. Since the pod is allocated a subnetmask of /32 this means that we are configuring the pod not to participate in any broadcast domains. It will not learn the mac addresses of other pods on the host nor will it annouce it's mac address to them.

This means that for pod to pod traffic on the same node we again route that traffic through the cilium_host interface.

Since we are handling the forwarding path the cilium_host ip address may not appear in traceroute.

Pod to Pod (Inter host)

I'd spend a little more time on how pod traffic is being routed.
Look again at this as two one way routes.

pod on host 1 -> host routing table -> default route out "net" interface -> static or learned route entry for destination pod subnet pointed to a net interface on host 2 -> host 2 -> cilium_host -> pod on host 2

And the same in reverse.

In this case there is a lot of static configuration that is making it so that this works at all. Which is definitely worth pointing out as the bgp implementation then automations much of this.