# Exercises - P4 - In-band Network Telemetry (INT)

In-band Network Telemetry (“INT”) is a framework designed to allow the collection and reporting of network state, by the data plane, without requiring intervention or work by the control plane.

In the INT architectural model, packets contain header fields that are interpreted as “telemetry instructions” by network devices.

These instructions tell an INT--capable device what state to collect and write into the packet as it transits the network.


Complementary information about the P4-based INT lab can be found at [here](https://drive.google.com/file/d/1dnNZ88iMu_xH4WCVVmvTNMkZDBOUtgep/view?usp=sharing)

After downloading and importing this ([8GB virtual machine](https://drive.google.com/drive/folders/1z6EpkWXYfPhkDEwfY21gQvzPI3iXs41z?usp=sharing)), we can start the exercises below:

:::info
VM username: INT
VM password: int
:::

Tip: Run the lab from the VM, not over SSH, because there are some windows that don't work well over an SSH connection.

## Part 1 - Iperf Test

Quick fix: Comment the following lines to disable the iperf clients.

cd ~/Desktop/p4factory/mininet
gedit ./int_ref_topology.py
# h1.cmd("iperf -c 10.2.1.4 -t 3000 > /dev/null &") ........line 104
# h3.cmd("iperf -c 10.2.1.2 -t 3000 > /dev/null &") ........line 105

For running the reference INT application, use:

cd ~/Desktop/p4factory/mininet
sudo ./int_ref_topology.py --model-dir=$HOME /home/

This script “int_ref_topology.py” add and configure the two spines and leaf switches and start the quagga on switches for exchanging the routing and topology information. The Reference topology is given below (More details can be found at [LINK](https://drive.google.com/file/d/1dnNZ88iMu_xH4WCVVmvTNMkZDBOUtgep/view?usp=sharing)):


After creating the mininet emulated network, you can see the full mesh of pings between hosts as background traffic.

Now , load the visualizer HTML page, opening the file below on google chrome. To do this, open another terminal and run:

google-chrome ~/Desktop/p4factory/apps/int/monitor/client/index.html

You can visualize the telemetry data generated by mininet topology over Network Dashboard.

:::info
Network dashboard has 4 components:
1. Reference topology indicates the current path in blue and old path (congested path) in red.
2. Real time hop latencies graph per switch, each bar indicates the time spend by a packet in a switch.
3. Loop detected window.
4. Flow filter, using this you can choose the particular flow to visualize.
:::

The script “int_ref_topology.py” starts the two iperf clients on host 1 and host 3, now you can start the iperf traffic manually as follows:

In the Mininet CLI use:

mininet> xterm h1

and in the h1 CLI use:

iperf -c 10.2.1.3 -t 60

Monitor and web client recognize the new connection. We can visualize the latency of this particular flow via flow filter on network dashboard.

:::danger
In the browser, looking at the topology paths (blue and red) and the latency measurements (of flow 10.2.1.1 -> 10.2.1.3), can you see any changes in per packet latency at leaf1 after starting the iperf traffic between h1 and h3? Explain.
:::

## Part 2 - Routing failover test
Exit the current mininet running topology using the command:

mininet> quit

So, restart the application running:

sudo ./int_cleanup.sh

(You can ignore the errors: "Cannot find device ...")

Now start again the reference application running:

sudo ./int_ref_topology.py --model-dir=$HOME /home/
mininet> xterm h1

and in the h1 CLI runing:

iperf -t 100 -c 10.2.1.3

Now open a terminal on leaf1 at mininet CLI:

mininet> xterm leaf1 

:::danger
Check which spine switch passses the traffic using the Web Client UI. If the connection is going through spine1 or spine2, shut down the port on leaf1 that is connected to spine1 (leaf1-eth3) or spine1 (leaf1-eth4), respectively.
:::
To do this, if spine 1 is active run the command below in the leaf1 xterm

ifconfig leaf1-eth3 down

However, if spine 2 is active run in the leaf1 xterm the command:

ifconfig leaf1-eth4 down

:::info
Note: When you bring up any particular interface, please wait for some time before bring down any other interface
:::
:::danger
After disable the particular port at leaf1, (in the browser topology) can you see the changes in the ongoing physical route after 3 sec ? (3 secs is the BGP hold-down time)
:::
## Part 3 - Analyzing INT headers using Tshark test
Exit the current mininet running topology using the command:

mininet> quit

Restart the application:

sudo ./int_cleanup.sh

(You can ignore the errors: "Cannot find device ...")

Comment the following lines to disable the iperf clients and full mesh of pings:

gedit ./int_cfg.py
# self.startPingMesh() .........line 156

Now run the reference INT Application:

sudo ./int_ref_topology.py --model-dir=$HOME /home/

Open two consoles leaf1 running in the mininet CLI:

xterm leaf1
xterm leaf1

Start the tshark to capture the packets at interfaces leaf1-eth1 and leaf1-eth2 running in the new xterms (1 per xterm):

tshark -i leaf1-eth1 -V -c 5
tshark -i leaf1-eth2 -V -c 5

Now, open a new h1 CLI running:

xterm h1

And generate the iperf udp traffic from host 1 to host 2:

iperf -c 10.2.1.2 -t 1 -u

You can see, the 5 frames have been captured at leaf1-eth1 and leaf1-eth2 interfaces

:::danger
Compare the 5th frame (identified as “Frame5:”) at both interfaces. Can you see any difference between them ? Explain.
:::
:::info
Note: tshark, wireshark or tcpdump are not currently able to visualize the INT shim and metadata headers because dissector for INT specification is not defined in the given packet capturing tools.
Therefore, these tools read INT headers as MPLS headers.
:::
:::warning
For analyzing the INT headers and metadata added by every switch, we need to manually read the frames bit by bit and verifying with INT specification.
To understand how we can analyze that what types of information added by every switch for telemetry, please download the int.pcap file by the following link: https://intrig.dca.fee.unicamp.br:8840/index.php/s/AEZhzHFTpU4kjaL
:::

:::warning
Open the int.pcap in wireshark and compare the every bit of 1st frame with different headers format and, for INT shim and metadata headers, follow INT specification (page 8, 9, 10 , 11 , 14 & 15) for verification.

Ethernet

00c0 a0a0 0001 0001
0000 0001 0800

IP
4500 009e bb5f 0000 3f11
a8ed 0a00 0202 0a00 0101

UDP
dd01 12b6 008a
0000

VXLAN
0c Flags
00 00 Reserved
05 Next Protocol (05 INT Protocol)
0000 0a VNI
00 Reserved

INT
00 Type
00 Reserved
06 Length
03 Next Protocol

INT Metadata

0003 2001 INT Metadata header 0000 0000 0000 0011 0010 0000 0000 0001

00 Version
00 Replication Requested
0 Copy
0 Max Hop Count exceeded
00000 Reserved
00011 Instruction Count (3)
0010 0000 Max Hop Count (32)
0000 0001 Total Hop Count (1)

b000 Intrictions Bitmap 1011 0000 0000 0000

1 bit0 (MSB): Switch ID
0 bit1: Ingress port ID
1 bit2: Hop latency
1 bit3: Queue occupancy
0 bit4: Ingress timestamp
0 bit5: Egress port ID
0 bit6: Queue congestion status
0 bit7: Egress port tx utilization
0000 Reserved
0000 Reserved

0000 Reserved
0000 00a1 1st INT METADATA **Stack Switch ID**
0000 0017 2nd INT METADATA **Stack Hop Latency**
8000 0000 last INT METADATA **Stack Queue occupancy**

Note: Here, we analyze 1st frame of int.pcap bit by bit till INT metadata.
:::

:::danger
In similar way, can you find what type of INT metadata and their
values have been added by leaf1 by analyzing the 5th frame at
interface leaf1-eth2 ? Explain.
:::

## Congratulations!
Now you know a little about In-band Network Telemetry!