---
title: ' REINFORCEMENT LEARNING BASED ROUTING ALGORITHM OPTIMIZATION ON VANET WITH PRISMA '
disqus: hackmd
---
{%hackmd @themes/dracula %}
# Machine Learning/Artificial Intelligence Research Documentation
A research under Adaptive Network Laboratory Telkom University. A detailed documentation made by yours truly,
Related Works :
[PRISMA: A Packet Routing Simulator for Multi-Agent Reinforcement Learning](https://hal.science/hal-03709948)
# Table of Contents
[TOC]
# Machine Specifications
**Host Machine :**
* OS : Windows 11 Home Single Language 64-bit (10.0, Build 22621)
* Processor : AMD Ryzen 7 6800U with Radeon Graphics (16 CPUs), ~2.7GHz
* Memory : 16384MB RAM
**Virtual Machine :**
* OS : Ubuntu 22.04.3 LTS
* Memory : 7048MB (6.6GiB) RAM
* Processors : 6
# Notes
[Slide Monthly Report 1](https://www.canva.com/design/DAFyVxt9-CY/jAejSsDF6sWfqkTemphIFg/edit?utm_content=DAFyVxt9-CY&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton)
[Slide Monthly Report 2](https://www.canva.com/design/DAF2detlJOI/hjWeLGtyhqo9AN0rXDW5kw/edit?utm_content=DAF2detlJOI&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton)
Progress Tracking :
* Prerequisite Installation
* Running PRISMA (Default script)
* Running SUMO
⚠️ **WARNING** ⚠️
> **This documentation is an ultimate ~~heresy~~ filled with dozens of foolish absolutely regrettable steps, tread with caution**.
# PRISMA
See also: [PRISMA Repository](https://github.com/rapariciopardo/PRISMA)
Packet Routing Simulator for Multi-Agent Reinforcement Learning (PRISMA) is the first Deep Reinforcement Learning (DRL) framework specifically devoted to the Distributed Packet Routing (DPR) problem. In the paper, the authors ilustrated its main functionalities by applying the tool to learn in a distributed manner a Shortest Path routing policy in two backbone networks.
## Literature Review 📖
### Packet Routing Models
Routing is the process of choosing a path across one or one more network, from one node to another. Routing algorithms can be classified into two types:
* **Static**
Also called as **"non-adaptive routing"** because its routing configuration is done **manually** by the network administrator.
(+) No bandwidth usage between routers, adds security because only an administrator can allow routing to particular networks.
(-) Hard to manage for large networks because administrators has to manually add each other route for the network in the routing table on each router.
* **Dynamic**
Also known as **"adaptive routing"**, a type of routing which is done **autonomously** without any human intervention. Packets are transmitted over a network using various shortest path algorithms and pre-determined metrics. The router adds a new routes to the routing table
(+) Easy to configure, more effective at discovering remote networks and selecting the best route to a destination remote network.
(-) Consumes more bandwidth for communicating with other neighbors.
**Dynamic protocols** are further classified as **"centralized"** and **"distributed"**. **Centralized protocols** focus on **a central node** for all routing decisions, while **distributed protocols** make **each device** on the network responsible for making routing decisions.
Although centralized and distributed routing protocols are dynamic routing protocols, their operation is very different. **The main difference between them is the network devices that make the routing decisions**. A central node is responsible for all routing decisions in centralized routing, while each device is responsible for routing decisions in distributed protocols.
Centralized protocols pose many problems compared to distributed protocols, such as the single point of failure and the potential congestion of the network around the central node. For these reasons, distributed protocols are more commonly used.
### Distributed Packet Routing Problem
---
The purpose of routing protocols is to learn about all the available paths to route data packets, automatically build routing table and take routing decisions based on specified metrics.
There are two primary types of routing protocols:
Distance Vector Routing (DVR) Protocol
A protocol where the routers send information about their directly connected networks "only to the neighbors". Each router keeps up a table containing data and selects best routes from the perspective of the neighbors. DVR is welll-suited for smaller systems. DVR uses [Bellman Ford Algorithm](https://www.geeksforgeeks.org/bellman-ford-algorithm-dp-23/) to find the shortest path.
Ex: BGP, EIGRP, RIP
Link State Routing
A protocol where the router attempts to construct its own internal map of the network topology by sending information about their links to "all routers". Each router makes router selecetion decisions itself. LSR uses [Djikstra's Algorithm](https://www.geeksforgeeks.org/introduction-to-dijkstras-shortest-path-algorithm/) to find the shortest path.
Ex: OSPF, ISIS
---
## Installation ⚙️
Based on the [installation guide](https://github.com/rapariciopardo/PRISMA), there are two ways of installation.
When the default VM doesn't have the Docker dependencies, here is a step-by-step of how to install it :
### Docker Installation (using the apt repository) 🐋
See also : [Official Docker Documentation](https://docs.docker.com/engine/install/ubuntu/), [Una's Complete Docker Documentation](https://)
**1. Set up Docker's apt repository.**
```bash!
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
```
> GPG (GNU Privacy Guard) keys are a way to verify the authenticity and integrity of software packages. [color=#3876c1]
```bash!
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
```
**2. Install the Docker packages**
To install the latest version, run:
```bash!
$ sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```
**3. Verify that the Docker Engine installation is successful**
Try running the hello-world image:
```
$ sudo docker run hello-world
```
> This command downloads a test image and runs it in a container. When the container runs, it prints a confirmation message and exits.
[color=#3876c1]
📸 :
### NS3-Gym Installation
See also: [NS3-Gym Repository](https://github.com/tkn-tub/ns3-gym)
Based on the [NS3-Gym submodule](https://github.com/tkn-tub/ns3-gym/tree/19bfe0a583e641142609939a090a09dfc63a095f) in PRISMA's repository, PRISMA was built in an older [version of NS3](https://github.com/tkn-tub/ns3-gym/tree/master?tab=readme-ov-file#versions) with waf build system.
**1. Start with cloning the repository.**
```bash!
$ sudo git clone -b master https://github.com/tkn-tub/ns3-gym.git
```
> The command above served to clone a specific branch from a Git repository, in this case it's "master" branch
**2. Install all required dependencies and update local index.**
```bash!
$ sudo apt-get install gcc g++ python-is-python3
```
```bash!
$ sudo apt update
```
**3. Install ZMQ and Protocol Buffers libs.**
```bash!
$ sudo apt-get install libzmq5 libzmq3-dev libprotobuf-dev protobuf-compiler
```
**4. Configure NS-3 Project.**
```bash!
$ ./waf configure --enable-examples
```
📸 :
**5. Build NS-3 Project**
```bash!
$ ./waf build -v
```
📸 :
> I suppose this really is an experiment of blasphemy. For further info check out: [NS-3 Build Failed Problem](https://hackmd.io/iDCp2kSgTXefrsQgvJJWrQ?both#NS3-Gym-build-failed). Currently working on this one!!
**6. Install NS3-Gym**
```bash!
$ sudo pip3 install --user ./src/opengym/model/ns3gym
```
📸 :
**7. Test whether NS3-Gym is installed**
```bash!
$ sudo pip3 show ns3gym
```
📸 :
**8. Run example**
```bash!
$ cd ./scratch/opengym
```
```bash!
$ sudo ./simple_test.py
```
📸 :
### PRISMA Installation (using Docker local image)
**1. Clone PRISMA's repository.**
```
$ sudo git clone https://github.com/rapariciopardo/PRISMA
```
**2. Build the docker image using the docker file.**
```
$ sudo docker build -t prisma .
```
📸 :
**3. Run the docker image and bind only the examples folder to the container.**
```bash!
$ sudo sudo docker run --rm --gpus all -v $(pwd)/prisma/examples:/app/prisma/examples -w /app/prisma prisma /bin/bash -c "python3 main.py $PARAMS"
```
## **Troubleshooting 🛠️**
#### “Low Disk Space on Filesystem root” ✅
See also: [VirtualBox: How to solve the issue with low disk space](https://blog.surges.eu/virtualbox-how-to-solve-the-issue-with-low-disk-space/)
I happened to stumble upon [this](https://hackmd.io/iDCp2kSgTXefrsQgvJJWrQ?both#VM-won%E2%80%99t-start-after-%E2%80%9CLow-Disk-Space-on-Filesystem-root%E2%80%9D) and ended up making an entire new VM because I forgot to take any Snapshot.
📸:
Here are the step-by-step on solving the problem above:
1. Create a Snapshot, and use the **Cloned Snapshot**.
2. Open the Virtual Media Manager “File > Virtual Media Manager”.
3. Select the machine that size you want to enlarge and use the size slider to change the value.
>The screenshot above was taken **after** I sucessfully fix the problem where my current VM is in a saved state. (That's why the slider is grayed)
[color=#00CED1]
5. Start VM and install 'gparted'.
```bash!
$ sudo apt update
$ sudo apt install gparted
```
6. Run gparted.
```bash!
$ gparted
```
7. Choose the partition. In my case is /dev/sda3
Unmount the file system
.
 .
8. Resize the partition by using the slider or entering a value and press the button “resize/move“.
9. Apply the changes, restart the virtual machine.
```bash!
$ reboot
```
10. Check the result
```bash!
$ df -k
```
#### PRISMA Installation using Docker local image.
> I followed the step-by-step [guide](https://github.com/rapariciopardo/PRISMA?tab=readme-ov-file#1-build-the-docker-image-locally) and it all went well, at first. I started with cloning the repo, navigate into PRISMA directory, and build docker image using the command below
```
$ docker build -t prisma .
```
📸 :
> I've done several rebooting, creating new VMs, etc. My guess is that the `newest NS3-Gym version doesn't have waf directory`
> See below :
> 
(pay attention to the branch version "app-ns-3.36+")
>It turns out, I was right! I just checked the "master" branch and found [this](https://github.com/tkn-tub/ns3-gym/tree/master?tab=readme-ov-file#versions) written in the Readme file
>
If you're experiencing similar issues, please kindly follow through the step-by-step above. [CASE CLOSED]
>**upset epic tragic moment**
#### NS-3 build failed
```bash!
$ ./waf build
```
📸:
## **Unsolved Problems 🪦**
#### VM won't start after “Low Disk Space on Filesystem root”.
> This happened after I installed PRISMA, which I thought was weird. At first I chose to **ignore** it. It turns out, I only made things worse 💀
[color=#FFB6C1]
> The VM lagged so bad at first, before it completely stopped responding. I decided to power off the machine and start it again.
📸 :
> I can't find anything to solve this, and ended up creating a new VM.
>It's best not to be too reckless 😂
Welp, that serves as these friendly reminders:
`Remember to do back-up, kids!` and `A Snapshot could save you a lifetime!`
#### PRISMA Installation using an existing environment from DockerHub.
It went well, at first.
```
$ docker pull allicheredha/prisma_env
```
📸:
Until I tried this command:
```bash!
$ sudo docker run --rm --gpus all -v $(pwd)/prisma:/app/prisma -w /app/prisma allicheredha/prisma_env /bin/bash -c "python3 main.py $PARAMS"
```
📸:
`docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].`
> I was hungry and sleepy. So I decided to just BlackboxAI-my-way-out of this error before I try running it again.
📸:
> HOW COME?!?!
Here lies some solutions I find in the amazing world of internet :
Source: [noot noot](https://stackoverflow.com/a/74724468)
> It didn't work. +1 Downvote 👎
https://forums.developer.nvidia.com/t/could-not-select-device-driver-with-capabilities-gpu/80200
Source: [Yeet](https://forums.developer.nvidia.com/t/could-not-select-device-driver-with-capabilities-gpu/80200/2)
#### No module named 'tensorflow'
📸:
### Program 'open-gym' not found;
📸:
# VANET
###### tags: `Documentation` `Research` `Machine Learning` `Deep Reinforcement Learning` `Network Simulator` `NS-3` `NS3-Gym` `Open-Gym`
Sign in with Wallet
Connect another wallet