---
layout: post
comments: true
title: Post Template
author: Zhenghao Peng (Team 00)
date: 2022-09-19
---
# Effects of Multi-agent Cooperation on Traffic Collision Avoidance
## Motivation
In this project, we aim to study the effects of multi-agent and cooperation between multi-agent in avoiding real-world traffic collisions. While autonomous driving has become increasingly popular in the past decade, it also raised safety concerns about whether human and AI driven cars can safely share the road together without increasing number of collisions. In this project, we ask the following question: is it safer to fit agents into a human-dominant traffic environment, or to give agents full control of all traffic on road without human drivers involved? The former can be considered a *hybrid environment* where the agents learn policies that align with existing traffic rules. The latter, however, does not contain any human driver and hence places no constraints on the behaviors of the agents. One advantage of the multi-agent model is that it permits real-time communications among agents, which could potentially decreases the risk of accidents.

In particular, we run a comparative study on the following three models:
* **Single-agent Model (Baseline).** In a traffic environment, we randomly pick one driver from all drivers and make it a controllable agent. The agent needs to learn a policy such that it does not cause any collision with other human drivers.
* **Hybrid Model.** We randomly pick half of the drivers to be controllable agents. In the case without cooperations, each agent delivers a policy based on its own state and observation. In the case with cooperations, the agents deliver policies that may depend on other agents' observations and actions.
* **Agents-only Model.** All the drivers in the environment are controllable agents. Similar to the cases for Hybrid Model, the agents may or may not share information depending on whether cooperations are allowed.

Different reward functions will be tuned to optimize the performance [Mataric]. We evaluate following metrics:
* **Collision Avoidance/Task Completion:** Whether or not the agents can avoid collisions while reaching the expected final location within the timeframe for each episode.
* **Training Resource:** Time/memory required for the RL training.
## Environment and Data

The environment we will be using is Nocturne [Vinitsky et al.]. Nocturne is a 2D driving simulator built from traffic scenarios in the Waymo Open Dataset. The uniqueness of this simulator is that it can compute an agent's visible objects as shown in Figure 1, and can generate more than 2000 states for each agents for a single second timeframe.

[add figure]

Figure 1: A visual showing the traffic for a specific scene on the road (Left). And the obstructed view of the single viewing blue agent (Right).

Currently, Nocturne generates environments based on a set of initial and final road object positions and (optionally) a set of trajectories for the road objects obtained from the Waymo Motion Dataset [Ettinger et al.] However, Nocturne currently does not support traffic lights, pedestrians or cyclists within the environment due to certain contraints and challenges within the Waymo Motion Dataset. So we will not be considering these within our exploration as well.

For states representations, Nocturne provides two different ways: an image and a vectorized representation of the image. For ease of use, we will be mainly using the vectorized representation for training.

<!-- Here we define the vehicle that is being trained and require observation from the environment as the traning vehicle agent (TVA). -->

Based on the environment and states, we could be able to get a list of following **observable infomation** for each of the vehicle in the environment:

* The speed of the vehicle
* The distance and angle to goal position
* The width and length of the vehicle
* The relative speed and heading compared to the expert driver (human data)
* The angle between the velocity vectors of a specific vehicle and all other vehicles in the environment
* The angle and distance to a specific vehicle from all other vehicles in the environment
* Other road objects' heading relative to the a specific vehicle

And each vehicle in the environment can take the following **actions**:
* acceleration: 6 actions from [-3, 2] $m/s^2$
* steering: 21 actions between [-0.7,0.7] radians
* head tilt: 5 actions between [-1.6,1.6] radians

Partial observability is achieved by setting a 120-degree view-cone for the training agent. Within the 120-degree view-cone, all vehicles that intersect with a single ray in the cone and is not blocked by other vehicles closer to the training agent is considered visible. During training, agents will only be able to obtain information from observable vehicles, and is able to use head tilt action to adjust the view-cone to obtain more information.

<!-- And each **episode** is defined as follows:
* 9 second per episode
* each step is 0.1 second -->

<!-- We will be using this generated environment for our Reinforcement Learning model training. -->

## RL Models
For Single-agent model, we consider the Deep Q-learning algorithm [Watkins, Dayan], [Mnih et al.], which follows Q-learning but usese a neural network to map each state and observation into pairs of action and Q-value. The policy is improved when updating the weights of the neural network.

For multi-agent model without cooperations, we simply create multiple agents and apply the Deep Q-learning algorithm for each agent independently.

For multi-agent model with coorperations, we consider the MADDPG algorithm [Lowe et al.], where a centralized Q function is used to integrate information from other agents.

We also consider a partial-observation environment. That is, at different time frames, an agent may only observe different portion of the environment. We apply the technique of Attention [Vaswani et al.] [Iqbal and Sha] to obtain embeddings with fixed lengths.

## Environments & Libraries
We may use the following packages (subject to change):
* Python3
* Nocturne and Waymo dataset packages
* OpenAI - MADDPG
* PyTorch
* Other Python packages such as Numpy, Pandas, etc.

## Reference
* Eugene Vinitsky, et al. Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. 2022. https://arxiv.org/abs/2206.09889.
Github: https://github.com/facebookresearch/nocturne
* Waymo Open Dataset: https://github.com/waymo-research/waymo-open-dataset
* Christopher J. C. H. Watkins, Peter Dayan. Q-learning. Mach Learn 8, 279–292 (1992). https://doi.org/10.1007/BF00992698
* Volodymyr Mnih et al. Human-level control through deep reinforcement learning. Nature. 2015. https://doi.org/10.1038/nature14236.
* Ryan Lowe et al. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017. https://arxiv.org/abs/1706.02275.
* Alexey Dosovitskiy et al. CARLA: An Open Urban Driving Simulator. Proceedings of Machine Learning Research. 2017. http://proceedings.mlr.press/v78/dosovitskiy17a/dosovitskiy17a.pdf.
* Quanyi Li and Zhenghao Peng and Bolei Zhou. Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization. International Conference on Learning Representations (ICLR). 2022. https://openreview.net/forum?id=0cgU-BZp2ky
* Maja J. Mataric. Reward Functions for Accelerated Learning. ICML 1994. https://www.sci.brooklyn.cuny.edu/~sklar/teaching/boston-college/s01/mc375/ml94.pdf
* Ariel Rosenfeld and Matthew E. Taylor and Sarit Krau. Leveraging Human Knowledge in Tabular Reinforcement Learning: Study of Human Subjects. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, {IJCAI} 2017, Melbourne, Australia. 2017. https://www.ijcai.org/proceedings/2017/534
* Ashish Vaswani et al. Attention is All you Need. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017. https://arxiv.org/pdf/1706.03762.pdf
* Shariq Iqbal and Fei Sha. Actor-Attention-Critic for Multi-Agent Reinforcement Learning. Proceedings of Machine Learning Research. 2019. http://proceedings.mlr.press/v97/iqbal19a/iqbal19a.pdf
* J. Ho and S. Ermon, “Generative adversarial imitation learning,” in Advances in Neural Information Processing Systems, pp. 4565–4573, 2016.
* Stanford Course Project: https://web.stanford.edu/~blange/data/Data-Driven%20Multi-agent%20Human%20Driver%20Modeling.pdfw
* Ettinger, S., Cheng, S., Caine, B., Liu, C., Zhao, H., Pradhan, S., Chai, Y., Sapp, B., Qi, C. R., Zhou, Y., et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 9710–9719.
<!--

> This block is a brief introduction of your project. You can put your abstract here or any headers you want the readers to know.

{: class="table-of-content"}
* TOC
{:toc}

## Main Content
Your article starts here. You can refer to the [source code](https://github.com/lilianweng/lil-log/tree/master/_posts) of [lil's blogs](https://lilianweng.github.io/lil-log/) for article structure ideas or Markdown syntax. We've provided a [sample post](https://ucla-rlcourse.github.io/CS269-projects-2022fall/2017/06/21/an-overview-of-deep-learning.html) from Lilian Weng and you can find the source code [here](https://github.com/ucla-rlcourse/CS269-projects-2022fall/blob/main/_posts/2017-06-21-an-overview-of-deep-learning.md)

## Basic Syntax
### Image
Please create a folder with the name of your team id under `/assets/images/`, put all your images into the folder and reference the images in your main content.

You can add an image to your survey like this:
{: style="width: 400px; max-width: 100%;"}
*Fig 1. YOLO: An object detection method in computer vision* [1].

Please cite the image if it is taken from other people's work.

### Table
Here is an example for creating tables, including alignment syntax.

| | column 1 | column 2 |
| :--- | :----: | ---: |
| row1 | Text | Text |
| row2 | Text | Text |

### Code Block

# This is a sample code block
import torch
print (torch.__version__)

### Formula
Please use latex to generate formulas, such as:

$$
\tilde{\mathbf{z}}^{(t)}_i = \frac{\alpha \tilde{\mathbf{z}}^{(t-1)}_i + (1-\alpha) \mathbf{z}_i}{1-\alpha^t}
$$

or you can write in-text formula $$y = wx + b$$.

### More Markdown Syntax
You can find more Markdown syntax at [this page](https://www.markdownguide.org/basic-syntax/).

## Reference
Please make sure to cite properly in your work, for example:

[1] Dwibedi, Debidatta, et al. "Counting out time: Class agnostic video repetition counting in the wild." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[Peng, et al.] Peng, Zhenghao, et al. "Maybe you can also use other format for reference as you wish." Nature. 2022.

---

## Data Rich and Physics Certain

| Experiment | Parameters | Results | Comments |
| :--- | :----: | :---: | ---: |
| **DL + Data** |

| Predicting only velocity | Dataset size : 10000<br> Network : 2->5->5->1 <br> activation: ReLU | ~100% accurate | Generalises well over various initial velocities |
| Predicting only displacement | Dataset size : 10000<br> Network : 2->16->16->1 <br> activation: ReLU | Reasonable | Better prediction for $u_0 \in dataset$, average prediction outside |
| Predicting both $v_t, s_t$ | Dataset size : 10000<br> Network : 2->16->16->2 <br> activation: tanh | Reasonable | Better prediction for $u_0 \in dataset$, poor prediction outside |

-----

| **DL + Physics** |
| Predicting both $v_t, s_t$, using Loss $L_{physics} = \|v_{predicted}^2-u_{initial}^2-2*g*s_{predicted}\|$ | Dataset size : 10000<br> Network : 2->16->16->1 <br> activation: ReLU | ~0% accuracy | Expected result as no supervision of any kind is provided |
| Predicting both $v_t, s_t$, using Loss $L_{velocity+phy} = (v_{predicted}-v_{actual})^2+\gamma*(v_{predicted}^2-u_{initial}^2-2*g*s_{predicted})^2$ | Dataset size : 10000<br> Network : 2->16->16->1 <br> activation: ReLU | Reasonable | Prediction of $v_t$ is good. Was able to learn $s_t$ reasonably well without direct supervision |
| Predicting both $v_t, s_t$, using Loss $L_{supervised+phy} = (v_{predicted}-v_{actual})^2+(s_{predicted}-s_{actual})^2+\gamma*(v_{predicted}^2-u_{initial}^2-2*g*s_{predicted})^2$ | Dataset size : 10000<br> Network : 2->16->16->1 <br> activation: ReLU | Reasonable | Not a better result w.r.t direct supervision |

**Observations :**
- Physics equations are certain in this case and are the best to use.
- Both DL, Hybrid(DL+Physics) methods performance are equivalent (actual accuracy/loss varies based on fine training, random dataset generation)

Re running the above experiments with Dataset size of 200(Data Starvation), yielded the following observations
- DL performance is comparable with 10000 dataset when trained on much mode epochs(5x)
- Hybrid(DL+Physics) without direct supervision on $s_t$ has comparable/better closeness than DL only method for limited epochs($\sim$300) training.

## Data Rich and Physics Uncertain

| Experiment | Parameters | Results | Comments |
| :--- | :----: | :---: | ---: |
| **DL + Data** |\
| Predicting both $v_t, s_t$ | Dataset size : 10000<br> Network : 2->16->16->2 <br> activation: tanh | Reasonable | Better prediction for $u_0 \in dataset$, poor prediction outside |
| **DL + Physics** |
| Predicting both $v_t, s_t$<br> using Loss $L_{physics} = \|v_{predicted}^2-u_{initial}^2-2*g*s_{predicted}\|$ | Dataset size : 10000<br> Network : 2->16->16->1 <br> activation: ReLU | ~0% accuracy | Expected result as no supervision of any kind is provided |
| Predicting both $v_t, s_t$<br> using Loss $L_{velocity+phy} = (v_{predicted}-v_{actual})^2+\gamma*(v_{predicted}^2-u_{initial}^2-2*g*s_{predicted})^2$ | Dataset size : 10000<br> Network : 2->16->16->1 <br> activation: ReLU | Reasonable | Prediction of $v_t$ is good. Was able to learn $s_t$ reasonably well without direct supervision |
| Predicting both $v_t, s_t$<br> using Loss $L_{supervised+phy} = (v_{predicted}-v_{actual})^2+(s_{predicted}-s_{actual})^2+\gamma*(v_{predicted}^2-u_{initial}^2-2*g*s_{predicted})^2$ | Dataset size : 10000<br> Network : 2->16->16->1 <br> activation: ReLU | Reasonable | Not a better result w.r.t direct supervision, but bettr than DL when $u0$ is out of dataset |

**Observations :**
- Both DL, Hybrid(DL+Physics) methods performance are similar, Hybrid(DL+Physics) is better when $u0$ is out of dataset, DL is better for $u0$ in dataset.
- Physics equations are not certain in this case and the above methods are better to use than Physics.

## Data Starvation and Physics Uncertain
- Similar observations as in data rich

-->