# MLC Seminar April 6, 2021: "Introduction to Reinforcement Learning" by Fabian Hart

This hackmd document will serve as the show notes for Fabian Hart's talk. Please use it for questions or important notes that add to the content of the speaker. After the talk, these notes will be stored and made publicly on the seminar website https://mlcdresden.gitlab.io/pages/

For the entire seminar, please be mindful of your peers and supportive in your communication. We henceforth make following the Dresden code of conduct obligatory for everyone connected:
https://dresden-code-of-conduct.org/de/
In light of this, please mute yourself if you are not talking!

## Community News :new:

Please share below one item of machine learning related news! This can be papers you or your colleagues published, blogs posts you wrote or that you liked, ...

- I am happy to share, that we have published an article in the [proceedings of the "Teaching Machine Learning" workshop](https://proceedings.mlr.press/v141/) from 2020 with the title ["Teaching Machine Learning in 2020"](http://proceedings.mlr.press/v141/steinbach21a.html)

# Notes & Questions

## Notes & Comments

post seminar mail by community member Max Pritzkoleit:

> Dear ML community,
>
> sadly I did not have the time to join the seminar, but for anyone further interested in the topic, my German written diploma thesis titled "Bestärkendes Lernen zur Steuerung und Regelung nichtlinearer dynamischer Systeme (engl. Reinforcement Learning for the Control of Nonlinear Dynamical Systems
)" [qucosa link](https://nbn-resolving.org/urn:nbn:de:bsz:14-qucosa2-377219) is about the topic of the seminar. It contains a pretty dense overview of the field of RL. I also used DDPG in my research and compared it to a control theoretic approach. It is probably worth a read .
>
> Best regards,
>
> Max Pritzkoleit

## Questions

- Question: how scale invariant are the actors once training has concluded?
For example, the traffic museum has (or used to have) a model of small ships going back and forth in harbor basin. Due to their mass (~1kg) I guess the forces are totally different than for a real vessel on the rhine. Would an actor that was trained on a rhine simulation work with these small ships?
- FH: would be really cool to try this out!
- FH: training on this real system is challenging (time investment and trials runs needed)
- OO on the chat: The dynamics of the vessel is purely physical. We have this is one goals. To run on the physical vessels.

- Question: why not using a sophisticated search (e.g. gradient based) to get the optimal action (I think slide 19 or 20)?
- problem: hard to find an action given a continuous output
- FH: tricky from a computational point of view

- Question: Debugging RL is quite hard and for your hierarchical system possibly even harder. Do you have any tipps & tricks on how to find problems or maybe ensure that your results are not too dependent on a lucky model initialization?
- FH: yes, RL is hard to debug
- RL on the chat: You can use explainable reinforcement learning for Debugging: https://towardsdatascience.com/dear-reinforcement-learning-agent-please-explain-your-actions-da6635390d4d
- FH: tricky as it turns out to be a hen-and-egg problem, simulator->model->rewardfunction->action
- PH: challenging as once the software goes into the wild, you need to explain the prediction (mostly if it )

- Question: Where would you embed "hard penalties" in the system, i.e. to make sure that the vessel does never ever hits the coastline (i.e. completely forbidden actions)
- FH: different concepts around to do this; FH allows any action (even if that leads to a collission), high penalty of `-1` so hitting the coast does happen
- OO on the chat: We will also use the true trajectory data in order to do kind of semi-supervised learning. This would help in training, and not so heavily depend on the starting values. (That was to the 3rd question)
- FH: there are methods known as "safe RL"
- LS: how to limit the actions to "safe" ones?
- FH: limit actions by hand (for the training process)
- RL: Another way to do it is to map the scale of actions only to those that are safe in a separate step after prediction
- you can have a look at this paper https://www.researchgate.net/publication/342747549_LongiControl_A_Reinforcement_Learning_Environment_for_Longitudinal_Vehicle_Control (figure 3)

- Question: how to define the granularity of each episode? what happens when there is an action taken and it changes the environment so that the choosen action is no longer valid, eg. speed up to overtake another boat, but the other boat speeds up to move away, so no overtaking should be done, so no speed up of the other boat so again there is a decision to overtake... does this feedback loop happen?
- FH: a huge problem that this question describes
- FH: simulator also has to model this -> vessels communicate with each other
- OO: As the partial solution, we also try to keep the gap between vessels big enough, to allow anyone to jump between. Similar as on the highway.
- FH: memory in decision taken into account -> maybe use this in the future

- Question: is it planned to consider high or low water, sedimentation or erosion varying the river bed?
- FH: all of these effects need to be considered, currently looking into more simple conditions for now

- Question: Are recurrent algorithms (LSTMs or similar) used, or just CNN based? If not, have you considered to use them?
- FH: LSTMs allow you to memorize the past, for my task the present and the future contain enough information for the prediction
- FH: future inroads to this project, use the past too so that the prediction fits more the "nature" of the vessel
- FH: In this work we just use fully-connected layers. We kind of "skip" the convolutional layers as we already have a feature map as an input for the NN

- Question: Are there ideas to include physics constraints, e.g. concerning Bernouli-Effect in the modeling?
- FH: not a focus yet
- FH: physical constraints are part of the "control path" of the vessel (provided by collaborators)
- RJ: control path fixed for the model or more dynamic depending on these physical constraints.
- FH: control path is parametrized
- FH: RL agent is provided these parameters

- Question: But do you feed in multiple past observations as a single state?
- OO: No, we use the current state only.

# Feedback

## Share something with us that you liked or that you learned during todays seminar :+1:

* Generally: Great presentation, easy to follow, I learned a lot! (LS)
* Nice introduction into RL. Even understandable for beginners. (NP)
* HackMD is good but I did not initially know what you meant by line 71, great work thanks

* That simple RL-agents are not too hard to put to work, but it gets complicated when several agents are supposed to interact

* DDPG might be more suitbale for my project compared to DQN

## Share something with us that you didn't like or that you'd like us to improve :-1:

* HackMD is good but I did not initially know what you meant by line 71, great work thanks

## Connection Details

Topic: RL Time: Apr 6, 2021 10:00 AM Amsterdam, Berlin, Rome, Stockholm, Vienna

Join Zoom Meeting https://us02web.zoom.us/j/86707148297?pwd=aHB0TXNTdnNjamdMT0dOcG9JaEhPZz09

Meeting ID: 867 0714 8297

Passcode: 244572

One tap mobile [phone],,86707148297#,,,,*244572# US (Washington DC) [phone],,86707148297#,,,,*244572# US (Chicago) Dial by your location [phone] US (Washington DC) [phone] US (Chicago) [phone] US (Houston) [phone] US (San Jose) [phone] US (New York) [phone] US (Tacoma) Meeting ID: 867 0714 8297 Passcode: 244572 Find your local number: https://us02web.zoom.us/u/kvUuizRoJ