# 6/7 Reinforcement Learning & ICRA 2019 workshop survey
###### tags:`explore`

[toc]

:::info

## Before Learning
# MDP (Markov Decision Process) 馬可夫決策過程
* MDP的一個重要觀念：”未來只取決於當前”
* 為什麼強化學習會跟MDP有關呢？我們先看什麼是State(狀態)
* 因為我們的大腦一開始並不知道環境的狀態是怎麼樣，所以只能從以前所經歷的observation，action，reward跟現在所得到的observation, reward來去當作現在的狀態
* 那如果我們要去估計下一個狀態(St+1)是怎麼樣的，是不是就要把S1~St的所有狀態給考慮進去，這樣模型便會非常的大，這時候Markov假說就有用了，Markov說的：未來只取決於當前，所以我們可以假設 下一個狀態只跟現在這個狀態有關，有這個假設就可以把模型給縮小，不過這個假設也只是理想的狀況下。

:::danger
補充：有限的MDP是由一個四個元素所構成 M=(S,A,P,R)。S表示狀態集，A表示動作集，P表示狀態轉移機率，R表示期望Reward回報值，而其實這四種元素在RL上都可以看到，因此可以把RL想像成是一種MDP模型

:::
:::
---
## Reinforcement Learning
:::info
* Supervised Learning (監督式學習): 訓練資料必需有input跟label(答案)，讓機器去mapping出一個最好的模型，常用的演算法為分類、回歸等演算法
* Unsupervised Learning (非監督式學習): 訓練資料只有input沒有label(答案)，讓機器從訓練資料中找出規則，常用的演算法為集群演算法
* Reinforcement Learning (強化學習): 從現在的環境來決定行為，是一種互動式的學習過程
:::
:::success

* 從上圖可以看出來，大腦代表agent，地球代表environment(環境)，reward(獎勵)就是環境所提供的反饋，reward由模型設計者定義，reward的定義對強化學習來說是個很重要的一環。以下用個簡單的例子來解釋。
* Agent部份(大腦)
* 簡單來說，就是訓練出一個agent(大腦)可以去適應environment(環境)
* 會將environment環境每一個時間點的observation(觀察)的集合當作環境的狀態(State)
* 從環境的狀態(State)跟reward(獎勵)再去選擇一個最好的action(動作)，稱為policy(策略)
* Environment部份(環境)
* 簡單來說，就是一個環境在不同的狀態下會有不同的情況，並將這些情況告訴大腦，讓大腦可以去學習。
* 會接收Agent執行的action(動作)，並吐出reward跟observation給agent。
:::
:::success
# DQN Deep Q-learning Network 深層Q-learning 網路
因為Q-learning如果是用在狀態比較多的情況(像是Atari game那種有非常多的情況)，會導致Q-learning無法把所有的狀態的Q值都計算出來，要計算出來記憶體也可能會爆炸，所以就想出DQN，RL+DL的學習方法，設計出一個神經網路，只需要將狀態輸入，輸出的是每個動作的Q值，這樣就可以解決記憶體爆炸的情況，這個就是Value function approximation
然後我們可以思考一下，神經網路的訓練就是要找一個最好的參數來優化loss function，然後要訓練就必須要有training data，input X跟label Y，我們現在將State當作input了，那我們要那裡找label呢？我們來看看Q-learning的更新公式
我們前面有說，中間那個就是所謂的Target Q，那我們把那個當作Label，不就代表，我們的Q function就會越來越接近Target Q function了嗎？所以我們就可以定義出Loss function，Label也定義出來了，那這樣就可以做Neural network的訓練了。

:::
:::warning
邊紅色線為初始狀態的第一步，如果不做這一步的話會沒辦法把 St, at, rt, S(t+1)給算出來並存放至記憶體裡面，第一步初始化做完之後，後面就是藍色線的flow，通過環境來儲存St, at, rt, S(t+1)，並將參數丟給對應的網路來計算LOSS Function，最後再更新網路的參數，一直不斷的重覆更新就可以找出最好的Q Function了。

:::
:::warning
# Conclusion
* 對話系統：有些已經把RL用在對話系統上，利用互動式學習，隨著時間不斷的提升對話系統
* 醫療：利用RL來尋找最佳的治療方案
* Google auto ML： 使用RL來為計算機視覺和語言建模生成神經網路架構
* 自動駕駛
:::

* [refer site_1](https://medium.com/%E9%9B%9E%E9%9B%9E%E8%88%87%E5%85%94%E5%85%94%E7%9A%84%E5%B7%A5%E7%A8%8B%E4%B8%96%E7%95%8C/%E6%A9%9F%E5%99%A8%E5%AD%B8%E7%BF%92-ml-note-reinforcement-learning-%E5%BC%B7%E5%8C%96%E5%AD%B8%E7%BF%92-dqn-%E5%AF%A6%E4%BD%9Catari-game-7f9185f833b0)
* [refer site_2](https://morvanzhou.github.io/tutorials/machine-learning/ML-intro/4-02-RL-methods/)
* [refer site_3](https://www.kdnuggets.com/2018/03/5-things-reinforcement-learning.html)

---

# International Conference on Robotics and Automation(ICRA) Workshops

### [Opportunities and Challenges in Soft Robotics Across Length Scales](https://sites.google.com/view/soft-robotics-workshop/abstract)

:::info
* Soft robots are machines that can actively control their geometries via elastic deformations. This technology can provide a wide range of opportunities to create machines with unprecedented mechanical functionalities, as well as robots that are intrinsically safe to interact with human beings.
:::
:::success
* Recent advancements of soft robots across all length scales
:::

---
### [The Future of Aerial Robotics: Challenges and Opportunities](https://www.aerial-robotics-workshop.com/)

:::info
* The analysis of the evolution of aerial robotics technologies and applications in order to identify the most relevant challenges and opportunities in the next decade – and beyond – both for the academia and industry including developers and end users.
:::
:::success
* increase the duration and the range of flight of current aerial robotics platforms
* morphology shaping
* bio-inspiration
* soft-robotics
* dexterity and accuracy in aerial manipulation.
:::
---
### [Advances and Challenges on the Development, Testing and Assessment of Internet of Robotic Things (IoRT): Experiences from Engineering and Human Science Research
](https://sites.google.com/view/iotrobotworkshopicra2019/call-for-contributions)
:::info
* Efficiently combining with Internet of Things (IoT) systems, the consolidation promises revolutionizing potential with the later providing large quantity of information services over pervasive networks, enabling intelligent devices to use both local and distributed ’intelligence’ for optimized action, interaction and autonomous behaviour.
* Internet of Robotic Things (IoRT) on assistant robots regarding development, testing and assessment from both robotics and IoT communities.
:::
:::success
* IoT technologies that enhance service robots’ capability
* Computation offloading / Service migration on IoT robot system
* Rehabilitation robotics with IoT applications
* Physically assistive devices / Wearable robotics / Welfare technology combined with IoT technologies
* Internet of Robotic Things in transportation
:::
---
### [Long-Term Human Motion Prediction](https://motionpredictionicra2019.github.io/index.html)

:::info
* Accurate long-term predictions of human movement trajectories, body poses, actions or activities may significantly improve the ability of robots to plan ahead, anticipate the effects of their actions or to foresee hazardous situations.
* The topic has received increasing attention in recent years across several scientific communities with a growing spectrum of applications in service robots, self-driving cars, collaborative manipulators or tracking and surveillance.
:::
:::success
* Motion trajectory prediction in 2D and 3D
* Predicting articulated human motion
* Early action and activity recognition
* Motion and Task Planning in dynamic environments considering motion predictions
* Anticipation of group and crowd motion
* Human motion prediction and safety
* Human-Robot Interaction considering predictions
* Evaluation of prediction algorithms: datasets, metrics and benchmarks
* Predictive planning and control
* Applications of motion prediction techniques
* Visual scene prediction
:::
---
### [Rhetoric and Robotics. An exercise to shape the values of the discipline of robotics.](http://rhetoricra2019.ulb.be/)
:::info
* Does a robot decide (and not just select)? Does a robot feel (and not just touch)?
* To produce a vivid description of a given object. This exercice is known for creating mental images and revealing a specific perception of the object described.
:::
:::success
* Wording Robotics
* Rhetorial style in Robotics:various conceptions of language and rationally
* Speaking about Robots: my trilingual daily challenge
:::
---
### [Underwater Robotics Perception](http://icra-2019-uwroboticsperception.ge.issia.cnr.it/)

:::info
* in underwater scenarios perception tasks have proved to be particularly challenging
* Common sensor feedback in Remote Operated Vehicles (ROVs) and Autonomous Underwater Vehicles (AUVs) are optical cameras and sonars; but camera images suffer from limited range and rapid degradation depending on light behavior and water turbidity, and sonars often lack the resolution needed
* Underwater Robotics Perception
* development of Unmanned Underwater (UUVs) and Remote Operating Vehicles (ROVs)
:::
:::success
* Object detection and recognition in different sensor domains (stereo vision, lasers, acoustic sensors i.e. sonars, multibeam).
* Image enhancement and distortion removal (dehazing, color correction).
* Sensor fusion and signal processing.
* Environment modeling (bathymetry, 3D representation and mapping, SLAM).
* Simulated environments and continuous system integration (hardware-in-the-loop, synthetic data generation, simulation to real-world transition).
* Deep learning practices and machine learning pipelines in any of the above topics.
* Heavily tested systems in field trials and best practices for deployment and data management.

:::
---
### [Second International Workshop on Lines, Planes and Manhattan models for 3-D Mapping (LPM 2019)](https://seis.bristol.ac.uk/~csadc/LPM19/)

:::info
* LPM 2019 aims to bring together researchers working on the use of lines, planes and Manhattan models in deriving 3-D maps of physical environments.
* The models have been used successfully in Robotics using a variety of sensors including LiDAR, monocular and binocular vision, and more recently, RGB-D sensors.
:::
:::success
* Using Planes for 3D Reconstruction, Place Recognition and Scene Understanding
* The Role of Planes and Edges in Lidar-Inertial Integration
* Aided INS with Geometric Features: Analysis, Algorithms, and Challenges
* Using line-based wind turbine representations for UAV localisation during autonomous inspection
* Low-Drift, Efficient Visual Odometry and SLAM Utilizing Environmental Structures
* Real-Time Structure and Object-Model Aware Sparse SLAM
:::
---
### []()
:::info

:::
:::success

:::
---