---
# System prepended metadata

title: '[Pipeline of paper](https://arxiv.org/pdf/1912.01603.pdf)'

---

# [Pipeline of paper](https://arxiv.org/pdf/1912.01603.pdf)
- [name=作者：Jeff]
- [time=Sat, August 1, 2020 17:30]
## Outline
[TOC]
## Important keywords
1. **Continuous Control**
2. **Robotics arm** (FetchReach-v1)
3. **Goal based mission**
    - Far from several scenarios in D2C or PlaNet papers
    - Taking "Walker-walk" and "Ant" for example, the two agents learn from own body reward, instead of the Fetch series agents learning reward between arms and objects.
4. **Model-based Deep Reinforcement Learning**
    - Build environment model
6. **Learn behaviors by latent**
    - image input via autoencoder
7. **The name of 'Planning' or 'imagination'**
    - via RNN, LSTM, or GRU
## Build environment
### Simulator - [MuJoCo](http://www.mujoco.org/)
1) Apply for the MuJoCo license and get a accout number from a mail
2) Download th sutiable [version](https://www.roboti.us/index.html)
3) Run the "getid" to register own PC. After sucessfull registration, an activation key will lock own PC.
:::warning
- Access to MuJoCo official document
```
export MJLIB_PATH=/home/{user_name}/.mujoco/mujoco200_linux/bin/libmujoco200.so
export MJKEY_PATH=/home/{user_name}/.mujoco/mujoco200_linux/bin/mjkey.txt
```
- Access to MuJoCo files
```
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/jeff/.mujoco/mujoco200_linux/bin
```
:::
### [OpenAI Gym](https://github.com/openai/gym)
```
pip install gym
```
### Python requirement toolkit

## Training
Using the authors'[the code](https://github.com/yusukeurakami/dreamer-pytorch) who has developed dream to control
### Prepocessing
- Set Reward strategies
- Select camera position in simulator
- actor noise

### Main loop (until converge) 
- Initial hyperparameters
```
* Commom Adjusion
- action_repeat : 1 (do same action several times)
- action_noise : 0.15 (Add noise to make agent learn from real action instead of only learning from "images")
- batch_size : 50 (How many datas for each episode)
- chunk_size : 50 (slice the data from batch size)
- model_lr: 1e-3 (Model learning rate)
- actor_lr : 8e-5 (Actor model learning rate)
- value_lr : 8e-5 (Value model learning rate)
- adam_epsilon : 1e-7 (Adam optimizer epsilon value)
- grad_clip_norm : 100 (The rate of grad_clip_norm to avoid gradient explosion and vanishing)
- planning_horizon : 15 (How long does the rssm plan to imagine)
- discount : 0.99 (Planning horizon distance)
- free_nats : 3 (Free nats are applied to the normalization and overshooting (but not global) KL losses before averaging over elements in the distribution)
- bit_depth : 5 (5 bit depth mean that only 32 colors in an image)
- seed_episodes : 5 (precollect episodes experience)

* Some setting:
- max_episode_length : 1000 (How long does an episode have? In the "fetach", there are 50 longs in an episodes.)
- experience_size : 500000 (the storation of experience)
- cnn_activation_function : relu
- dense_activation_function : elu
- embedding_size : 1024 (train rssm model input dim)
- hidden_size : 200 (train through linear layers' dim)
- belief_size : 200 (Belief/hidden size)
- state_size : 30 (State/latent size)
- collect_interval: 100 (Collect interval)
- overshooting_distance : 50 (Latent overshooting distance/latent overshooting weight for t = 1)
- overshooting_kl_beta : 0 (Latent overshooting KL weight for t > 1 (0 to disable))
- overshooting_reward_scale : 0 (Latent overshooting reward prediction weight for t > 1 (0 to disable))
- global_kl_beta : 0 (Global KL weight (0 to disable))
- disclaim : 0.95 (discount rate to compute return)
- optimisation_iter : 10 (Planning optimisation iterations)
- candidates : 1000 (Candidate samples per iteration)
- top_candidates : 100 (Number of top candidates to fit)
- worldmodel_LogProbLoss : default = True (use LogProb loss for observation_model and reward_model training)
```
- Intial several models
- Collect seed episodes
- Learn from episodes

## Testing

## Result

## Related work
- The concept like dreamer
    - [Dyna, an Integrated Architecture
for Learning, Planning, and Reacting
](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.48.6005&rep=rep1&type=pdf)
- [MuJoCo](https://homes.cs.washington.edu/~todorov/papers/TodorovIROS12.pdf)
- Survey
    - [A brief survey of deep reinforcement learning](https://arxiv.org/pdf/1708.05866.pdf)
    - [Reinforcement Learning in Robotics: A Survey](https://www.ias.informatik.tu-darmstadt.de/uploads/Publications/Kober_IJRR_2013.pdf)
    - [Survey of Robotic Manipulation Studies Intending Practical Applications in Real Environments: —Object Recognition, Soft Robot Hand, Challenge Program and Benchmarking](https://www.researchgate.net/publication/326401265_Survey_of_Robotic_Manipulation_Studies_Intending_Practical_Applications_in_Real_Environments_-Object_Recognition_Soft_Robot_Hand_Challenge_Program_and_Benchmarking-)
- Robotics
    - [Reacher](https://gym.openai.com/envs/Reacher-v2/) (OpenAI Gym)
    - [Fetch Robotics](https://fetchrobotics.com/)
    - [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://arxiv.org/pdf/1802.09464.pdf)
    - [Asymmetric Actor Critic for Image-Based Robot Learning](https://arxiv.org/pdf/1710.06542.pdf)
    - [TEMPORAL DIFFERENCE MODELS:MODEL-FREE DEEP RL FOR MODEL-BASED CONTROL](https://arxiv.org/pdf/1802.09081.pdf)
    - [Pick and Place Without Geometric Object Models](https://arxiv.org/pdf/1707.05615.pdf)
- Deep reinforcement learning
    1) Model-based DRL which learn from latent (Image-based)
        - [PlaNet](https://arxiv.org/pdf/1811.04551.pdf)
        - [I2A](https://arxiv.org/pdf/1707.06203.pdfs)
    2) Model-based DRL which learn with planning
        - [Muzero](https://arxiv.org/pdf/1911.08265.pdf)
    3) Else model-based DRL
        - [BREMEN](https://arxiv.org/pdf/2006.03647.pdf)
    4) About reward (dense and sparse)
        - Sparse Reward:
            1) [HER](https://arxiv.org/pdf/1707.01495.pdf)
            2) [SHER](https://arxiv.org/pdf/2002.02089.pdf)
            3) [PlanGAN](https://arxiv.org/pdf/2006.00900.pdf)
    5) [KL divergence in DRL](https://arxiv.org/pdf/1905.01240.pdf)
- Dynamics learning
    - [Improving PILCO with Bayesian Neural Network Dynamics Models](http://mlg.eng.cam.ac.uk/yarin/PDFs/DeepPILCO.pdf)
    - [Deep reinforcement learning in a handful of trials using probalistic Dynamics models](https://arxiv.org/pdf/1805.12114.pdf)
- Robotics with DRL
    - Image-based model-based deep reinforcement learning
        - [SOLAR](https://arxiv.org/pdf/1808.09105.pdf)
        - [Deep Visual foresight for planning robot motion](https://arxiv.org/pdf/1610.00696.pdf)
        - [Visual foresight: model-based deep reinforcement learning for vision-based robotic control](https://arxiv.org/pdf/1812.00568.pdf)
        - [E2C](https://arxiv.org/pdf/1506.07365.pdf)
        - [RCE](https://arxiv.org/pdf/1710.05373.pdf)
    - Else
        - [Model-Based Planning with discrete and continuous actions](https://arxiv.org/pdf/1705.07177.pdf)
        - [Leveraging Deep Reinforcement Learning for Reaching Robotic Tasks](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8014805)
        - [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://arxiv.org/pdf/1708.02596.pdf)
        - [Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods](https://arxiv.org/pdf/1802.10264.pdf)
        - [Review of Deep Reinforcement Learning for Robot Manipulation]( https://arxiv.org/pdf/1610.00633.pdf) 
        - [KEEP DOING WHAT WORKED: BEHAVIOR MODELLING PRIORS FOR OFFLINE REIN-FORCEMENT LEARNING](https://arxiv.org/pdf/2002.08396.pdf)