# Pytorch dqn & ddqn
##### 1. DQN與DDQN簡單介紹
##### 2. 程式搭建部分
##### 2. 比較兩者訓練結果
##### 4. 參考資料
##### 5. 程式碼連結
---

## 一、DQN & DDQN 簡單介紹
DQN其實就是Deep Learning + Reinforcement Learning,藉由神經 網路的部分來預測接下來將採取的動作，這麼做不但可以省去儲存龐大Q表的空間更可以計算更複雜的環境狀態來增強強化學習的效果。

而DDQN與DQN最大的差異就在他們之間的更新部分，DQN的實際訓練的網路(eval_net)與目標網路(target_net)的更新同時執行的，而DDQN則是將我們的目標網路更新給延後，更新時會直接把 evaluation network 的參數整組複製過來。

接著簡單說明一下DQN的訓練概念，由於每個狀態之間都存在著時序關係，為了不讓這個時序關係影響到我們的訓練結果，我們會採用經驗收集法(store_transition)來儲存各個狀態、動作、獎勵等，然後採取隨機回傳來打亂彼此的時序關係並執行訓練。

---

## 二、程式比較與說明
1. #### 匯入所需要的模組與設定參數

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import gym

# Hyper Parameters
BATCH_SIZE = 32
LR = 0.01                   # learning rate
EPSILON = 0.9               # greedy policy
GAMMA = 0.9                 # reward discount
TARGET_REPLACE_ITER = 100   # target update frequency
MEMORY_CAPACITY = 2000
env = gym.make('CartPole-v0')
env = env.unwrapped
N_ACTIONS = env.action_space.n
N_STATES = env.observation_space.shape[0]
ENV_A_SHAPE = 0 if isinstance(env.action_space.sample(), int) else env.action_space.sample().shape

2. #### 搭建我們神經網路

class Net(nn.Module):
    def __init__(self, ):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(N_STATES, 50)
        self.fc1.weight.data.normal_(0, 0.1)   # initialization
        self.out = nn.Linear(50, N_ACTIONS)
        self.out.weight.data.normal_(0, 0.1)   # initialization

    def forward(self, x):
        x = self.fc1(x)
        x = F.relu(x)
        actions_value = self.out(x)
        return actions_value

3. #### 接著搭建我們的DQN也就是智能體(Agent)

class DQN(object):
    def __init__(self):
        self.eval_net, self.target_net = Net(), Net()
        self.learn_step_counter = 0                                     # for target updating
        self.memory_counter = 0                                         # for storing memory
        self.memory = np.zeros((MEMORY_CAPACITY, N_STATES * 2 + 2))     # initialize memory
        self.optimizer = torch.optim.Adam(self.eval_net.parameters(), lr=LR)
        self.loss_func = nn.MSELoss()

    def choose_action(self, x):
        x = torch.unsqueeze(torch.FloatTensor(x), 0)
        # input only one sample
        if np.random.uniform() < EPSILON:   # greedy
            actions_value = self.eval_net.forward(x)

            action = torch.max(actions_value, 1)[1].data.numpy()
           
            action = action[0] if ENV_A_SHAPE == 0 else action.reshape(ENV_A_SHAPE)  # return the argmax index
        else:   # random
            action = np.random.randint(0, N_ACTIONS)
            action = action if ENV_A_SHAPE == 0 else action.reshape(ENV_A_SHAPE)
        return action

    def store_transition(self, s, a, r, s_):
        transition = np.hstack((s, [a, r], s_))
        # replace the old memory with new memory
        index = self.memory_counter % MEMORY_CAPACITY
        self.memory[index, :] = transition
        self.memory_counter += 1

    def learn(self):
        if len(self.memory) < 32:
            return
        # target parameter update
        if self.learn_step_counter % TARGET_REPLACE_ITER == 0:
            self.target_net.load_state_dict(self.eval_net.state_dict())
        self.learn_step_counter += 1
        
        # sample batch transitions
        sample_index = np.random.choice(MEMORY_CAPACITY, BATCH_SIZE)
        b_memory = self.memory[sample_index, :]
        
        
        b_s = torch.FloatTensor(b_memory[:, :N_STATES])       
        b_a = torch.LongTensor(b_memory[:, N_STATES:N_STATES+1].astype(int))
        b_r = torch.FloatTensor(b_memory[:, N_STATES+1:N_STATES+2])
        b_s_ = torch.FloatTensor(b_memory[:, -N_STATES:])
        
        # q_eval w.r.t the action in experience
        q_eval = self.eval_net(b_s).gather(1, b_a)  # shape (batch, 1)
        q_next = self.target_net(b_s_).detach()     # detach from graph, don't backpropagate
        q_target = b_r + GAMMA * q_next.max(1)[0].view(BATCH_SIZE, 1)   # shape (batch, 1)
        loss = self.loss_func(q_eval, q_target)

        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

## 三、比較兩者訓練差異
#### 1.DQN的訓練結果

DQN的訓練過程通常要到差不多270~300步時才能有較好的成果，但也能發現非常地不穩定，而DDQN則可以稍微快一些的達到訓練效果，也比DQN穩定一些。

---

## 四、參考資料
* https://pyliaorachel.github.io/blog/tech/python/2018/06/14/deep-q-learning.html
* https://medium.com/pyladies-taiwan/reinforcement-learning-%E9%80%B2%E9%9A%8E%E7%AF%87-deep-q-learning-26b10935a745
* https://morvanzhou.github.io/tutorials/machine-learning/reinforcement-learning/

## 五、程式碼連結
https://drive.google.com/drive/u/1/folders/1IoU4V9mG37hVNxU9fBhK0UX30js5UaOP