---
# System prepended metadata

title: Taxi-v2

---

# Taxi-v2

[TOC]

&emsp;
&emsp;
## 訓練次數不同，Q-table 的差異
- 訓練約 2500 次後 [Q-table](https://hackmd.io/QOHobDNpSWOR264loGjv3g) 不會再改變
- 訓練次數低於 900 次所獲得的 Q-table ，在測試時可能會進入無止盡的循環。

&emsp;
&emsp;
訓練800次時，測試遇到的狀況 : 
&emsp;

![](https://i.imgur.com/Wy46OXB.png)


```python=
state = env.encode(4,0,1,2) # (taxi row,taxi column,passenger index, destination index)
print("State : ",state)
```

State :  406

```python=
q_table[406]
```

array([ -6.11187636,  -6.15386317,  -6.1232991 ,  -6.04170499,-14.82243346, -15.14841173])

- 0: south
- 1: north
- 2: east
- 3: west
- 4: pickup
- 5: dropoff


&emsp;
&emsp;
## 更改 Reward 

&emsp;
原來的 Q-table_2500 (錯誤的接送行為的獎勵為 -10)

array([[ 0.        ,  0.        ,  0.        ,  0.        ,  0.        , 0.        ],
       [ 1.62261467,  2.9140163 ,  1.62261467,  2.9140163 ,  4.348907  , -6.0859837 ],
       [ 4.348907  ,  5.94323   ,  4.348907  ,  5.94323 ,  7.7147    ,-3.05677   ],
       ...,
       [ 7.7147    ,  9.683     ,  7.71469995,  5.94322993, -1.28530002,-1.28530008],
       [ 1.62261467,  2.9140163 ,  1.62261467,  2.9140163 , -7.37738533, -7.37738533],
       [14.3       , 11.87      , 14.3       , 17.        ,  5.3       ,5.3       ]])


&emsp;
&emsp;

將錯誤的接送行為的獎勵改成 -50

Q-table_2500 :


array([[  0.        ,   0.        ,   0.        ,   0.        ,0.        ,   0.        ],
       [  1.62261467,   2.9140163 ,   1.62261467,   2.9140163 ,4.348907  , -46.0859837 ],
       [  4.348907  ,   5.94323   ,   4.348907  ,   5.94323   ,7.7147    , -43.05677   ],
       ...,
       [  7.7147    ,   9.683     ,   7.7147    ,   5.94323   ,-41.2853    , -41.2853    ],
       [  1.62261467,   2.9140163 ,   1.62261467,   2.9140163 ,-47.37738533, -47.37738533],
       [ 14.3       ,  11.87      ,  14.3       ,  17.        , -34.7       , -34.7       ]])
       
       
&emsp;
&emsp;
            
將錯誤的接送行為的獎勵改成 -1  (與其他動作的獎勵相同)


array([[ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,0.        ],
       [ 1.62261467,  2.9140163 ,  1.62261467,  2.9140163 ,  4.348907  ,2.9140163 ],
       [ 4.348907  ,  5.94323   ,  4.348907  ,  5.94323   ,  7.7147    , 5.94323   ],
       ...,
       [ 7.7147    ,  9.683     ,  7.7147    ,  5.94323   ,  7.71469999, 7.71469997],
       [ 1.62261467,  2.9140163 ,  1.62261467,  2.9140163 ,  1.62261467, 1.62261467],
       [14.3       , 11.87      , 14.3       , 17.        , 14.3       ,14.3       ]])
       
   
&emsp;
&emsp;

將完成任務的獎勵改為 -20 時

儘管訓練100000次後 ( 每個episode 訓練的時間更長 )，每次測試都會進入無止盡的循環 。


&emsp;
&emsp;

## 更改 MAP

原來的 MAP

![](https://i.imgur.com/03dbrVK.png)

訓練2500次，測試100次 的結果 : 

Results After 100 episodes
Average timestep per Episode :12.21
Average Penalties per Episode : 0.0

- Average timestep per Episode 基本上介於 12~13 之間

&emsp;
&emsp;

將MAP的牆壁移除

![](https://i.imgur.com/sSYLA9o.png)

訓練2500次，測試100次 的結果 : 

Results After 100 episodes
Average timestep per Episode :10.73
Average Penalties per Episode : 0.0

- Average timestep per Episode 基本上介於 10~11 之間


&emsp;
&emsp;

增加牆壁量v1

![](https://i.imgur.com/c3q7HNv.png)

訓練2500次，測試100次 的結果 :

Results After 100 episodes
Average timestep per Episode :13.25
Average Penalties per Episode : 0.0

- Average timestep per Episode 基本上介於 13~14 之間

&emsp;
&emsp;


增加牆壁量 v2

![](https://i.imgur.com/KOrA8qt.png)

訓練2500次，測試100次 的結果 :

Results After 100 episodes
Average timestep per Episode :18.49
Average Penalties per Episode : 0.0

- Average timestep per Episode 基本上介於 18~19 之間
- 測試時有可能會進入無止盡的循環