# Machine Learning method

<style>
.red {
color: red;
font-weight:bold;
}
.blue {
color: blue;
font-weight:bold;
}

.top-right {
position: absolute;
top: 8px;
right: 16px;
color: blue;
}

</style>

### life time variable

$l_{(t+1)}: lifetime(t+1)$
$l_{(t)}: lifetime(t)$
$a_{(t+1)}: action(t+1)$
$d_{(t+1)}: download(t+1)$
$l_{max}:$ max lifetime = 5
1. $l_{(t+1)} \leq (l_{(t)} - 1 ) \cdot a_{(t+1)} + l_{max} \cdot d_{(t+1)}$
2. $l_{(t+1)} \geq (l_{(t)} - 1 ) \cdot a_{(t+1)}$
3. $l_{(t+1)} \geq l_{max} \cdot d_{(t+1)}$


### Choose action in testing
$(S_t, A_t, P_t) + (A_t, T_{t+1})$
因為$A_t$基本上就是$S_{t+1}$
所以可以看成$(S_t, A_t, P_t) + (S_{t+1}, T_{t+1})$

### Usable variables
* $S_{t+1}$: Cache State
* $T_{t+1}$: Time fraction (one hot encoding)
* $P_{t+1}$: map popularity (need to peek)
* $R_{t+1}$: remain_time

## NEW MAP LEN:150

The numbers of training data: 160
The numbers of testing data: 3

Big: 12
Small: 39
Total: 51

B_size_list = [10]
D_size_list = [6]
lifetime_list = [5]
Reward_hit_list = [2]

### Methods
1. $(S_{t+1}, T_{t+1})$
2. $(S_{t+1},P_{t}, T_{t+1})$
3. $(S_{t+1},P_{t}, T_{t+1}, G_{t})$ <Only do reversed version>
4. $(L_{t+1},P_{t}, T_{t+1}, G_{t})$ <只存五種地圖>
$L_{t+1}=L_{t+1}*S_{t+1}$
5. $(D_{t+1},P_{t}, T_{t+1}, G_{t})$ <仍有負值、Loss變小、較不會重複存檔>

Reversed:
1. $(S^r_{t+1}, T_{t+1})$
2. $(S^r_{t+1},P_{t}, T_{t+1})$
3. $(S^r_{t+1},P_{t}, T_{t+1}, G_{t})$ <仍有負值、Loss變小、較不會重複存檔>

* 強制正weight會導致loss大，並且無法逼近預期的值。

## New test file (=18 fractions)

### Alpha_ 2:10
| Methods | Cost | Used maps |
| ------- | ----- | --------------------------------------------------------- |
|Local optimal|5112||
|Total optimal|4121||
|Big only popular|5208||
|popular|9324||
|LFU|12349||
|LRU|14346||
|<span class="red">RL method(LR:0.001, gamma:0.7)</span>|<span class="red">4812</span>||


* G7_2_rl target=100 [400] r:4806

* G7_2_rl target=160 [3480] r:4806

* G8_2_rl target=160 [740] r:4814

* G9_2_rl target=160 [280] r:4850

### Alpha_ 5:10
| Methods | Cost | Used maps |
| ------- | ----- | --------------------------------------------------------- |
|Local optimal|8975||
|Total optimal|7029||
|Big only popular|12437||
|popular|9324||
|LFU|12349||
|LRU|14346||
|<span class="red">RL method New (LR:0.0001, gamma:0.7)</span>|<span class="red">8953</span>||

* 5:10 **[1840]**

* 8:10 **[260]** r:9240

## Veh 200

---
## Test case(=3)

### Alpha_ 2:10
| Methods | Cost | Used maps |
| ------- | ----- | --------------------------------------------------------- |
|Local optimal|5134||
|Local optimal peek|4297||
|<span class="blue">$(S_{t+1}, T_{t+1})$ immediate reward</span>|<span class="blue">6823</span>||
|Total optimal|4086||
|Big only popular|5236||
|popular|11167||
|LFU|12876||
|LRU|14940||
|<span class="blue">$(S^r_{t+1},P_{t}, T_{t+1}, G_{t})$ use random{4\*r+ 1\*OPT}</span>|<span class="blue">11125</span>||
|<span class="blue">$(S^r_{t+1},P_{t}, T_{t+1}, G_{t})$</span>|<span class="blue">11999</span>||
|$(S_{t+1}, T_{t+1})$|19350||
|RL GAMMA:0.7 Target replace= 160|4735||
|RL GAMMA:0.7 Target replace= 100|||
|RL GAMMA:0.8 Target replace= 160|||
|RL GAMMA:0.8 Target replace= 100|4749||
|RL GAMMA:0.9 Target replace= 160|||
|RL GAMMA:0.9 Target replace= 100|4796||

* RL GAMMA:0.7 Target replace= 160 **[560]**

* RL GAMMA:0.8 Target replace= 100

* RL GAMMA:0.9 Target replace= 100

### Alpha_ 5:10
| Methods | Cost | Used maps |
| ------- | ----- | --------------------------------------------------------- |
|Local optimal|9224||
|Local optimal peek|||
|Total optimal|6975||
|Big only popular|12367||
|popular|11167||
|<span class="red">$(S_{t+1}, T_{t+1})$ (Gamma:0.9)</span>|<span class="red"></span>||
|<span class="red">$(S_{t+1}, P_{t})$ (Gamma:0.9)</span>|<span class="red"></span>||
|<span class="red">RL method New (LR:0.0001, gamma:0.7)</span>|<span class="red">9157</span>||

* 5:10 lr:0.0001

### Alpha_ 8:10
| Methods | Cost | Used maps |
| ------- | ----- | --------------------------------------------------------- |
|Local optimal|9465||
|Local optimal peek|||
|Total optimal|7208||
|Big only popular|19499||
|popular|11167||
|<span class="red">RL method New (LR:0.001, gamma:0.8)</span>|<span class="red"></span>||
|<span class="red">RL method New (LR:0.001, gamma:0.7)</span>|<span class="red">9413</span>||

### Questions
1. testing: MIN時有負數
Front_vale: 34.14666799999999, Q_value: -4402.586396225645

## Maximize

### Alpha_ 5:10
| Methods | Profit | Used maps |
| ------- | ----- | --------------------------------------------------------- |
|Local optimal|15012||
|Local optimal peek|16942||
|Total optimal|16983||
|Big only popular|||
|popular|||
|LFU|||
|LRU|||
|<span class="blue">$(S^r_{t+1},P_{t}, T_{t+1}, G_{t})$ use random{4\*r+ 1\*OPT}</span>|<span class="blue">15028</span>||
|<span class="blue">$(S^r_{t+1},P_{t}, T_{t+1}, G_{t})$ use random{8\*r+ 1\*OPT}</span>|<span class="blue">13281</span>||
|<span class="blue">$(S^r_{t+1},P_{t}, T_{t+1}, G_{t})$ {1\*OPT}</span>|<span class="blue">13873</span>||
|<span class="red">RL method </span>|<span class="red"></span>||

## <span class="red">Average =3 file</span>

### Alpha 2:10
| Methods | Cost | Used maps |
| ------- | ----- | --------------------------------------------------------- |
|Local optimal|51.50||
|Local optimal peek |41.81||
|Total optimal|40.09||
|Big only popular|50.80||
|popular|93.81||
|LRU|134.03||
|LFU|132.21||
|RL 0.8, lr:0.0001|47.00||
|ML peek|41.98||

* 2:10

### Alpha 5:10
| Methods | Cost | Used maps |
| ------- | ----- | --------------------------------------------------------- |
|Local optimal|91.63||
|Total optimal|68.49||
|Big only popular|121.75||
|popular|93.81||
|LRU|134.03||
|LFU|132.21||
|RL g=0.8 lr:0.001|88.0||
|RL g=0.8 lr:0.0001|||
|RL g=0.7 lr:0.001 |88.4|一開始|
|RL g=0.9 lr:0.001 |待補||
|RL g=0.9 lr:0.0001 |待補||

#### Cache:

* Local

> [2022-10-31 09:46:51,214][line: 526] ==> cache size: 8 | average ep_r: 107.830000
[2022-10-31 09:46:53,610][line: 526] ==> cache size: 9 | average ep_r: 99.070000
[2022-10-31 09:46:56,276][line: 526] ==> cache size: 10 | average ep_r: 91.600000
[2022-10-31 09:46:58,963][line: 526] ==> cache size: 11 | average ep_r: 83.300000
[2022-10-31 09:47:02,097][line: 526] ==> cache size: 12 | average ep_r: 76.990000

* RL
> [2022-10-31 09:42:23,389][line: 526] ==> cache size: 8 | average ep_r: 106.190000
[2022-10-31 09:42:25,913][line: 526] ==> cache size: 9 | average ep_r: 96.740000
[2022-10-31 09:42:28,979][line: 526] ==> cache size: 10 | average ep_r: 87.980000
[2022-10-31 09:42:32,487][line: 526] ==> cache size: 11 | average ep_r: 79.650000
[2022-10-31 09:42:35,001][line: 526] ==> cache size: 12 | average ep_r: 74.

* Total
> [2022-10-31 09:45:36,804][line: 413] ==> cache size: 8 | average ep_r: 85.880000
[2022-10-31 09:45:38,911][line: 413] ==> cache size: 9 | average ep_r: 76.660000
[2022-10-31 09:45:41,230][line: 413] ==> cache size: 10 | average ep_r: 68.490000
[2022-10-31 09:45:43,397][line: 413] ==> cache size: 11 | average ep_r: 60.990000
[2022-10-31 09:45:45,902][line: 413] ==> cache size: 12 | average ep_r: 54.260000

#### Dowload:

* Local
> [2022-10-31 09:52:57,787][line: 526] ==> download size: 2 | average ep_r: 95.990000
[2022-10-31 09:53:00,274][line: 526] ==> download size: 4 | average ep_r: 91.740000
[2022-10-31 09:53:03,019][line: 526] ==> download size: 6 | average ep_r: 91.600000
[2022-10-31 09:53:05,799][line: 526] ==> download size: 8 | average ep_r: 91.380000
[2022-10-31 09:53:08,013][line: 526] ==> download size: 10 | average ep_r: 91.530000

* RL
> [2022-10-31 09:53:53,288][line: 526] ==> download size: 2 | average ep_r: 91.550000
[2022-10-31 09:53:57,568][line: 526] ==> download size: 4 | average ep_r: 87.760000
[2022-10-31 09:54:00,799][line: 526] ==> download size: 6 | average ep_r: 87.980000
[2022-10-31 09:54:04,100][line: 526] ==> download size: 8 | average ep_r: 88.090000
[2022-10-31 09:54:06,876][line: 526] ==> download size: 10 | average ep_r: 88.090000

#### Graphs:

* 5:10 , g =0.8

* 5:10 , g =0.7

* 5:10 , g =0.9, lr:0.0001

### Alpha 8:10
| Methods | Cost | Used maps |
| ------- | ----- | --------------------------------------------------------- |
|Local optimal|94.030||
|Total optimal|70.74||
|Big only popular|192.70||
|popular|93.81||
|LRU|134.03||
|LFU|132.21||
|RL |92.00||

* 8:10

## MAP:

* New mix (seed: 5, p: 5)

* New mix (seed: 5, p: 1)
* Original

## 改成求最大化

* $H_s$ = 小張地圖cache : +10
* $H_b$ = 大張地圖cache : +2

* !!問題!! 如果

RL : 舊
RL : 新

---

* 嘗試資料

## veh 50

### Alpha_ 2:10
| Methods | Cost | Used maps |
| ------- | ----- | --------------------------------------------------------- |
|Local optimal|1917||
|Total optimal|1824||
|Big only popular|2662||
|popular|5509||
|<span class="blue">$(S^r_{t+1},P_{t}, T_{t+1}, G_{t})$ positive weight</span>|<span class="blue">3510</span>||
|<span class="blue">$(S^r_{t+1},P_{t}, T_{t+1}, G_{t})$ </span>|<span class="blue">4384</span>||

---

## time=5

### Alpha_ 2:10
| Methods | Cost | Used maps |
| ------- | ----- | --------------------------------------------------------- |
|Local optimal|1672||
|Total optimal|1466||
|Big only popular|||
|popular|3057||
|<span class="blue">$(S^r_{t+1},P_{t}, T_{t+1}, G_{t})$ positive weight</span>|<span class="blue"></span>||
|<span class="blue">$(S^r_{t+1},P_{t}, T_{t+1}, G_{t})$ </span>|<span class="blue">2937</span>||

---