---
# System prepended metadata

title: Nickname 測驗結果

---

# Nickname 測驗結果
## 檔案資訊:
* Testdata : **NicknameTestDataWithLabel.csv**
### 直接 test
* 原始 information:
![](https://i.imgur.com/40ae50c.png)

#### 結果
主要是正常及廣告，但句子過長且非中文(泰文、英文等)過多，故預測效果不好
* Accuracy:
![](https://i.imgur.com/wQNZgzx.png)

* 混淆矩陣:
![](https://i.imgur.com/5eMaNZZ.png)

* 預測錯誤:
![](https://i.imgur.com/TgcBYk1.png)
![](https://i.imgur.com/ayAe0Zu.png)

### 改善後流程
1.以 langid 過濾出中文 (45090 -> 26444)
2.斷詞丟入 model,以數量最多的 label 當作該 nickname label

* Information:
![](https://i.imgur.com/58frW03.png)

#### 結果

* result: 
**nickname_labeled_zh_210112_0055CC_210113_0732.csv**
![](https://i.imgur.com/nOyK8Vy.png)
https://docs.google.com/spreadsheets/d/1rC6c4M62UBgJ9Ktcj7-crKyB9P8ADBj0dBIMoZ6Nz_o/edit?usp=sharing

* Accuracy:
![](https://i.imgur.com/eoWpg64.png)


* 混淆矩陣
![](https://i.imgur.com/UIzzt6m.png)

* 結果
https://docs.google.com/spreadsheets/d/1ze0Rx4T9w1YK1TT9mxKqwWcs9iFlR8r_4F5845q-e7w/edit?usp=sharing
* 預測錯誤原因
    + 人名皆被判斷成 1 [ex: row 12.13.14 等]
    + 數美標錯( or 我們的模型比較嚴格) [ex: row 34.94 等]
    + 廣告帳號濾掉數字後，只剩下正常的詞彙 [ex: row 1114 等]
 
 ### 可以改善的方法 & 待確認問題
 1. 將過長的數字串列入判斷廣告條件
 Q: 人名皆判斷成涉政(之前的要求)，若想用自己名字當 nickname 也會被判斷成涉政
 
 # Update
* 修正廣告的帳號
* 去除重複中文暱稱的概況
* 蒙古內蒙古蒙古國距離
* 帳號去除人工流水號後情況(元豪)

## 修正廣告的帳號
#### Before
廣告訊息因濾除數字而只剩名詞被模型判斷為正常或涉政(人名)

![](https://i.imgur.com/EgDtaeN.png)

#### After
用正則表示式將含有 11 碼數字的帳號，在新增的 phone 欄位標示為 1，並直接標示為廣告

![](https://i.imgur.com/Z4tkgU1.png)

![](https://i.imgur.com/Yj9FYpD.png)

## 去除重複中文暱稱的概況
* Testdata : **NicknameTestDataWithLabel.csv**
##### 將原有的 Testdata drop 掉剩餘 5112 筆
 ![](https://i.imgur.com/PLH1YiH.png)
##### 重複的 nickname 舉例
![](https://i.imgur.com/srhSyBe.png)

### 結果
https://docs.google.com/spreadsheets/d/1Pju1ax6ebGnehCgzFwv-rryPn0-QZ_E6or5tCZ_CFBA/edit?usp=sharing

![](https://i.imgur.com/7llWm3c.png)
![](https://i.imgur.com/IXEdtP7.png)

#### 預測錯誤舉例
![](https://i.imgur.com/wexW1mp.png)


## 蒙古 vs 內蒙古 vs 蒙古國距離
經過XLNET embedding 後會產生 n*768 維向量，若 n 不同則無法直接以`cosine_similarity`比較
(進入 textCNN 前會 padding 至 15 維)
#### n 相同 (n = 3)
![](https://i.imgur.com/N7tzrH6.png)

#### n 相同 (n = 2)
![](https://i.imgur.com/WoWMAWg.png)



#### padding 至較長的詞彙長度
![](https://i.imgur.com/fL4QMVq.png)


#### padding 至 15 維
![](https://i.imgur.com/KJt4HZL.png)


#### padding 至 5 維
![](https://i.imgur.com/IhSMcbj.png)


### 總結
雖然因 padding 會影響不同維度間的相似度比較，但總結來說
```蒙古 vs 蒙古國``` 相似度最高，距離最近(因維度相同)
且
```內蒙古 vs 蒙古國``` `內蒙古 vs 蒙古` 相似度也都大於其他維度不同的詞彙
並且
同為地名的相似度也較高


# Update 2021.02.03
Test data: **NicknameLabelCheck.csv**
Test data information:

![](https://i.imgur.com/poDBFHk.png)

### 數美標的(labelNo) vs 我們人工標記(Correct)
#### 正確率
![](https://i.imgur.com/rPDs0A1.png)

#### 混淆矩陣
![](https://i.imgur.com/Z3rvXUK.png)

#### 數美標錯
https://docs.google.com/spreadsheets/d/1Ast7xHIRY_6Sl1FqA1RGWZRVvxYYR_tM1l3uVBaMsFc/edit?usp=sharing

### 模型預測 vs 我們人工標記(Correct)
#### 正確率
![](https://i.imgur.com/bTh1dPo.png)

#### 混淆矩陣
![](https://i.imgur.com/2pSKQGo.png)

#### 模型預測錯
https://docs.google.com/spreadsheets/d/173lbLwJpgXbRU0Y9CBOYYslDGmtSeRraZ1QYb87awEc/edit?usp=sharing

* 模型預測人名還是會全部預測為1 (尚未加入正常人名進去 Train)
![](https://i.imgur.com/NYlHrJn.png)

* 詞彙因為切詞被切開故無法正確預測
![](https://i.imgur.com/4ifcQ2k.png)
![](https://i.imgur.com/ZaHUjPA.png)
![](https://i.imgur.com/aqU7YWz.png)

* 綽號模型無法正確預測為涉政
![](https://i.imgur.com/sFu0FH6.png)
![](https://i.imgur.com/DP2WX0U.png)

# Update 2021.03.10
## Cross Validation
拿之前人工標記過的
**NicknameLabelCheck.csv** (1720筆)
加上
**word_label_data_remove_long_word_210222_add normal name_and_Label.csv** (40910筆)
做 5-fold

```
word_label_data_remove_long_word_210222_add normal name_and_Label.csv 

total: 40910
0.0    36509
1.0     2662
4.0      816
2.0      378
5.0      326
3.0      218
```
### fold 0
```
train fold data: 42286
0.0    37038
1.0     2929
4.0      937
2.0      696
5.0      410
3.0      275

test fold data: 344
0    128
2     79
1     72
4     27
5     26
3     12
```
```
predict Accuracy: 0.6366279069767442
數美 Accuracy:  0.7325581395348837

predict    0   1   2  3  4   5
correct                       
0        101   8   1  2  9   3
1         42  25   0  0  4   0
2         18   0  49  0  8   2
3          8   1   0  3  0   0
4         14   0   4  0  7   2
5          1   0   0  0  1  21
```
Predict Error: 
https://docs.google.com/spreadsheets/d/1JQKdDzpLQsqMjJddF3zy4zqHKX-NHMqlzhcG8wJukcs/edit?usp=sharing

### fold 1
```
train fold data: 42286
0.0    37043
1.0     2934
4.0      935
2.0      687
5.0      416
3.0      270

test fold data: 344
0    123
2     88
1     67
4     29
5     20
3     17
```

```
predict Accuracy: 0.6104651162790697
數美 Accuracy:  0.7877906976744186
predict   0   1   2  3   4   5
correct                       
0        98  12   4  0   6   2
1        43  23   0  0   0   1
2        14   1  48  3  13   8
3         9   2   0  4   1   1
4        12   0   5  0  12   0
5         1   2   2  1   1  12
```

Predict Error: 
https://docs.google.com/spreadsheets/d/1d6QZUC2mAQVHCaiarc4zadE634SUAkOOSlyp_sfohYE/edit?usp=sharing

### fold 2
```
train fold data: 42286
0.0    37026
1.0     2949
4.0      932
2.0      689
5.0      409
3.0      280

test fold data: 344
0    140
2     86
1     52
4     32
5     27
3      7
```

```
predict Accuracy: 0.6308139534883721
數美 Accuracy:  0.7965116279069767

predict    0   1   2  3   4   5
correct                        
0        102  22   2  7   5   0
1         23  25   1  3   0   0
2         20   3  48  5   7   1
3          1   3   0  3   0   0
4         10   2   5  2  13   0
5          5   1   1  1   2  17
```

Predict Error: 
https://docs.google.com/spreadsheets/d/12X3sNAakyvpvuZlrzTaX7reFNdFTH_aLBBA5BNoNfRE/edit?usp=sharing

### fold 3
```
train fold data: 42286
0.0    37019
1.0     2930
4.0      940
2.0      703
5.0      421
3.0      272

test fold data: 344
0    147
2     72
1     71
4     24
5     15
3     15
```

```
predict Accuracy: 0.6337209302325582
數美 Accuracy:  0.7674418604651163
predict    0   1   2  3  4  5
correct                      
0        105  26   6  2  5  1
1         28  39   0  3  0  0
2         22   0  36  5  9  0
3          9   2   0  4  0  0
4         10   0   4  0  9  1
5          3   2   1  0  1  8
```

Predict Error: 
https://docs.google.com/spreadsheets/d/1V8Mc3pxJXhiTxe-G2A9NHeup0NqJg4JTIBehm4QLuCM/edit?usp=sharing

### fold 4
```
train fold data: 42286
0.0    37047
1.0     2924
4.0      928
2.0      703
5.0      414
3.0      269

0    119
1     77
2     72
4     36
5     22
3     18
```

```
predict Accuracy: 0.6075581395348837
數美 Accuracy:  0.7936046511627907
predict   0   1   2  3  4   5
correct                      
0        90  13   8  1  4   1
1        40  29   3  3  1   0
2        14   1  47  1  8   1
3         7   3   1  5  2   0
4        21   1   5  0  9   0
5         1   0   0  0  0  20
```

Predict Error: 
https://docs.google.com/spreadsheets/d/1-7CEAtDhbR24-j6jEIO9vcNPl0FyXSDyqkGup94wStA/edit?usp=sharing

### 比較

| fold     | Predict     |  數美   |
| ----     | -------     | ----- |
| 0        | 0.637       | 0.733 |
| 1        | 0.610       | 0.787 |
| 2        | 0.630       | 0.795 |
| 3        | 0.633       | 0.767 |
| 4        | 0.607       | 0.793 |
| Range    | 0.60~ 0.63  | 0.73~0.79 |

# 所有 nickname(沒斷詞) 當 test data(train data 沒看過)

```
Training data: 40910
0.0    36509
1.0     2662
4.0      816
2.0      378
5.0      326
3.0      218

NicknameLabelCheck.csv
Testing data: 1720
0    657
2    397
1    339
4    148
5    110
3     69
```

```
predict correct number:  854 
predict Accuracy:  0.5035377358490566
predict    0    1    2   3   4  5
label                            
0        528   69   11   9  25  4
1        186  136    2   9   0  3
2        186    4  131  21  45  5
3         43    9    0  14   3  0
4         82    3   19   5  38  1
5         20   55    1  19   3  7
```

Predict Error:
https://docs.google.com/spreadsheets/d/1RZJ_vQxLpkXBOC6ClfS_igldZ3n0-XyEHF8eVYb5PQU/edit?usp=sharing

### 結論
* 暱稱仍然需要斷詞，暱稱只有某部分符合不正常(1~5類)，如果不斷詞 model 抓不到
* 加入 80% 相似資料進入訓練效果有限

# Update 2021.03.17
## Cross Validation
0. data: **NicknameTestDataWithLabel.csv** (45090筆)
1. 過濾中文及去除重複 (45090-> 5112)
2. 加上**word_label_data_remove_long_word_210222_add normal name_and_Label.csv** (40910筆)
3. 做 5-fold  `註: 太長的字(維度超過 13)會被 Pass`

### 結果
| fold     | Predict     | 
| ----     | -------     | 
| 0        | 0.916       |
| 1        | 0.887       |
| 2        | 0.782       | 
| 3        | 0.901       |
| 4        | 0.918       | 
| Range    | 0.78 ~ 0.91 |
### fold 0
```
fold 0 
train fold data: 44998
0.0    39685
1.0     2725
5.0     1102
4.0      840
2.0      423
3.0      223

test fold data: 1023
0    781
5    211
1     12
4     11
2      8
```
``` 
Train RuntimeError_count: 159
Test RuntimeError_count: 39

Predict Accuracy: 0.9159

predict    0   1  2  4    5
label                      
0        711  49  1  5   10
1          8   4  0  0    0
2          4   0  4  0    0
4          9   0  1  1    0
5          4   0  0  0  173
```

### fold 1
```
train fold data: 44998
0.0    39685
1.0     2717
5.0     1114
4.0      845
2.0      415
3.0      222

test fold data: 1023
0    781
5    199
1     20
2     16
4      6
3      1
```

```
Train RuntimeError_count: 158
Test RuntimeError_count: 40

Predict Accuracy: 0.8866

predict    0   1  2  3   4    5
label                          
0        721  35  2  1  10    4
1         11   8  0  0   1    0
2         12   0  2  0   1    0
3          0   0  0  1   0    0
4          4   0  1  1   0    0
5         13   2  0  0   1  152
```

### fold 2
```
train fold data: 44999
0.0    39657
1.0     2724
5.0     1131
4.0      847
2.0      419
3.0      221

test fold data: 1022
0    809
5    182
1     13
2     12
4      4
3      2
```

```
Train RuntimeError_count: 156
Test RuntimeError_count: 42

Predict Accuracy: 0.7818

predict    0
label       
0        799
1         13
2         11
3          2
4          4
5        151
```

### fold 3
```
train fold data: 44999
0.0    39669
1.0     2722
5.0     1121
4.0      843
2.0      423
3.0      221

test fold data: 1022
0    797
5    192
1     15
4      8
2      8
3      2
```

```
Train RuntimeError_count: 159
Test RuntimeError_count: 39

Predict Accuracy: 0.9099

predict    0   1  2  4    5
label                      
0        744  30  4  6   10
1          6   8  0  0    1
2          4   0  3  0    1
3          2   0  0  0    0
4          5   0  1  2    0
5          8   0  0  0  148

```

### fold 4
```
train fold data: 44999
0.0    39677
1.0     2722
5.0     1110
4.0      845
2.0      422
3.0      223

test fold data: 1022
0    789
5    203
1     15
2      9
4      6
```
```
Train RuntimeError_count: 162****
Test RuntimeError_count: 36

Predict Accuracy: 0.9178

predict    0   1  2  3  4    5
label                         
0        736  21  3  4  6   12
1          7   8  0  0  0    0
2          6   1  1  0  0    1
4          2   0  2  1  1    0
5          6   0  0  0  0  168
```

### 結果分析
* 這份 Data 絕大部分都是 正常/廣告，對於其他類別較無法分析
* 會將預測錯誤的結果調整 Train data 標記

# Update 2021.03.21
## 過濾非中文暱稱&去除重複 
* Train data: **word_label_data_remove_long_word_210222_add normal name_and_Label.csv (40910筆)** 
* Test data : **NicknameTestDataWithLabel.csv**


| 描述     | 檔案名                                    | 筆數      |
| --------| --------------------------------------   | -------- |
| 原始     | NicknameTestDataWithLabel.csv            |45090     |
| 濾出中文  | nickname_labeled_zh_210112_0055CC.csv    |26444     |
| 去除重複  | 使用 drop_duplicates語法(如下)             |5112     |

![](https://i.imgur.com/UgpyoR8.png)

```python=
test_data = pd.read_csv('nickname_labeled_zh_210112_0055CC.csv')
test_data.drop_duplicates(inplace=True)
```
##### 重複的 nickname 舉例
![](https://i.imgur.com/srhSyBe.png)

### 結果
| 狀態                     | Accuracy |
| --------                | -------- |
| (上次) 有斷詞&帳號偵測    | 0.8057     | 
| (此次) 直接 predict      | 0.742     | 

![](https://i.imgur.com/qqyxMaz.png)

* 廣告偵測不大出來

predict error: https://docs.google.com/spreadsheets/d/1LUCwbK_zQ39hcHL0X0HYk7kH5shAyckiDmbl_I-xktg/edit?usp=sharing

# Update 2021.04.06
## Training data
* File: `new word dataset_add label_normal name_default.csv`
* 組成: 
```
word_label_data_remove_long_word_210106.csv (37217筆)
+ 正常人名_簡_去掉非中文.csv(1450筆)
+ Label.csv (2246筆)
+ nickname_wechat_jieba_labeled.csv(2688筆)
+ nickname_default_jieba_labeled(6173筆)

1.去除 emoji & 無義標點符號 & 微信號(都是數字)
2.drop_duplicates
3.dropna
4.修正過去 predict error 的 label
```
* information:
```
Training data: 44786
0    40956
1     2039
4      830
2      406
5      342
3      213
```
## Testing data
* File: `NicknameLabelCheck.csv`
* information:
```
Testing data: 1720
0    657
2    397
1    339
4    148
5    110
3     69
``` 
* note: 有數美標記(label) 與 我們人工標記(Correct) 的欄位

## Model
* **text_cnn_best_99.47767857142857_LR0.001_BATCH100_EPOCH100**


## Accuracy

| | [數美](#Baseline-數美)| [Search](#搜尋模式-Search)  | [Full](#全模式-Full-Best)            | [Precise](#精確模式-Precise)         |
| -------- | ---- | ------------------------------------ | ------------------------------------ | ------------------------------------ | -- |
| Accuracy |0.82558 | 0.71453| 0.711627 | 0.7058139  |
| Error    | 300    | 491    | 496      | 506        |

### Baseline: 數美
```
數美         0    1    2   3    4   5   All
Correct                                  
0        477  107    3   7   62   1   657
1         18  320    0   1    0   0   339
2          1    3  338   0   55   0   397
3          0   25    0  44    0   0    69
4          2    0    0   0  146   0   148
5          1   11    0   0    3  95   110
All      499  466  341  52  266  96  1720

Accuracy: 0.8255813953488372
Error amount: 300 / 1720

Type 0 Accuracy: 477 / 657 = 0.726027
Type 1 Accuracy: 320 / 339 = 0.943953
Type 2 Accuracy: 338 / 397 = 0.851385
Type 3 Accuracy: 44 / 69 = 0.637681
Type 4 Accuracy: 146 / 148 = 0.986486
Type 5 Accuracy: 95 / 110 = 0.863636
```

### 搜尋模式-Search
* Result Spreadsheet:
https://docs.google.com/spreadsheets/d/16kwsiDSAawzeR6cyqUD72IVelgC1KbykniqJDThpzjU/edit?usp=sharing
```
Predict    0    1    2   3    4    5   All
Correct                                   
0        589   38   10   3   13    4   657
1        188  146    0   3    1    1   339
2         90    1  274   0   32    0   397
3         16   10    0  43    0    0    69
4         49    1   15   0   82    1   148
5          9    2    0   2    2   95   110
All      941  198  299  51  130  101  1720

Accuracy: 0.7145348837209302
Error amount: 491 / 1720

Type 0 Accuracy: 589 / 657 = 0.896499
Type 1 Accuracy: 146 / 339 = 0.430678
Type 2 Accuracy: 274 / 397 = 0.690176
Type 3 Accuracy: 43 / 69 = 0.623188
Type 4 Accuracy: 82 / 148 = 0.554054
Type 5 Accuracy: 95 / 110 = 0.863636
```
### 全模式-Full-Best
* Result Spreadsheet:
https://docs.google.com/spreadsheets/d/1SW-nucHjmcdANstnvaVBaPpA4HsediQu8f2Ko059ffI/edit?usp=sharing
```
Predict    0    1    2   3    4    5   All
Correct                                   
0        596   29   10   4   14    4   657
1        211  122    0   4    1    1   339
2         68    1  290   0   37    1   397
3         17    9    0  43    0    0    69
4         51    1   17   0   78    1   148
5          9    3    0   1    2   95   110
All      952  165  317  52  132  102  1720

Accuracy: 0.7116279069767442
Error amount: 496 / 1720

Type 0 Accuracy: 596 / 657 = 0.907154
Type 1 Accuracy: 122 / 339 = 0.359882
Type 2 Accuracy: 290 / 397 = 0.730479
Type 3 Accuracy: 43 / 69 = 0.623188
Type 4 Accuracy: 78 / 148 = 0.527027
Type 5 Accuracy: 95 / 110 = 0.863636

```
### 精確模式-Precise
* Result Spreadsheet:
https://docs.google.com/spreadsheets/d/1Tg3Qo9Ztkp9GCWVIHOav8butgVzQWIK7fcBefs20lZU/edit?usp=sharing
```
Predict    0    1    2   3    4   5   All
Correct                                  
0        590   38   10   3   13   3   657
1        189  145    0   3    1   1   339
2         93    2  271   0   31   0   397
3         17    7    0  45    0   0    69
4         49    1   14   0   83   1   148
5         24    2    0   2    2  80   110
All      962  195  295  53  130  85  1720

Accuracy: 0.7058139534883721
Error amount: 506 / 1720

Type 0 Accuracy: 590 / 657 = 0.898021
Type 1 Accuracy: 145 / 339 = 0.427729
Type 2 Accuracy: 271 / 397 = 0.682620
Type 3 Accuracy: 45 / 69 = 0.652174
Type 4 Accuracy: 83 / 148 = 0.560811
Type 5 Accuracy: 80 / 110 = 0.727273
```

## 預測錯誤原因
### model 漏判 涉政(1) 原因
1.model 未識別 相關政治人物（數美判斷正確）`ex: 林建荣、杨添福`
2.暱稱模型判斷不出來 `ex:習平、金三胖`

### model 漏判 辱罵(2) 原因
1.切開後個別的詞，model 看不出來是辱罵
2.model 斷詞後被偵測到 涉政字眼 `ex:台湾`
3.model 誤判為色情 `ex:逼逼、骚货`
### model 漏判 違禁(3) 原因
1.宗教後來才定義為違禁，可能還沒改完整 `ex:释迦、佛陀`
2.model 沒有學習到 違禁(3) 字眼 `ex: 沙漠之鷹、掌心雷`
### model 漏判/誤判 色情(4) 原因
1.切開後個別的詞，model 看不出來是色情 `ex:加v看我喷水`
2.辱罵常常提到性器官與性行為，與色情重疊

### model 漏判 廣告(5) 原因
1.切詞沒切準確 `ex:[('上', 0), ('下分加', 0), ('薇', 0), ('f', 0), ('89', 0), ('w98', 0)]`
2.整句才看的出來是廣告，個別的詞皆正常 `ex:新美达健身游泳报名负责人文仔`

# Update 2021.04.12
## Training data
* File: `new word dataset_add label_normal name_Censor.csv`
* 組成: 
```
new word dataset_add label_normal name_Censor.csv
(44786筆)
+ CensorWordsCorrected(35838筆)

1.drop_duplicates
2.dropna
```
### Information
```
CensorWordsCorrected: 35838
0    16192
5    14341
3     2220
1     1925
4     1041
2      119

new word dataset_add label_normal name_Censor: 66290
0    57148
1     3964
3     2433
4     1871
2      525
5      349

```
## Testing data
* File: `NicknameLabelCheck.csv`
* information:
```
Testing data: 1720
0    657
2    397
1    339
4    148
5    110
3     69
``` 
## Model
* **text_cnn_best_99.47767857142857_LR0.001_BATCH100_EPOCH100**

## Accuracy
| | [數美](#Baseline-數美)| [Search](#搜尋模式-Search)  | [Full](#全模式-Full-Best)            | [Precise](#精確模式-Precise)         |
| -------- | ---- | ------------------------------------ | ------------------------------------ | ------------------------------------ | -- |
| (0406)Accuracy |0.82558 | 0.71453| 0.711627 | 0.7058139  |
| Error    | 300    | 491    | 496      | 506        |
| (add Censor)Accuracy |0.82558 | 0.68837| 0.68197 | 0.68837  |
| Error    | 300    | 536    | 547      | 536        |
| (add Censor without url)Accuracy |0.82558 | 0.68953| 0.68488 | 0.675  |
| Error    | 300    | 534    | 542      | 559        |



### 搜尋模式-Search
```
Predict    0    1    2   3    4   5   All
Correct                                  
0        582   43    8   2   20   2   657
1        186  148    0   3    1   1   339
2         86    2  238   1   70   0   397
3         16    9    0  44    0   0    69
4         54    2   13   1   77   1   148
5          7    6    0   0    2  95   110
All      931  210  259  51  170  99  1720

Accuracy: 0.6883720930232559
Error amount: 536 / 1720
```
### 全模式-full
```
Predict    0    1    2   3    4   5   All
Correct                                  
0        568   55   10   2   20   2   657
1        191  143    0   4    0   1   339
2         63    3  249   1   81   0   397
3         17    8    0  44    0   0    69
4         52    3   17   1   74   1   148
5          6    7    0   0    2  95   110
All      897  219  276  52  177  99  1720

Accuracy: 0.6819767441860465
Error amount: 547 / 1720
```
### 精確模式-precise 
```
Predict    0    1    2   3    4   5   All
Correct                                  
0        584   42    8   2   19   2   657
1        188  146    0   3    1   1   339
2         86    3  235   1   72   0   397
3         17    6    0  46    0   0    69
4         54    2   12   1   78   1   148
5          7    6    0   0    2  95   110
All      936  205  255  53  172  99  1720

Accuracy: 0.6883720930232559
Error amount: 536 / 1720
```

# Update 2021.04.18
## Training data

| 時間軸   | 檔名                               | 筆數     |備註                                     |
| --------| --------                          | --------|----                                     |
| 4/7     | CensorWords_Drop Duplates_labeled | 35840   |數美標的                                  |
| 4/11    | CensorWordsCorrected              | 35840   |學姊修改過                                |
| 4/14    | CensorWords_predict               | 14497   |數美標0的label 給 model predict           |
| 4/15    | CensorWords_predict_checked       | 14497   |數美標0的label 給 model predict 且學姊修改過|
| 4/7     | CensorWords_Ground Truth          | 19534   |詳見下方說明                                |
### CensorWords_Ground Truth
* CensorWordsCorrected 刪除網址
* CensorWordsCorrected(1~5類) +  CensorWords_predict_checked
```
CensorWords_Ground Truth: 19534
0    12895
1     2654
3     2388
4     1256
5      183
2      158
```

## Accurancy
### 數美 vs Ground Truth
```
數美 Accuracy :  0.8237432169550527
數美
correct      0     1   2     3    4    5
label                                   
0        12894  1116  59  1125  801  179
1            1  1536   1    96    0    0
2            0     0  91     0   27    0
3            0     2   0  1146    6    0
4            0     0   7    20  421    1
5            0     0   0     1    1    3

```
### Model vs Ground Truth
```
Predict Accuracy :  0.829118460120815
Model
correct      0     1    2     3    4   5
predict                                 
0        12877   692   34  1337  432  85
1            8  1923    2   166   14   0
2            2     2  108    12   55   0
3            1    31    0   446    5   0
4            6     5   12   123  746   2
5            1     1    2   304    4  96
```

## Nickname Test
### Model
* text_cnn_best_99.04503105590062_LR0.001_BATCH100_EPOCH100
### Accuracy
| 類別 | 筆數      | Accuracy    |備註    |
| --- | -------- | --------    |------- |
| 涉政 | 1376     | 0.91782     |        |
| 辱罵 | 302      | 0.48013     |        |
| 辱罵 | 302      | 0.5298      |切詞預測  |
| 違禁 | 1182     | 0.97453     |        |
| 色情 | 415      | 0.91566     |        |
| 廣告 | 5529     | 0.00217     |        |
| 廣告 | 5529     | 0.066712    |切詞預測  |

```
predict   0     1  2   3  5
label                      
1        60  1586  1  80  1
```

```
predict   0  1    2   3   4  5
label                         
2        97  9  145  11  39  1

[切詞]
Predict    0  1    2   4  5  
label                           
2        108  4  160  28  2
```

```
predict   0  1     3  4  5
label                     
3        15  6  1148  6  3
```

```
predict   0   3    4  5
label                  
4        14  19  380  2
```
```
predict     0    1    2    3    4   5
label                                
5        4903  201  156  107  150  12

[切詞]
Predict     0  4    5   
label                      
5        5398  2  38
```

## 辱罵 & 廣告 Analysis
### 涉政
https://docs.google.com/spreadsheets/d/14QEuSDEXL258ZHzNhah4p8sTRmZgGjCC_PkNqLZGzow/edit?usp=sharing
### 辱罵
* 判斷成色情
* 辱罵與涉政皆有(涉政順位較高)
https://docs.google.com/spreadsheets/d/1_AzDL6ZWVoEijxby7IdgoSut5eqRI9Robi5t3Q32K98/edit?usp=sharing

### 違禁
https://docs.google.com/spreadsheets/d/1J1KnJgg9ZVeGXOqntyb-DKJAV6yIelnPjjn_bjw2yXk/edit?usp=sharing

### 色情
https://docs.google.com/spreadsheets/d/19kPWaJXN5xqxlZgZ1pXcTNck_c451wS6Qjihk23-Hl8/edit?usp=sharing
### 廣告
* URL 偵測不出(Train_data 加 URL 也無法)
https://docs.google.com/spreadsheets/d/10ZFL1HyrPF6WiVK1vGD7oRvrBK1wneMP4SDcoZV1Fws/edit?usp=sharing

# Update 2021.04.26
## 作法
**CensorWords_Ground Truth.csv** 拿去斷詞 predict
## 結果
**CensorWords_Ground Truth_precise_210419_1316.csv**

```
斷詞前
Predict Accuracy :  0.829118460120815
Model
correct      0     1    2     3    4   5
predict                                 
0        12877   692   34  1337  432  85
1            8  1923    2   166   14   0
2            2     2  108    12   55   0
3            1    31    0   446    5   0
4            6     5   12   123  746   2
5            1     1    2   304    4  96
```

```
CensorWords_Ground Truth: 19534
0    12895
1     2654
3     2388
4     1256
5      183
2      158

Label Accuracy :  0.8237432169550527
數美
correct      0     1   2     3    4    5
label                                   
0        12894  1116  59  1125  801  179
1            1  1536   1    96    0    0
2            0     0  91     0   27    0
3            0     2   0  1146    6    0
4            0     0   7    20  421    1
5            0     0   0     1    1    3

Predict Accuracy :  0.8374116924337053
Model
correct      0     1   2     3    4    5
predict                                 
0        12753   911  65   987  450  111
1           92  1703   2   130   14    1
2            3     1  79     0   41    0
3           29    35   1  1020   14    0
4           14     3   9    29  734    2
5            4     1   2   222    3   69
```

https://docs.google.com/spreadsheets/d/1E_jU-_UOfK1xrCzh0k7mrVtD28-bhn544JED-BsMdWU/edit?usp=sharing

## URL
### 作法
![](https://i.imgur.com/UmbbfxH.png)

https://www.notion.so/0ea0785783454d2fb05b7afc2f0d7401?v=4ac201766ff1426eb514b67750643558&p=b6ffd73c1c1a4404bfe56b539033c99a

### 結果
* Test data: 廣告
* URL 數: 約 5345個
#### regular expression + urlextract
`dataset containing URL: 5342 / 5786`

#### only regular expression
`dataset containing URL: 5349 / 5786`

#### only urlextract
`dataset containing URL: 5342 / 5786`
https://docs.google.com/spreadsheets/d/1au6tyiSB7nAK2QMJUOSyPBKeguWtzQOZY40qXlJihIA/edit?usp=sharing

# Update 2021.04.28
## model 
**text_cnn_best_99.04885057471265_LR0.001_BATCH100_EPOCH100**
### training data
* File: `new word dataset_add label_normal name_default_Ground truth_Corrected_2.csv`
* 組成: 
```
new word dataset_add label_normal name_default.csv
+ Ground truth
+ 常見違禁字(ex: 代开、代开发票、咨询电话、刺刀、代办、刺刀、身分证、户口本、刻章)
```
* information:
```
Training data: 69536
0    53851
1     6566
3     4795
4     3102
2      677
5      545
```
## Result
```
Predict Accuracy :  0.8963857888809256

Model
correct      0     1    2     3    4    5
predict                                  
0        12684   360   27   601  235   73
1          112  2258    3   152   16    1
2            0     0  110     0   13    0
3           44    31    0  1380   16    0
4           44     5   16    53  974    5
5           11     0    2   202    2  104
```
https://docs.google.com/spreadsheets/d/1lk9wwRFJniaAQwPWpwdsAKz5yTd6SFxms9wYCROCwWc/edit?usp=sharing

# Update 2021.04.30
## Model
`text_cnn_best_98.63338788870703_LR0.001_BATCH100_EPOCH100`
### Training data
* file: `new word dataset_add label_normal name_default_Ground truth_Corrected__prohibit`
* 組成
```
new word dataset_add label_normal name_default_Ground truth_Corrected_2.csv
+ 違禁_jieba_corrected

* drop_duplicated()
```

* Information
```
Training data: 61055 (drop duplicate 前 :71013 筆)
0.0    51339
1.0     3967
3.0     2940
4.0     1780
5.0      517
2.0      509

違禁_jieba_corrected: 1461
0    773
3    577
1     85
5     14
4     11
2      1
Corrected similarity :  0.6386036960985626
```

## Result
### CensorWords_Ground Truth.csv

```

Predict Accuracy :  0.8995085491962732

Model
correct      0     1    2     3    4   5
predict                                 
0        12679   483   31   472  322  72
1           57  2125    2    72    9   0
2            1     0  106     0   19   0
3          132    40    3  1701   38  14
4           14     5   14    36  865   2
5           12     1    2   107    3  95

```
https://docs.google.com/spreadsheets/d/15ERx7iORf7hVp2oSz8cU_Ap0EVlAsNqe8JT9eMuWXcs/edit?usp=sharing

### 廠商提供的涉政、辱罵、違禁、色情、廣告
#### 流程

#### 結果

| 類別 | 筆數  |   上次 |A.斷詞    |B.無斷詞|
| --- | -----| -------|-------|-----|
| 涉政 | 1376 | **0.917** |0.729  |0.894|
| 辱罵 | 302  | **0.480** |0.470  |0.417|
| 違禁 | 1182 | 0.974 |0.799  |**0.974**|
| 色情 | 415  | 0.915 |0.534  |**0.920**|
| 廣告 | 5529 | 0.002 |0.991  |**0.991**|

```
[斷詞]
涉政_210502_1610.csv
Predict    0     1   3  4  5   All
label                             
1        370  1265  90  3  8  1736
All      370  1265  90  3  8  1736

Accuracy: 0.7286866359447005
Error amount: 471 / 1736


辱罵_210502_1611.csv
Predict    0  1    2   4  5  All
label                           
2        113  3  142  43  1  302
All      113  3  142  43  1  302

Accuracy: 0.47019867549668876
Error amount: 160 / 302


違禁_210502_1611.csv
Predict    0   1    3  4   5   All
label                             
3        144  28  945  5  60  1182
All      144  28  945  5  60  1182

Accuracy: 0.799492385786802
Error amount: 237 / 1182


色情_210502_1612.csv
Predict    0  1  2   3    4  5  All
label                              
4        157  6  4  20  222  6  415
All      157  6  4  20  222  6  415

Accuracy: 0.5349397590361445
Error amount: 193 / 415


廣告_210502_1612.csv
Predict   0  1  4     5   All
label                        
5        47  1  2  5736  5786
All      47  1  2  5736  5786

Accuracy: 0.9913584514344971
Error amount: 50 / 5786

```
```
[無斷詞]
涉政_210502_1801.csv
Predict   0     1    3  4  5   All
label                             
1        71  1552  105  2  6  1736
All      71  1552  105  2  6  1736

Accuracy: 0.8940092165898618
Error amount: 184 / 1736


辱罵_210502_1801.csv
Predict    0  1    2  3   4  All
label                           
2        113  6  126  8  49  302
All      113  6  126  8  49  302

Accuracy: 0.41721854304635764
Error amount: 176 / 302


違禁_210502_1802.csv
Predict   0  1     3  4  5   All
label                           
3        18  3  1151  8  2  1182
All      18  3  1151  8  2  1182

Accuracy: 0.9737732656514383
Error amount: 31 / 1182


色情_210502_1802.csv
Predict  0   3    4  5  All
label                      
4        7  21  382  5  415
All      7  21  382  5  415

Accuracy: 0.9204819277108434
Error amount: 33 / 415


廣告_210502_1802.csv
Predict   0  1   3  4     5   All
label                            
5        37  1  12  4  5732  5786
All      37  1  12  4  5732  5786

Accuracy: 0.9906671275492568
Error amount: 54 / 5786
```
https://docs.google.com/spreadsheets/d/1Mz0YOHm6lrIYmUfH6-M1eSOOMq9G3lP-R0y0r9Nqukg/edit?usp=sharing

# Update 2021.05.10
## URL 時間比較 (套件 vs regular expression)
* 套件: `urlextract`
* regular expression: `[a-zA-Z0-9]+\.[a-zA-Z]+`
### result
* Test data: 廣告(URL: 約5345/5789)
* 環境: colab(cpu)
* 時間單位: 秒


| 方法               | 時間  | URL數 |  先掃 rule-base 再斷詞 predict (total)   |
| ------------------ | ----- | -----| --- |
| regular expression | 6.004 | 5349 | 59.410 |
| urlextract         | 7.903 | 5342 | 71.159    |
| Both               | 7.737 | 5342 | 71.32    |

##### 註: 確切時間屆時再拿打包給廠商的那版程式測一次

## 掃 Rulebase Time and Accuracy
* 環境: colab(cpu)
* 觀察: 先掃 rule-base 的詞語是否出現在 nickname 中，不在rule-base的話再送去斷詞 & predict 花費時間較少 (因為不用每個詞都斷詞及 embedding)，正確率也較高

### 流程
* 5/10 以前的方法
```
1. 找出 帳號/電話/URL => 直接標記 5
2. Jieba 斷詞 / 完全不斷詞
3. 比對 rule base，完全相同的給對應 label

nickname: "你這個蠢蛋"
rule base = ['白癡', '蠢蛋' ]
[A.斷詞:]
case(1) " 你 這個 蠢蛋" 因為 "蠢蛋" 完全相同 rule base 的 "蠢蛋" => 標記 2
case(2) "你 這個 蠢 蛋" 無與 "蠢蛋" 完全相同的字 => model predict

[B.無斷詞]
將 "你這個蠢蛋" 與 rule base 比對，無相同 => model predict

4. model predict
```
* 5/10 新採用的方法
```
1. 找出 帳號/電話/URL => 直接標記 5
2. 比對 rule base 的詞有無出現在帳號中

rule base = ['白癡', '蠢蛋' ]
nickname_1 = "你這個蠢蛋"
nickname_1 = "你這個帥哥"

以"白癡"與"蠢蛋" 去比對是否存在於 nickname 中
=> 解決因斷詞被斷開而無法透過 rule base 找出的情況

3. model predict
[C.不斷詞 predict]
「你這個蠢蛋: 2」

[D.斷詞後 predict ]
「你 這個 帥哥」 =>「你:0 這個:0 帥哥:0」 => 「final label:0」




```
### Time
| 類別 | 筆數 | A.斷詞後檢查 rule-base | B.不斷詞檢查 rule-base    | C.先掃 rule-base 不斷詞 predict | D.先掃 rule-base 再斷詞 predict |
| ---- | ---- | ---------------------- | --- | ------------------------------- | ------------------------------- |
| 涉政 | 1376 | 360.492                | 188.859  | 263.931                         | 483.819                         |
| 辱罵 | 302  | 88.320                 | 31.945   | 24.005                          | 62.088                          |
| 違禁 | 1182 | 391.988                | 133.854  | 185.525                         | 567.888                         |
| 色情 | 415  | 112.990                | 45.805  | 65.774                          | 161.557                         |
| 廣告 | 5529 | 23.158                 | 16.497  | 23.525                          | 74.786                          |

### Accuracy


| 類別 | 筆數 | A.斷詞後檢查 rule-base | B.不斷詞檢查 rule-base    | C.先掃 rule-base | D.先掃 rule-base 再斷詞 predict |
| ---- | ---- | ------| --- | -------- | ------ |
| 涉政 | 1376 | 0.729  | 0.894    | **0.895**   | 0.680  |
| 辱罵 | 302  | 0.470  | 0.417    | 0.708    | **0.745**  |
| 違禁 | 1182 | 0.799  | **0.974**    | 0.934    | 0.759  |
| 色情 | 415  | 0.534  | **0.920**    | 0.889    | 0.460  |
| 廣告 | 5529 | 0.991  | 0.991    | 0.991    | 0.991  |


``` 
[說明]
INFO:root:dataset containing rule_politics: 含涉政 rule - base

INFO:root:dataset containing rule_abuse: 含辱罵 rule - base

INFO:root:dataset containing rule_porn: 含色情 rule - base

INFO:root:dataset containing rule: 含廣告 rule - base
```
```
[涉政]
INFO:root:dataset containing rule_politics: 23 / 1736

INFO:root:dataset containing rule_abuse: 20 / 1736

INFO:root:dataset containing rule_porn: 0 / 1736

INFO:root:dataset containing rule: 7 / 1736

[辱罵]
INFO:root:dataset containing rule_politics: 0 / 1182

INFO:root:dataset containing rule_abuse: 27 / 1182

INFO:root:dataset containing rule_porn: 0 / 1182

INFO:root:dataset containing rule: 20 / 1182

[違禁]
INFO:root:dataset need decode: 0 / 415

INFO:root:dataset containing account: 0 / 415

INFO:root:dataset containing phone number: 0 / 415

INFO:root:dataset containing URL: 4 / 415

[色情]
INFO:root:dataset containing rule_politics: 5 / 415

INFO:root:dataset containing rule_abuse: 10 / 415

INFO:root:dataset containing rule_porn: 0 / 415

INFO:root:dataset containing rule: 4 / 415

[廣告]
INFO:root:dataset need decode: 0 / 5786

INFO:root:dataset containing account: 26 / 5786

INFO:root:dataset containing phone number: 362 / 5786

INFO:root:dataset containing URL: 5349 / 5786

```
* 無斷詞 辱罵 Predict result: https://docs.google.com/spreadsheets/d/1QNbVK3weAca4HHQm69f-XLih8fzyC7wQGXnlQY2Lc-4/edit?usp=sharing


## 斷詞 Threshold
### 作法
Nickname 長度超過 Threshold 則斷詞

```
[檔案中最長 nickname]
涉政: 25
辱罵: 13
違禁: 17
色情: 13
廣告: 20
```

| 類別/Threshold | 0    | 4     | 5     | 6         | 7     | 8     | 9     | 10        |不斷詞|
| -------------- | ---- | ----- | ----- | --------- | ----- | ----- | ----- | ---------| --- |
| 涉政           | 0.68 | 0.77  | 0.84  | 0.87      | 0.892 | 0.895 | 0.896 | **0.896** |0.895|
| 辱罵           | 0.74 | 0.768 | 0.768 | **0.768** | 0.764 | 0.74  | 0.725 | 0.721     |0.708|
| 違禁           | 0.75 | 0.79  | 0.82  | 0.84      | 0.86  | 0.89  | 0.920 | **0.927** |0.934|
| 色情           | 0.46 | 0.52  | 0.65  | 0.72      | 0.77  | 0.81  | 0.86  | **0.88**  |0.889|
| 廣告           | 0.99 | 0.99  | 0.99  | 0.99      | 0.99  | 0.99  | 0.99  | **0.99**  |0.991|

* `廣告` 因為幾乎都是 URL ，所以都是靠 regular expression 偵測出來的，故正確率相同