# A Survey on Security Threats and Defensive Techniques of Machine Learning: A Data Driven View
###### tags: `paper` `Machine Learning` `Security` `Survey` `adversarial`

:::info
QIANG LIU,
PAN LI,
WENTAO ZHAO,
WEI CAI,
SHUI YU 3,
VICTOR C. M. LEUNG
:::

:::danger
網路資源:
[Poisoning Attack: Label Contamination]("https://medium.com/@falconives/day-73-poisoning-attack-label-contamination-7bb88c0c0741")
[機器學習潛在的隱私問題]("https://medium.com/trustableai/%E6%A9%9F%E5%99%A8%E5%AD%B8%E7%BF%92%E6%BD%9B%E5%9C%A8%E7%9A%84%E9%9A%B1%E7%A7%81%E5%95%8F%E9%A1%8C-9410eb951411")
[微軟對ML安全威脅的說明文件]("https://docs.microsoft.com/zh-tw/security/engineering/threat-modeling-aiml")
[对抗样本相关论文及最新进展]("https://ai.baidu.com/forum/topic/show/956726")
:::
:::success
可記憶點:
Dalvi et al. [12] email spam detect
:::

:::warning
整份備忘:
1. 定義什麼是基於機器學習的安全威脅
2. 依據觀點將各種攻擊分類(II.c)
3. 根據攻擊實施的階段分類(III)
4. 介紹現有防禦技術(IV) (train、test parse、Data security and privacy)
5. 結論
:::

## ABSTRACT
- researchers from academia and industry have found out many security threats against a variety of learning algorithms, including naive Bayes, logistic regression, decision tree, support vector machine (SVM), principle component analysis, clustering, and prevailing deep neural networks.
- 研究發現，目前已有多種針對學習算法的對抗(攻擊)，包括樸素貝葉斯，邏輯回歸，決策樹，支持向量機器（SVM），主成分分析，聚類、深度神經網絡。

- we categorize current defensive techniques of machine learning into four groups: security assessment mechanisms, countermeasures in the training phase, those in the testing or inferring phase, data security, and privacy.
- 我們將當前防禦技術分為四類：安全評估機制，培訓階段的對策，對於
1. training phase (訓練階段)
2. testing or inferring phase(推斷 預測階段)
3. data security (資料安全)
4. privacy(隱私性)

## I. INTRODUCTION
- In 2004, Dalvi et al. [12] introduced the concept of adversarial classification and analyzed the detection evasion problem of early spam detection

- Recently, some surveys on the security perspective of artificial intelligence and machine learning have been presented [15], [16]. Amodei et al. [15] gave a general introduction of security issues about artificial intelligence, especially the supervised and reinforcement learning algorithms.

- we emphasize the data distribution drifting caused by adversarial samples and sensitive information violation problems in statistical machine learning algorithms.

## II. BASIC CONCEPT, MODEL AND TAXONOMY
### A. BASICS OF MACHINE LEARNING
### B. ADVERSARIAL MODEL

- a well-defined adversarial model should be constructed with four dimensions, goal, knowledge, capability and attacking strategy [19]. Specifically, the adversarial goal can be clearly described using both the expected impacts and the attack specificity of security threats.

- Regarding the adversarial knowledge, it can be divided into two groups named constrained knowledge and complete knowledge by examining whether or not an attacker know training data, features, learning algorithms, decision functions, classifier parameters and feedback information.

- the **capability** can be qualitative interpreted from three aspects:
1. Is the impact of security threats causative or exploratory?
2. What is the percentage of training and testing data that are controlled by the attacker?
3. What is the extent of features and parameters that are known by the attacker?

### C. SECURITY THREAT TAXONOMY
- The taxonomy of security threats towards machine learning was proposed in [20] in three different perspectives, the influence on classifiers, the security violation and the attack specificity, as illustrated in Fig. 1


## III. SECURITY THREATS TOWARDS MACHINE LEARNING

### A. SECURITY THREATS AGAINST THE TRAINING PHASE
如果能在訓練階段攻擊訓練資料效果會最好,eg: poisoning attack
#### 1. POISONING WITHOUT MODIFYING FEATURES OR LABELS
將惡意data注入是可行的(像是對需要Retrain的model)，eg:
- adaptive facial recognition systems (FRSs) [9], [38][40]
- malware classification [41]
- spam detection [6]

#### 2. POISONING WITH MODIFYING FEATURES OR LABELS
- label contamination attack (LCA)
---
Label contamination attack 是指敵對方在機器學習模型訓練過程中，有意地修改了標籤(label)的資料，以達到影響模型預測結果的目的。這種攻擊方式通常發生在模型訓練過程中，而非在模型發佈後。

例如，假設我們正在訓練一個負責區分狗和貓的圖像分類模型。敵對方可能會有意地將一些貓的圖像標記為狗，或者將一些狗的圖像標記為貓，這樣一來，訓練出來的模型可能就會對某些圖像分類出錯，即使這些圖像在原本的標籤下是容易分類的。

這種攻擊可以用不同的方式實現，包括在標記資料時有意做錯、使用自動化工具操縱標記等。這種攻擊在實際應用中可能導致模型的精度大幅下降，甚至無法使用。

為了防止 label contamination attack，可以通過以下方式來提高模型的安全性：

檢查標記資料：在訓練模型之前，需要對標記資料進行嚴格的審查，確保每個標記都是正確的。

多樣化訓練資料：如果敵對方想要操縱標記資料，他們需要先了解我們的訓練資料集，因此我們可以通過增加訓練資料集的多樣性來減少攻擊的可能性。

標記資料的可信度評估：當我們接收到新的標記資料時，可以對這些資料進行可信度評估，以確保它們沒有被敵對方修改。

使用魯棒性強的模型：我們可以使用一些魯棒性強的模型，例如集成學習、深度學習中的對抗訓練等，來提高模型對於 label contamination attack 的抵抗能力。
---

### B. SECURITY THREATS AGAINST THE TESTING/INFERRING PHASE
###

攻擊對象:
- image recognition [59]
- malware detection [60], [61] (趨勢科技)
- intrusion detection [62] based on machine learning.
對DNN特別有效、因為DNN用來辨識物件的的feature set 很小

### C. SUMMARY OF ADVERSARIAL ATTACKS AGAINST MACHINE LEARNING

整理各種攻擊手法、其優缺點、和攻擊樣本的loss

[攻擊定義]("https://mlconference.ai/blog/tricking-an-autonomous-vehicle-into-not-recognizing-a-stop-sign-is-an-evasion-attack/")

### IV. DEFENSIVE TECHNIQUES OF MACHINE LEARNING

#### A. SECURITY ASSESSMENT MECHANISMS
兩種防禦技術

#### B. COUNTERMEASURES IN THE TRAINING PHASE
如前所述，中毒攻擊應該是通過將指定的對抗樣本注入訓練數據以影響下的最終決策功能特定的機器學習算法。因此，確保訓練數據的純度[76]和提高(robustness)學習算法的健壯性[77] – [79]是兩個主要對策

另外還有設計新的安全演算法
Demontis et al. 設計出sec-SVM (安全的 支援向量機)

#### C. COUNTERMEASURES IN THE TESTING/INFERRING PHASE
- 將對抗樣本加入訓練
- 特徵降維(防止evasion attacks，作用於SVMs and DNN)

#### D. DATA SECURITY AND PRIVACY
- DP to preserve the privacy of different learning algorithms,
including SVM [102], deep learning [103] and Bayesian
optimization [104].
- 補充資料
- [在差分隐私中，为什么需要消耗隐私预算？](https://www.zhihu.com/question/407819411/answer/1348363105)
- 同態加密

#### E. SUMMARY OF DEFENSIVE TECHNIQUES OF MACHINE LEARNING

### V. CHALLENGES AND FUTURE OPPORTUNITIES
Nowadays, machine learning is the core technology of big
data, Internet of Things (IoT), cloud computing and artificial intelligence. Accordingly, various security threats and

(1) New security threats towards machine learning are
constantly emerging.

(2) Security assessment on machine learning based
decision systems in adversarial environments becomes a
prevailing research area

(3) Data privacy plays an important role in protecting
the security of machine learning.

(4) Secure deep learning is a new growth point in the
field of machine learning security.

(5) Jointly optimizing security, generalization perfor-
mance and overhead is required to design secure learning
algorithms.