# Evaluation Metrics for imbalanced Classification

###### tags: `Evaluation Metrics`

> This note is about the comparison of different metrics to evaluate the quality of classification.

We can divide evaluation metrics into three useful groups-
1. Threshold Metrics
2. Ranking Metrics
3. Probability Metrics

### [1] Threshold Metrics
- **Accuracy** = Correct Predictions / Total Predictions
- **Error** = Incorrect Predictions / Total Predictions = 1 - **Accuracy**

Although widely used, classification accuracy is almost universally inappropriate for imbalanced classification. The reason is, a high accuracy (or low error) is achievable by a no skill model that only predicts the majority class.

**Confusion Metrix**

               | Positive Prediction | Negative Prediction
               ---------------------------------------------
Positive Class | True Positive (TP)  | False Negative (FN)
Negative Class | False Positive (FP) | True Negative (TN)

#### Sensitivity-Specificity Metrics

- **Sensitivity** = TruePositive / (TruePositive + FalseNegative)
(it means how well the positive class was predicted)
- **Specificity** = TrueNegative / (FalsePositive + TrueNegative)
(it means how well the negative class predicted)
- **G-Mean** = sqrt(Sensitivity * Specificity)
(to balance both above concern)

#### Precision-Recall Metrics

- **Precision** = TruePositive / (TruePositive + FalsePositive)
- **Recall** = TruePositive / (TruePositive + FalseNegative) = **Sensitivity**
- **F-Measure** = (2 * Precision * Recall) / (Precision + Recall)

These are probably the most popular and some other metrics are Kappa, Macro-Average Accuracy, Mean-Class-Weighted Accuracy, Optimized Precision, Adjusted Geometric Mean, Balanced Accuracy, and more.

One **limitation of threshold metrics** is that they assume that the class distribution observed in the training dataset will match the distribution in the test set and in real data when the model is used to make predictions. This is often the case, but when it is not the case, the performance can be quite misleading.

### [2] Ranking Metrics
Rank metrics are more concerned with evaluating classifiers based on how effective they are at separating classes.
- **TruePositiveRate** = TruePositive / (TruePositive + FalseNegative)
- **FalsePositiveRate** = FalsePositive / (FalsePositive + TrueNegative)
- **Receiver Operating Characteristic (ROC) curves**


**ROC AUC** The area under the ROC curve can be calculated and provides a single score to summarize the plot that can be used to compare models. A no skill classifier will have a score of 0.5, whereas a perfect classifier will have a score of 1.0.

- **Precision-Recall (PR) curve**


We can calculate **PR AUC** similarly as above.

**Note:** If positive class is important than negative class we use **ROC AUC** insted of **PR Curve** and vice-versa.

### [3] Probability Metrics
Probabilistic metrics are designed specifically to quantify the uncertainty in a classifier’s predictions.

**LogLoss or cross-entropy:**
For a binary classification dataset
$$
LogLoss = -((1-y)*(log(1-\hat{y}) + y*log(\hat{y}))
$$
For multi classes-
$$
LogLoss = \sum_{c \in C}{y_c*log(\hat{y_c})}
$$

**Brier Score:**
$$
BrierScore = \sum_{i}^{N}{(\hat{y_i}-y_i)^2}
$$

## :memo: How to choose an Evaluation Metric

## :memo: Where do I start?

### Step 1: Change the title and add a tag

- [ ] Create my first HackMD note (this one!)
- [ ] Change its title
- [ ] Add a tag

:rocket:

### Step 2: Write something in Markdown

Let's try it out!
Apply different styling to this paragraph:
**HackMD gets everyone on the same page with Markdown.** ==Real-time collaborate on any documentation in markdown.== Capture fleeting ideas and formalize tribal knowledge.

- [x] **Bold**
- [ ] *Italic*
- [ ] Super^script^
- [ ] Sub~script~
- [ ] ~~Crossed~~
- [x] ==Highlight==

:::info
:bulb: **Hint:** You can also apply styling from the toolbar at the top :arrow_upper_left: of the editing area.

:::

> Drag-n-drop image from your file system to the editor to paste it!

### Step 3: Invite your team to collaborate!

Click on the <i class="fa fa-share-alt"></i> **Sharing** menu :arrow_upper_right: and invite your team to collaborate on this note!


- [ ] Register and sign-in to HackMD (to use advanced features :tada: )
- [ ] Set Permalink for this note
- [ ] Copy and share the link with your team

:::info
:pushpin: Want to learn more? ➜ [HackMD Tutorials](https://hackmd.io/c/tutorials)
:::

---

## BONUS: More cool ways to HackMD!

- Table

| Features | Tutorials |
| ----------------- |:----------------------- |
| GitHub Sync | [:link:][GitHub-Sync] |
| Browser Extension | [:link:][HackMD-it] |
| Book Mode | [:link:][Book-mode] |
| Slide Mode | [:link:][Slide-mode] |
| Share & Publish | [:link:][Share-Publish] |

[GitHub-Sync]: https://hackmd.io/c/tutorials/%2Fs%2Flink-with-github
[HackMD-it]: https://hackmd.io/c/tutorials/%2Fs%2Fhackmd-it
[Book-mode]: https://hackmd.io/c/tutorials/%2Fs%2Fhow-to-create-book
[Slide-mode]: https://hackmd.io/c/tutorials/%2Fs%2Fhow-to-create-slide-deck
[Share-Publish]: https://hackmd.io/c/tutorials/%2Fs%2Fhow-to-publish-note

- LaTeX for formulas

$$
x = {-b \pm \sqrt{b^2-4ac} \over 2a}
$$

- Code block with color and line numbers：

var s = "JavaScript syntax highlighting";
alert(s);

- UML diagrams

Alice->Bob: Hello Bob, how are you?
Note right of Bob: Bob thinks
Bob-->Alice: I am good thanks!
Note left of Alice: Alice responds
Alice->Bob: Where have you been?

- Auto-generated Table of Content
[ToC]

> Leave in-line comments! [color=#3b75c6]

- Embed YouTube Videos

{%youtube PJuNmlE74BQ %}

> Put your cursor right behind an empty bracket {} :arrow_left: and see all your choices.

- And MORE ➜ [HackMD Tutorials](https://hackmd.io/c/tutorials)