---
# System prepended metadata

title: Toxicity Classification

---


# Toxicity Classification

## Preprocessing
We try to convert all sentecne to lower case in LSTM model, but normal case gets highly score in LB than lower case. 

Preprocessing as follows:
 *  all http(url) were substituted to url
 *  all emoji substitute to ' '
 *  using flashtext to find mispell words and replace to true words
 *  all emoji were substituted by ' '
 *  \n\t were substituted by ' '
 *  \s{2,} were substituted by ' '

## Get statistics features
We try to get some statistics feature and put in LSTM model training

We get statistics feature as follows:
* swear word 
* upper word 
* uniqen word
* emoji
* characters 


## Embedding
We used pretrainned word embedding as follows:
* Fasttext
* Glove

In our works, fasttext is a little bit better than Glove.
We didn't concatenate fasttext and Glove due to time consuming. (However, in the nearly end of the competition, everyone used BERT model haha.)


## Model
### LSTM model
Our lstm model are different with public version, it comsisted of lstm cells without gru cells.
* Attention didn't improve LB significantly.
* Spatial Dropout had improvement in LB.
* blending of three models, each lstm model got the LB 0.935x~0.938x. After blended three models, we got the LB 0.93963.


### BERT model

We used pretrained bert model from: [pytorch-pretrained-BERT](https://github.com/huggingface/pytorch-pretrained-BERT) and `BertForSequenceClassification` for sequence classification.
* The result with text preprocessing or without text preprocessing are similar.

* The batch size from 16 to 32 improve the LB. I think the batch size significantly influnces acurracy
* The learning rate we set is `2e-5`
* We got the single model with LB 0.9415x~0.94220
* We ensemble five single BERT model and got LB 0.94294


### GPT2 model
* We only got LB around 0.938 in single GPT2. Therefore we focused on training BERT models. 

### Ensemble model

* We used ensemble of 3 LSTM models and ensemble of 5 BERT models and blended them with weights 0.3 and 0.7 respectively.
* In the end, we got LB 0.9443
