---
# System prepended metadata

title: Model Improvement Scoping Document

---

# Model Improvement Scoping Document

We will focus initially on the LE-weighted and LE-imbalanced configuration as currently those appears the most relevant for deployment. We should aim generalize work as much as possible so that they can be applied to the other configurations.

## **Proposed Tasks (in order of priority)**

- **Update embeddings** in `embeddings_creation` repo.
    - Currently in the process of re-doing the word embeddings
        - We could try fine tuning different pre-trained embeddings i.e. various BERTS - DistilBERT is smaller so will be efficient, as is ELECTRA. Maybe try few-sample method and layerwise decay to reduce instability and overfitting.  
        - Could we get away with a simpler, positional based method like BoW, word2vec and pair up with a simpler model i.e. random forest? Can use feature hashing (although collisions might be a problem if corpus is big) or kmeans to avoid sparse one-hot encoding. Would run fairly quick and I'm sure wont run into much overfitting issues compared to deep learning.
- **Augmentation** Would probably want to create augmentations of the LE data, but would have to be careful not to bring in any leakage. If we get it right we can hopefully boost accuracy in test sets and help regularize for robust future/wider deployment. 
    - Can try multiple methods - there will be plenty of generative transformers will do a decent job. 
- **Multi-task learning** - We alternate batches from different tasks through the network and sum their losses. Can also work if tasks have different loss functions. Perhaps we could try the baseline and tactics as two separate tasks. Could also work if we can find a dataset that is similar i.e. cyber bullying / toxic comments i.e. https://www.kaggle.com/c/jigsaw-toxic-severity-rating 
- **Psuedo labelling** - might not be an attractive option but could we ask for more unlabelled LE data? We could semi-supervise to generate more labels and experiment with the condfidence threshold to get meangingful labels. The police wouldnt have to do any extra work other than source a dataset.
- **Other models** - try additional models such as XLNet and SetFit (a few shot model) https://huggingface.co/blog/setfit. 
    - This approach would add a "second opinion" functionality to predictions and an agreement analysis might help us figure out where models are weak and identify a focus for improvement.

## Notes from meeting

### Datasets

* PAN 2012 Perverted Justice anonymised dataset (publicly available). [Paper link here.](https://ceur-ws.org/Vol-1178/CLEF2012wn-PAN-InchesEt2012.pdf)
* Law enforcement non-anonymised dataset provided by UK Police to the DRAGON-S Project.

### Tasks

- **Baseline**: Grooming binary outcome
- **Tactics**: Multi-label classification of tactics (N=13)

### Test datasets

The PAN and LE data have been arranged into a train/test split over multiple configurations. *We should recommend which is most relevant to give confidence of a model deployed within UK LE.*

- **Tactics**: All data with tactics (LE + Neg PJ annotated)
- **LE-weighted**: LE data to which negative conversations from PAN have been added ~ LE conversations represent 2% of the final corpus approximately
- **LE-imbalanced**: LE data + all of PAN's negative conversations.
- **All**: same as LE-imbalanced + PAN's positive conversations.
- **PAN**: The PAN data only

### Models

Various model experiments have been tried that include training on the same configurations to match the 5 different test sets, as well as additional experiments that pre-train on baseline grooming and then finetune using tactics information. 

---

## Project environment

- **Compute Access**
    - via ssh to a machine in Swansea (may req) ssh may require access to toran
- **Git server** https://cs-dragons.swansea.ac.uk/git/user/login
    - 2 repos we will be working on for this collaboration: 
        - `embedding_creation`
        - `refactorisation` 
    - We will create PRs to the `refactor` branches in both
- **Experiment Design**
    - PyTorch Lightning 
    - Managing dependencies with conda environments (environment.yml files)
    - Using .yml files to pass model params experiments/config.yml
    - experiment folder contains different configuration files
    - experiment/scripts/ contains metrics and outputs
    - main.py has functionality for different options
        - training and testing
        - testing
        - data loader
        - checkpoints 
- **Data Loader** (shouldn't need to be changed): try to optimize how much data can be stored in memory:
    - sorts by convo length and creates batches of similar length



