# Bioinformatics A.A. 2018/2019 - Report Sviluppo di un set di reti neurali, basate su LSTM, per la discriminazione di sequenze oncogeniche in fusioni di geni. - **Stefano Brilli** - s249914@studenti.polito.it - **Francesco Chiarlo** - s253666@studenti.polito.it - **Michele D'Amico** - s246501@studenti.polito.it ## Architetture & Risultati Per valutare in maniera generale e comparabile i risultati di ogni modello, si è optato per un approccio in tre fasi: - **1 - Holdout** - Selezione dei migliori parametri; - **2 - Training** - Fit del modello con i parametri scelti; - **2.a<sup>*</sup>** algoritmo 7.2 attraverso opzione: `--early_stopping_epoch` - **2.b<sup>*</sup>** algoritmo 7.3 attraverso opzione: `--early_stopping_on_loss` - **3 - Testing** - Predizione su dati *mai visti* dal modello. Nota che nello stilare i seguenti risultati è stato utilizzato esclusivamente l'algoritmo **2.b**, in quanto ha fornito migliori prestazioni. > <sup>*</sup> (Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. The MIT Press, pp. 246-250.) --- &nbsp; ### Modello #1 - LSTM unidirezionale su sequenze di DNA &nbsp; <img src=https://i.imgur.com/LBuTDAX.png height=180> &nbsp; **Caratteristiche della rete:** - Codifica one-hot - Dimensione del batch: **16** - Numero di unità del layer FC: **32** - Dropout rate: **0.4** #### Risultati: |Holdout| learning rate|LSTM units|Dropout|ES_epoch|Loss|Accuracy | F1|Precision|Recall| |:-----:|:------------:|:--------:|:-----:|:------:|:--:|:-------:|:-:|:-------:|:----:| | 1|1e-3|32|0.1| 18|0.589|0.738|0.689|0.838|0.627| | 2|1e-3|32|0.3| 31|0.609|0.753|0.707|0.862|0.629| | 3|1e-3|32|0.5| 34|0.586|0.738|0.708|0.812|0.658| | 4|1e-4|48|0.1| 5|0.609|0.671|0.572|0.799|0.477| | 5|1e-3|48|0.1| 26|0.715|0.563|0.367|0.589|0.289| | 6|1e-4|48|0.3| 23|0.605|0.740|0.696|0.835|0.629| | 7|1e-4|48|0.5| 56|0.709|0.587|0.421|0.663|0.331| | 8|1e-3|64|0.1| 18|0.581|0.740|0.698|0.814|0.646| | 9|1e-4|64|0.1| 47|0.639|0.704|0.619|0.778|0.535| |10|5e-4|64|0.3| 12|0.602|0.717|0.614|0.852|0.512| |11|1e-3|64|0.3| 24|0.596|0.758|0.723|0.839|0.659| |12|1e-3|64|0.5| 6|0.622|0.686|0.604|0.799|0.525| |13|1e-4|64|0.5| 45|0.628|0.693|0.616|0.734|0.549| |**14**|**1e-4**|**16**|**0.1**|**24**|**0.539**|**0.799**|**0.799**|**0.820**|**0.802**| |15|1e-4|16|0.3| 24|0.559|0.717|0.701|0.746|0.691| |16|1e-4|16|0.5| 24|0.559|0.755|0.720|0.825|0.659| |Test|Modello|Loss|Acc |F1-Score|Precision|Recall|AP |TN |FP |FN |TP | |:--:|:--:|:--:|:--:|:------:|:-------:|:----:|:-:|:--:|:--:|:--:|:--:| |-|14|0.386|0.871|0.865|0.919|0.828|0.930|211|19|40|190| --- &nbsp; ### Modello #2 - LSTM bidirezionale su sequenze di DNA &nbsp; <img src=https://i.imgur.com/wi2kkeR.png height=180> &nbsp; <!-- ![](https://i.imgur.com/wi2kkeR.png =x180) --> **Caratteristiche della rete:** - Codifica one-hot - Max pooling size: **2** - Dimensione del batch: **20** - Learning rate: **5e-4** #### Risultati: |Holdout|Dropout|lstm_units|kernel_size|conv_num_filters|ES_epoch|Loss|Accuracy| F1 |Precision|Recall| |:-----:|:-----:|:--------:|:---------:|:--------------:|:------:|:--:|:------:|:--:|:-------:|:----:| | 1|[0.1;0.1;0.1;0.1]| 16| 3 | 50| 7 | 0.57295 | 0.73866 | 0.71711 | 0.77771 | 0.68804 | | 2|[0.1;0.1;0.1;0.1]| 16| 5 | 50| 5 | 0.64208 | 0.69978 | 0.62190 | 0.77794 | 0.54552 | | 3|[0.1;0.1;0.1;0.1]| 16| 10| 50| 3 | 0.62138 | 0.71490 | 0.61748 | 0.87659 | 0.50720 | | 4|[0.3;0.3;0.3;0.3]| 16| 3 | 50| 42| 0.60901 | 0.69330 | 0.68989 | 0.71246 | 0.71297 | | 5|[0.3;0.3;0.3;0.3]| 16| 5 | 50| 17| 0.50554 | 0.76674 | 0.69009 | 0.89745 | 0.60612 | | 6|[0.3;0.3;0.3;0.3]| 16| 10| 50| 8 | 0.62940 | 0.69114 | 0.70468 | 0.69709 | 0.76915 | | 7|[0.3;0.3;0.3;0.3]| 32| 5 | 50| 15| 0.49978 | 0.76026 | 0.71858 | 0.85039 | 0.65371 | | 8|[0.3;0.3;0.3;0.3]| 32| 10| 50| 6| 0.62290 | 0.68035 | 0.61435 | 0.80636 | 0.52854 | |**9**|**[0.3;0.3;0.3;0.3]**|**16**|**3**| **25**| **48**| **0.48879** | **0.79914** | **0.77643** | **0.86375** | **0.72218** | |10|[0.3;0.3;0.3;0.3]| 16| 5 | 25| 14| 0.55411 | 0.78186 | 0.74186 | 0.85337 | 0.69907 | |11|[0.3;0.3;0.3;0.3]| 16| 10| 25| 9 | 0.64211 | 0.63715 | 0.62165 | 0.68523 | 0.60172 | | Test | Modello | Loss | Accuracy | F1 | Precision | Recall | AP | TN | FP | FN | TP | |:------:|:---:|:----:|:--------:|:--:|:---------:|:------:|:---:|:--:|:--:|:--:|:--:| | -| 9 |0.706| 0.711|0.677|0.750|0.633| 0.799| 180 | 50|83|147| --- &nbsp; ### Modello #3 - LSTM unidirezionale su sequenze di proteine &nbsp; <img src=https://i.imgur.com/lA81Rtd.png height=200> &nbsp; **Caratteristiche della rete:** - Codifica one-hot #### Risultati: | Holdout | Layer size\* | Dropout\* | L2 Regularization\* | Learning rate | Batch size | ES_epoch | Loss | Accuracy | F1 | Precision | Recall | |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|-| | 1| [128;128;128] | [0.5;0.5;0.5;0.5] | [0;0;0;0] | 1e-4 | 32 | 11 | 0.51546 | 0.73002 | 0.63162 | 0.94690 | 0.50496| | 2| [128;128;128] | [0.5;0.5;0.5;0.5] | [1e-3;1e-3;1e-3;1e-3] | 1e-4 | 32 | 10 | 0.73175 | 0.73218 | 0.64949 | 0.88817 | 0.54384| | 3| [128;128;128] | [0.5;0.5;0.5;0.5] | [1e-4;1e-4;1e-4;1e-4] | 1e-4 | 32 | 11 | 0.54805 | 0.73002 | 0.63080 | 0.94690 | 0.50474| | 4| [128;128;128] | [0.5;0.5;0.7;0.7] | [0;0;0;0] | 1e-4 | 32 | 11 | 0.52107 | 0.71490 | 0.62148 | 0.89304 | 0.51461| | 5| [128;128;128] | [0.3;0.3;0.5;0.5] | [0;0;0;0] | 1e-4 | 32 | 11 | 0.53654 | 0.74082 | 0.66415 | 0.87473 | 0.57688| | 6| [128;128;128] | [0.1;0.1;0.5;0.5] | [0;0;0;0] | 1e-4 | 32 | 10 | 0.52756 | 0.72570 | 0.62877 | 0.92785 | 0.50865| | 7| [128;128;128] | [0.4;0.4;0.2;0.2] | [0;0;0;0] | 5e-4 | 32 | 11 | 0.51169 | 0.72786 | 0.68798 | 0.85348 | 0.60160| | 8| [128;128;128] | [0.4;0.4;0.5;0.5] | [0;0;0;0] | 5e-4 | 32 | 11 | 0.51184 | 0.73650 | 0.64521 | 0.92785 | 0.52804| | 9| [128;128;128] | [0.4;0.4;0.8;0.8] | [0;0;0;0] | 5e-4 | 32 | 10 | 0.53321 | 0.70842 | 0.63742 | 0.81173 | 0.55339| |10| [128;128;128] | [0.4;0.4;0.8;0.8] | [0;0;0;0] | 1e-3 | 32 | 6 | 0.51737 | 0.73002 | 0.69366 | 0.79503 | 0.63938| |11| [128;128;64] | [0.5;0.5;0.5;0.5] | [0;0;0;0] | 1e-4 | 32 | 10 | 0.52251 | 0.73002 | 0.66371 | 0.87113 | 0.57291| |12| [128;128;64] | [0.5;0.5;0.5;0.5] | [0;0;0;0] | 1e-4 | 16 | 7 | 0.53382 | 0.74730 | 0.69236 | 0.87422 | 0.60221| |13| [64;64;64] | [0.5;0.5;0.5;0.5] | [0;0;0;0] | 1e-4 | 32 | 24 | 0.52053 | 0.74082 | 0.64615 | 0.91960 | 0.54510| |14| [64;64;32] | [0.3;0.3;0.3;0.3] | [0;0;0;0] | 1e-4 | 16 | 17 | 0.52438 | 0.74730 | 0.65596 | 0.82823 | 0.57388| |15| [32;32;32] | [0.5;0.5;0.5;0.5] | [0;0;0;0] | 1e-4 | 32 | 39 | 0.53357 | 0.74730 | 0.70721 | 0.87628 | 0.63022| |16| [32;32;32] | [0.5;0.5;0.5;0.5] | [0;0;0;0] | 1e-4 | 16 | 25 | 0.52169 | 0.76890 | 0.72858 | 0.88953 | 0.64982| |17| [32;32;32] | [0.3;0.3;0.5;0.5] | [0;0;0;0] | 1e-4 | 32 | 28 | 0.53589 | 0.75162 | 0.70089 | 0.87739 | 0.62547| |**18**|**[16;16;32]**|**[0.3;0.3;0.3;0.3]**|**[0;0;0;0]** |**1e-4**|**16**|**27**|**0.53081**| **0.75378** | **0.76873** | **0.77319** | **0.79635**| \* [LSTM; LSTM; FC] | Test | Modello | Loss | Accuracy | F1 | Precision | Recall | AP | TN | FP | FN | TP | |:------:|:---:|:----:|:--------:|:--:|:---------:|:------:|:---:|:--:|:--:|:--:|:--:| | -| 18 | 0.708 | 0.652 | 0.643 |0.680| 0.634 | 0.697| 153 | 77 | 83 | 147 | --- &nbsp; ### Modello #4 - LSTM bidirezionale su sequenze di proteine &nbsp; ##### Modello A <img src=https://i.imgur.com/c6fa65w.png height=250> &nbsp; **Caratteristiche della rete:** - Codifica embedding - Dimensione del batch: **32** - Learning rate iniziale: **5e-4** - Dropout rate: **[0.2, 0.4, 0.6]** - Numero di unità LSTM: **0.5 \* emb_size** #### Risultati: |Holdout|Dropout|emb_size|ES_epoch|Loss|Accuracy| F1|Precision|Recall| |:-----:|:-----:|:------:|:------:|:--:|:------:|:-:|:-------:|:----:| |A1 |[0.2,0.2,0.1]|16|20|0.674|0.736|0.723|0.766|0.699| |A2 |[0.2,0.2,0.3]|16|61|0.605|0.711|0.680|0.797|0.620| |A3 |[0.2,0.2,0.5]|16|45|0.628|0.739|0.727|0.755|0.725| |A4 |[0.4,0.4,0.1]|16|45|0.634|0.745|0.737|0.769|0.729| |A5 |[0.4,0.4,0.3]|16|19|0.670|0.711|0.718|0.726|0.731| |A6 |[0.4,0.4,0.5]|16|37|0.644|0.719|0.729|0.717|0.765| |A7 |[0.6,0.6,0.1]|16|35|0.635|0.730|0.729|0.752|0.726| |A8 |[0.6,0.6,0.3]|16|24|0.663|0.752|0.728|0.791|0.687| |A9 |[0.6,0.6,0.5]|16|47|0.624|0.732|0.743|0.721|0.787| |**A10**|**[0.2,0.2,0.1]**|**32**|**35**|**0.673**|**0.767**|**0.743**|**0.834**|**0.685**| |A11|[0.2,0.2,0.3]|32|47|0.643|0.732|0.666|0.881|0.559| |A12|[0.2,0.2,0.5]|32|45|0.692|0.719|0.726|0.720|0.752| |A13|[0.4,0.4,0.1]|32|31|0.731|0.741|0.717|0.797|0.668| |A14|[0.4,0.4,0.3]|32|32|0.667|0.754|0.730|0.781|0.705| |A15|[0.4,0.4,0.5]|32|35|0.740|0.708|0.713|0.752|0.707| |A16|[0.6,0.6,0.1]|32|26|0.717|0.713|0.724|0.719|0.748| |A17|[0.6,0.6,0.3]|32|19|0.764|0.706|0.699|0.754|0.670| |A18|[0.6,0.6,0.5]|32|26|0.696|0.717|0.724|0.711|0.751| &nbsp; ##### Modello B <img src=https://i.imgur.com/1PK16VV.png height=180> &nbsp; **Caratteristiche della rete:** - Codifica one-hot - Dimensione del batch: **20** - Learning rate iniziale: **5e-4** - Dropout rate: **[0.5, 0.6]** - Numero di unità LSTM: **[6, 10]** - Dimensione kernel convoluzionale: **3** - Max pooling size: **2** #### Risultati: |Holdout|Dropout|lstm_units|conv_num_filters|ES_epoch|Loss |Accuracy|F1 |Precision|Recall| |:-----:|:-----:|:--------:|:--------------:|:------:|:---:|:------:|:--:|:-------:|:----:| |B1|[0.5,0.6,0.5]| 6|50|10|0.550|0.721|0.721|0.758|0.738| |B2|[0.5,0.5,0.6]| 6|50|12|0.545|0.717|0.717|0.740|0.741| |B3|[0.6,0.5,0.5]| 6|50|12|0.554|0.726|0.714|0.763|0.711| |B4|[0.5,0.6,0.6]| 6|50|10|0.541|0.715|0.722|0.756|0.727| |B5|[0.6,0.5,0.6]| 6|50|12|0.544|0.728|0.716|0.763|0.715| |B6|[0.6,0.6,0.5]| 6|50|12|0.546|0.724|0.707|0.786|0.675| |**B7**|**[0.6,0.6,0.6]**| **6**|**50**|**10**|**0.541**|**0.715**|**0.722**|**0.756**|**0.727**| |B8|[0.5,0.5,0.5]|10|50|12|0.544|0.706|0.680|0.779|0.639| |Test|Modello|Loss |Accuracy| F1|Precision|Recall|AP |TN |FP |FN |TP | |:--:|:-----:|:---:|:------:|:-:|:-------:|:----:|:--:|:--:|:--:|:--:|:--:| |-|A10|0.729|0.652|0.610|0.709|0.551|0.760|173|57|103|127| |-|B7|0.708|0.667|0.615|0.722|0.584|0.772|172|58|95|135|