# GA2 Attempts vince

## Model with dropouts in convolutions


Training set Accuracy: 0.9690
Training set Loss: 0.0892

Validation set Accuracy: 0.6523
Validation set Loss: 1.9806

## Same model but higher learning rate 0.01
Terrible, stopped early

## Same model but lower learning rate 0.0005

Training set Accuracy: 0.9795
Training set Loss: 0.0621

Validation set Accuracy: 0.6650
Validation set Loss: 1.8523

Pretty decent, might perform even better with more epochs. Is slow tho.

## Lets try halving the model :p


Training set Accuracy: 0.5536
Training set Loss: 1.7838

Validation set Accuracy: 0.3993
Validation set Loss: 3.0703

Pure shit

## Lets try halving the amount of feature maps of the model

Training set Accuracy: 0.9317
Training set Loss: 0.2108

Validation set Accuracy: 0.6467
Validation set Loss: 1.4852

Not too bad, i assume that more epochs might help!

## Same previous model but with learning rate of 0.0008

Training set Accuracy: 0.9016
Training set Loss: 0.2942

Validation set Accuracy: 0.6180
Validation set Loss: 1.5384

Certainly shows potential, is not overfitting on train yet. We can increase epochs or increase learning rate.

## Same previous model but with learning rate of 0.0012

Training set Accuracy: 0.9070
Training set Loss: 0.2851

Validation set Accuracy: 0.6222
Validation set Loss: 1.6377

I am starting to like what i see. Lets try to increase the learning rate even more.

## Same previous model but with learning rate of 0.002

Training set Accuracy: 0.9284
Training set Loss: 0.2131

Validation set Accuracy: 0.6448
Validation set Loss: 1.6223

## Same previous model but with even higher learning rate? 0.0035


Training set Accuracy: 0.9429
Training set Loss: 0.1737

Validation set Accuracy: 0.6622
Validation set Loss: 1.4686

Is a pretty decent curve. Maybe the batch size is a bit too large. Lets try to bring it down a bit

## Same previous model but with even higher learning rate? 0.0035, batch 256

Training set Accuracy: 0.9494
Training set Loss: 0.1608

Validation set Accuracy: 0.6560
Validation set Loss: 1.4607

Not that much better, lets keep the batch size and throw some more epochs at it

## Same previous model but with even higher learning rate? 0.0035, 70 epochs


Hell ye, got some serious potential

Training set Accuracy: 0.9413
Training set Loss: 0.1885

Validation set Accuracy: 0.6440
Validation set Loss: 1.7422

## General
Tried deep model: issue, not stable, certainly in the last layers
Tried very wide model: issue, a lot of parameters, learns very slow

The decreased the wideness of the model: way less parameters. Still pretty good. Is not overfitting tho. More parameters? More epochs?

### Model that i tried a lot before but without dropout and with dense layer added
Training set Accuracy: 0.9788
Training set Loss: 0.0661

Validation set Accuracy: 0.6138
Validation set Loss: 2.6347

I notice jumping around 1.0, to high learning rate??

### Now without the extra dense layer

Jumping around 1.0, to high learning rate??

### Now with a wider dense layer? 512

## Meeting

### Model jason - val 63%
Pretty decent

### Model jason with reg: dropout 0.3 and 0.4 (lr 0.0014)

Training set Accuracy: 0.9689
Training set Loss: 0.0966

Validation set Accuracy: 0.6557
Validation set Loss: 1.6772

### Adding two layers of 512 to model of jason, unreg

Training set Accuracy: 0.9999
Training set Loss: 0.0003

Validation set Accuracy: 0.7203
Validation set Loss: 1.5902

## Adding two layers of 512, dropout 0.3, 0.4

Training set Accuracy: 0.9748
Training set Loss: 0.0747

Validation set Accuracy: 0.6742
Validation set Loss: 1.7410

## Adding two layers of 512, dropout 0.1, 0.25

Training set Accuracy: 0.9793
Training set Loss: 0.0618

Validation set Accuracy: 0.6607
Validation set Loss: 1.9741

## Only one layer of 512, reg 0.05, 0.2, 0.3

Training set Accuracy: 0.9774
Training set Loss: 0.0706

Validation set Accuracy: 0.6635
Validation set Loss: 2.0208

## Only one layer of 512, reg 0.1, 0.3, 0.4

Training set Accuracy: 0.9752
Training set Loss: 0.0724

Validation set Accuracy: 0.6517
Validation set Loss: 2.0366

## Only 0.5 in the second to last layer

Training set Accuracy: 0.9850
Training set Loss: 0.0434

Validation set Accuracy: 0.6698
Validation set Loss: 1.9377

## Very nice model!

Training set Accuracy: 0.9880
Training set Loss: 0.0382

Validation set Accuracy: 0.6972
Validation set Loss: 1.7832