---
# System prepended metadata

title: 'GA2: Exploration Vince'

---

# GA2: Exploration Vince

## Initial Run

### Network
![](https://i.imgur.com/NRNk72Q.png)
batch: 32
epochs: 10

### Performance
![](https://i.imgur.com/wDFN0nS.png)


## From Github

### Network
![](https://i.imgur.com/q7jJssE.png)
batch: 32
epochs: 10

## Performance
![](https://i.imgur.com/EyDbgs3.png)
Training set Accuracy: 0.6513
Training set Loss: 1.0855

Validation set Accuracy: 0.4907
Validation set Loss: 1.7243

**More epochs: 20**

![](https://i.imgur.com/EYOFU8U.png)
Training set Accuracy: 0.8518
Training set Loss: 0.4523

Validation set Accuracy: 0.4918
Validation set Loss: 2.4849

**We are clearly overfitting now.**
When we try to increase the batch size to 64, we get the following
![](https://i.imgur.com/2fHMTG3.png)
More squiggly curves

If we increase the batch size even more (128)
![](https://i.imgur.com/OVHHArM.png)
Not much difference

**Lets add some dropout (0.4)**
![](https://i.imgur.com/lyiyUDX.png)

Looks a bit cleaner. Best values found at epoch 12

## Other network found

### Network
![](https://i.imgur.com/IZl5bVO.png)
batch: 32
epochs: 10
**This network is fucking massive**

## Performance
![](https://i.imgur.com/4699IOR.png)

**Yikes**

Lets remove the dropout.

![](https://i.imgur.com/cwAlfvW.png)

I guess its better but no clue what happened in epoch 16 :o. Lets keep it at 10 epochs.

![](https://i.imgur.com/r71D90e.png)

I dont really like this model, its not that powerful, yet it has 7 million parameters. Lets skip it for now.

## One more found
![](https://i.imgur.com/kRAlYKo.png)

batch: 32
epochs: 10

## Performance
![](https://i.imgur.com/rwWAybB.png)
Training set Accuracy: 0.7575
Training set Loss: 0.7547

Validation set Accuracy: 0.4498
Validation set Loss: 2.1709

Not to shabby, we see increase in validation loss from epoch 7. We still try to give it more epochs. Now 20 instead of 10

![](https://i.imgur.com/AZHJE4q.png)

Indeed, we get a major overfit. Lets try to add some dropout to hopefully decrease this overfit a bit

![](https://i.imgur.com/YWOqrAN.png)

Now its just terrible. Maybe the other model was also this bad because of the dropout, lets remove it.

Lets try to add batchnormalization to the layers

![](https://i.imgur.com/N5Qx7rX.png)

Alright alright, not too bad. Maybe make the dense layer a bit smaller

![](https://i.imgur.com/fDQzbb3.png)

Does not really make a difference in performance but seriously decreases the amount of parameters.

Now we try to play a bit with the convolutions.

![](https://i.imgur.com/R5npTE8.png)

Very similar, mabye worse but the amount of parameters has doubled. So lets not do this.

Next I tried to change the activation function to elu instead of relu. It might be nice if the model can use negative weights?

![](https://i.imgur.com/5KGRvUm.png)

Generally better scores, more validations over 60 in epochs.

![](https://i.imgur.com/hj5oRJI.png)
![](https://i.imgur.com/rwo11u1.png)

![](https://i.imgur.com/iGxPssy.png)
![](https://i.imgur.com/83QXp6A.png)

We add some dropout (0.3)

![](https://i.imgur.com/ugGBSTK.png)


## Our network
![](https://i.imgur.com/3472Nss.png)
![](https://i.imgur.com/jB9vch3.png)
Training set Accuracy: 0.9999
Training set Loss: 0.0003

Validation set Accuracy: 0.6317
Validation set Loss: 1.7453