# Pertubation Experiments
# ML 2021/09/23
Refers to [You don't know JAX](https://colinraffel.com/blog/you-don-t-know-jax.html)
## Conclusion
## Next Goal
## Results
The way we compute the norm of the `gradient` is the same as the `perturbation`. Both of them are the average of the Euclidean norm.
## Experiment Group A
### Common Settings
The experiment 1 ~ 7 are a group of experiments for searching t and perturbation coefficient
- t: 65536.0 * [64, 55, 50, 45, 40, 35, 30]
- Perturbation Coefficient: [0.004, 0.0045, 0.005, 0.0055, 0.006]
The experiment 1 ~ 7 have following mutual settings
- Batch Size: 2048
- Dataset Size: 2048
- Noise Size: 10
- Epoch: 10000
- Exponential Decay Perturbation
### Experiment 1: Exponential Decay Method With t = 64
- T: 65536.0 * 64
Without perturbation
#### Perturbation Coefficient = 0.004
**Curves**
#### Perturbation Coefficient = 0.005
**Curves**
#### Perturbation Coefficient = 0.0055
**Curves**
### Experiment 2: Exponential Decay Method With t = 55
- T: 65536.0 * 55
Without perturbation
#### Perturbation Coefficient = 0.004
**Curves**
#### Perturbation Coefficient = 0.0045
**Curves**
#### Perturbation Coefficient = 0.005
**Curves**
#### Perturbation Coefficient = 0.0055
**Curves**
#### Perturbation Coefficient = 0.006
**Curves**
### Experiment 3: Exponential Decay Method With t = 50
- T: 65536.0 * 50
Without perturbation
#### Perturbation Coefficient = 0.004
**Curves**
#### Perturbation Coefficient = 0.0045
**Curves**
#### Perturbation Coefficient = 0.005
**Curves**
#### Perturbation Coefficient = 0.0055
**Curves**
#### Perturbation Coefficient = 0.006
**Curves**
### Experiment 4: Exponential Decay Method With t = 45
- T: 65536.0 * 45
Without perturbation
#### Perturbation Coefficient = 0.004
**Curves**
#### Perturbation Coefficient = 0.0045
**Curves**
#### Perturbation Coefficient = 0.005
**Curves**
#### Perturbation Coefficient = 0.0055
**Curves**
#### Perturbation Coefficient = 0.006
**Curves**
### Experiment 5: Exponential Decay Method With t = 40
- T: 65536.0 * 40
Without perturbation
#### Perturbation Coefficient = 0.004
**Curves**
#### Perturbation Coefficient = 0.0045
**Curves**
#### Perturbation Coefficient = 0.005
**Curves**
#### Perturbation Coefficient = 0.0055
**Curves**
#### Perturbation Coefficient = 0.006
**Curves**
### Experiment 6: Exponential Decay Method With t = 35
- T: 65536.0 * 35
Without perturbation
#### Perturbation Coefficient = 0.004
**Curves**
#### Perturbation Coefficient = 0.0045
**Curves**
#### Perturbation Coefficient = 0.005
**Curves**
#### Perturbation Coefficient = 0.0055
**Curves**
#### Perturbation Coefficient = 0.006
**Curves**
### Experiment 7: Exponential Decay Method With t = 30
- T: 65536.0 * 30
Without perturbation
#### Perturbation Coefficient = 0.004
**Curves**
#### Perturbation Coefficient = 0.0045
**Curves**
#### Perturbation Coefficient = 0.005
**Curves**
#### Perturbation Coefficient = 0.0055
**Curves**
#### Perturbation Coefficient = 0.006
**Curves**
# ML 2021/09/17
## Conclusion
1. With fixed batch size, the larger dataset size, the the blurrier image.
2. The effective perturbation coefficient for exponential perturbation is about < 0.007 and > 0.004. When the coefficient is too large, the image would contain some noise and need more epoch to converge. The image would become pure noise after the coefficient > 0.007.
3. It seems that the perturbation is good at making the fake image a little bit different from the ground truth image, but just a little, for example, changing skin color seems easily to success like [this](#perturbation-coefficient-=-0.004). According to the results, We can speculate that **the acceptance region of the discreminator(means the region taht the discriminator judge as true) is very small and close to the ground truth**. On the other hand, when the perturbation is too large, that is, exceed the acceptance region of the discreminator, the image would become blurry or vague. Guess it is due to the discriminator might not fit well on the manifold out of the acceptance region of the discreminator.
4. Note that, I've tried larger training epoch 30000 and different batch size/dataset size/t combinations, but all of them don't help. The images are still blurry, so we can exclude the underfit/improper dataset size/improper t as the possible resaons.
5. When it comes to larger dataset size, the pertubation tolerance would be lower. That is, it can only accept smaller perturbation.
## Next Goal
Next time, I will try larger batch size and dataset size, it may extend the acceptance region of the the discreminator. By extending the acceptance region, it may help to generate more creative image.
## Results
The way we compute the norm of the `gradient` is the same as the `perturbation`. Both of them are the average of the Euclidean norm.
### Experiment 1: Exponential Decay Method With 256 Batch Size & 256 Dataset Size
- Batch Size: 256
- Dataset Size: 256
- Noise Size: 10
- Epoch: 10000
- T: 65536.0 * 6
- Exponential Decay
Without perturbation
#### Perturbation Coefficient = 0.004
No effect
**Curves**
#### Perturbation Coefficient = 0.005
Pretty great creativity
**Curves**
#### Perturbation Coefficient = 0.0055
**Curves**
#### Perturbation Coefficient = 0.006
**Curves**
#### Perturbation Coefficient = 0.0065
**Curves**
#### Perturbation Coefficient = 0.007
**Curves**
#### Perturbation Coefficient = 0.008
Totally blury
**Curves**
### Experiment 2: Exponential Decay Method With 256 Batch Size & 300 Dataset Size
- Batch Size: 256
- Dataset Size: 300
- Noise Size: 10
- Epoch: 10000
- T: 65536.0 * 6
- Exponential Decay
#### Perturbation Coefficient = 0.004
**Curves**
#### Perturbation Coefficient = 0.005
**Curves**
#### Perturbation Coefficient = 0.0055
**Curves**
#### Perturbation Coefficient = 0.006
**Curves**
#### Perturbation Coefficient = 0.0065
**Curves**
### Experiment 4: Exponential Decay Method With 256 Batch Size & 400 Dataset Size
- Batch Size: 256
- Dataset Size: 400
- Noise Size: 10
- Epoch: 10000
- T: 65536.0 * 6
- Exponential Decay
#### Perturbation Coefficient = 0.004
**Curves**
#### Perturbation Coefficient = 0.005
**Curves**
#### Perturbation Coefficient = 0.0055
**Curves**
#### Perturbation Coefficient = 0.006
**Curves**
#### Perturbation Coefficient = 0.0065
**Curves**
### Experiment 5: Exponential Decay Method With 256 Batch Size & 512 Dataset Size
- Batch Size: 256
- Dataset Size: 512
- Noise Size: 10
- Epoch: 10000
- T: 65536.0 * 6
- Exponential Decay
Without perturbation
#### Perturbation Coefficient = 0.003
**Curves**
#### Perturbation Coefficient = 0.004
**Curves**
#### Perturbation Coefficient = 0.005
**Curves**
#### Perturbation Coefficient = 0.0055
**Curves**
#### Perturbation Coefficient = 0.006
**Curves**
### Experiment 6: Fixed Exponential Decay Method With Different 'training_t_rate'
- Batch Size: 256
- Dataset Size: 256
- Noise Size: 10
- Epoch: 10000
- T: training_t_unit * training_t_rate = 65536.0 * [6, 5, 4, 3, 2]
- Exponential Decay, Perturbation Coefficient = 0.005
#### training_t_rate = 6.0
Without perturbation
With perturbation
**Curves**
#### training_t_rate = 5.0
Without perturbation
With perturbation
**Curves**
#### training_t_rate = 4.0
Without perturbation
With perturbation
**Curves**
#### training_t_rate = 3.0
Without perturbation
With perturbation
**Curves**
#### training_t_rate = 2.0
Without perturbation
With perturbation
**Curves**
# ML 2021/09/02
## Goal
Add perturbation on the gradient to make the generator more creative.
## Method
Denote $g$ as the gradient that update on image, and $\eta$ as the learning rate.
$$
grad \_ accumulator = gradient + \eta \cdot perturbation
$$
With Yurong's momentum update
$$
gradint = \eta \cdot grads + momentum \cdot grad \_ accumulator
$$
Update the image
$$
x \_noise = x \_noise - grad \_ accumulator
$$
Denote the **perutrbation coefficient** as $\gamma$ in the follow formulas. The `gradient` is the momentum gradient, so the implementation of the perturbation is
```python=1
# add to momentum
perturb = noise_generator.generate(shape=x_noise.shape)
grad_accumulator = grads * learning_rate + momentum * grad_accumulator + learning_rate * perturb
```
### Method 1: Static Noise
$$
perturbation = \mathcal{N}(0, 1) \cdot \gamma
$$
### Method 2: Exponential Decay
Decay along the epochs.
$$
perturbation = \mathcal{N}(0, 1) \cdot \gamma^{epoch}
$$
### Method 3: Anneal Decay
$$
perturbation = \frac{\mathcal{N}(0, 1)}{(1 + epoch)^{\gamma}}
$$
Refers to [Adding Gradient Noise Improves Learning for Very Deep Networks](https://arxiv.org/abs/1511.06807)
## Perturbation Norm & Gradient Norm
## Results
The way we compute the norm of the `gradient` is the same as the `perturbation`. Both of them are the average of the Euclidean norm.
Common Settings
- Batch Size: 256
- Dataset Size: 256
- Noise Size: 10
- Epoch: 10000
- T: 65536.0 * 6
### Exponential Decay
#### Perturbation Coefficient = 0
Same as no-perturbation
#### Perturbation Coefficient = 0.009
The image become pure noise.
#### Perturbation Coefficient = 0.09
The image become pure noise.
#### Perturbation Coefficient = 0.9, 0.99, 0.999
The image become pure noise.
### Static
#### Perturbation Coefficient = 1e-7
Some images get blurry.
#### Perturbation Coefficient = 1e-6, 1e-5, 1e-3,
The image become pure noise.
Sign in with Wallet
Connect another wallet