---
# System prepended metadata

title: Pertubation Experiments

---

# Pertubation Experiments

# ML 2021/09/23

Refers to [You don't know JAX](https://colinraffel.com/blog/you-don-t-know-jax.html)

## Conclusion



## Next Goal



## Results

The way we compute the norm of the `gradient` is the same as the `perturbation`. Both of them are the average of the Euclidean norm.

## Experiment Group A

### Common Settings

The experiment 1 ~ 7 are a group of experiments for searching t and perturbation coefficient

- t: 65536.0 * [64, 55, 50, 45, 40, 35, 30]

- Perturbation Coefficient: [0.004, 0.0045, 0.005, 0.0055, 0.006]

The experiment 1 ~ 7 have following mutual settings

- Batch Size: 2048
- Dataset Size: 2048
- Noise Size: 10
- Epoch: 10000
- Exponential Decay Perturbation

### Experiment 1: Exponential Decay Method With t = 64

- T: 65536.0 * 64

Without perturbation



#### Perturbation Coefficient = 0.004

![](https://i.imgur.com/4lvQUhQ.png)
![](https://i.imgur.com/RMBL4sX.png)
![](https://i.imgur.com/5h0yLu2.png)
![](https://i.imgur.com/ackIN7h.png)

**Curves**

![](https://i.imgur.com/DWbyrS3.png)
![](https://i.imgur.com/nwAYmK4.png)
![](https://i.imgur.com/7Cgy3aY.png)

#### Perturbation Coefficient = 0.005

![](https://i.imgur.com/0TSWs5n.png)

**Curves**

![](https://i.imgur.com/0AqBOWA.png)
![](https://i.imgur.com/A3SZvUu.png)
![](https://i.imgur.com/xkRFsxJ.png)

#### Perturbation Coefficient = 0.0055

![](https://i.imgur.com/h1il2BP.png)

**Curves**

![](https://i.imgur.com/hBXoBv6.png)
![](https://i.imgur.com/F4zLt9S.png)
![](https://i.imgur.com/M3czUqO.png)

### Experiment 2: Exponential Decay Method With t = 55

- T: 65536.0 * 55

Without perturbation



#### Perturbation Coefficient = 0.004

![](https://i.imgur.com/f3kYbH8.png)
![](https://i.imgur.com/WwbohSw.png)
![](https://i.imgur.com/HFO5t2Z.png)
![](https://i.imgur.com/I6dhPjJ.png)
![](https://i.imgur.com/vHvi2br.png)

**Curves**

![](https://i.imgur.com/2l82wVY.png)
![](https://i.imgur.com/BdKXcSr.png)
![](https://i.imgur.com/hJwKa4W.png)

#### Perturbation Coefficient = 0.0045

![](https://i.imgur.com/ZwKXhE1.png)
![](https://i.imgur.com/wmkAYYO.png)
![](https://i.imgur.com/StLSKgb.png)
![](https://i.imgur.com/MMcuOGE.png)
![](https://i.imgur.com/1TeWOCU.png)

**Curves**

![](https://i.imgur.com/a7qw0qE.png)
![](https://i.imgur.com/t90wFdE.png)
![](https://i.imgur.com/Ovkxg4T.png)

#### Perturbation Coefficient = 0.005

![](https://i.imgur.com/zneBVyK.png)
![](https://i.imgur.com/Uhb7YfF.png)
![](https://i.imgur.com/egQ3jOt.png)
![](https://i.imgur.com/x1Y4bd1.png)

**Curves**

![](https://i.imgur.com/5WhA24z.png)
![](https://i.imgur.com/bCIpV2i.png)
![](https://i.imgur.com/d886nmc.png)

#### Perturbation Coefficient = 0.0055

![](https://i.imgur.com/TzBEEs6.png)
![](https://i.imgur.com/TOa6LCf.png)
![](https://i.imgur.com/jFhvgZb.png)
![](https://i.imgur.com/8eKNvUZ.png)

**Curves**

![](https://i.imgur.com/vVk51q4.png)
![](https://i.imgur.com/6UudcHk.png)
![](https://i.imgur.com/UXpv68V.png)

#### Perturbation Coefficient = 0.006

![](https://i.imgur.com/TMqCXl8.png)
![](https://i.imgur.com/wOfmT7O.png)
![](https://i.imgur.com/fOQh3l3.png)
![](https://i.imgur.com/GRvO6cP.png)
![](https://i.imgur.com/jSbxywx.png)
![](https://i.imgur.com/35YwdtP.png)
![](https://i.imgur.com/Wc5TuvM.png)
![](https://i.imgur.com/xdVrqGg.png)
![](https://i.imgur.com/0mFDiXZ.png)

**Curves**

![](https://i.imgur.com/rKe2f1C.png)
![](https://i.imgur.com/ujocJxV.png)
![](https://i.imgur.com/3NpUVE4.png)

### Experiment 3: Exponential Decay Method With t = 50

- T: 65536.0 * 50

Without perturbation



#### Perturbation Coefficient = 0.004

![](https://i.imgur.com/VNblNTJ.png)
![](https://i.imgur.com/05aRaoA.png)
![](https://i.imgur.com/aZy3of1.png)
![](https://i.imgur.com/pxwZdDO.png)
![](https://i.imgur.com/zATx6v9.png)
![](https://i.imgur.com/IIWnlVp.png)

**Curves**

![](https://i.imgur.com/slsCceX.png)
![](https://i.imgur.com/9Ez5ob9.png)
![](https://i.imgur.com/3ps80Zo.png)

#### Perturbation Coefficient = 0.0045

![](https://i.imgur.com/Vo2FCQO.png)
![](https://i.imgur.com/vMjioTf.png)
![](https://i.imgur.com/DO1yCuX.png)
![](https://i.imgur.com/5LBttts.png)
![](https://i.imgur.com/pyHW5jO.png)

**Curves**

![](https://i.imgur.com/mJRQvwq.png)
![](https://i.imgur.com/vMp91ne.png)
![](https://i.imgur.com/2nyyWbs.png)

#### Perturbation Coefficient = 0.005

![](https://i.imgur.com/8zI2Xzp.png)
![](https://i.imgur.com/sz8nm1p.png)
![](https://i.imgur.com/zDCfm97.png)
![](https://i.imgur.com/pK8H7vN.png)

**Curves**

![](https://i.imgur.com/mVrmtnC.png)
![](https://i.imgur.com/mnBOVpz.png)
![](https://i.imgur.com/hIyX7ou.png)

#### Perturbation Coefficient = 0.0055

![](https://i.imgur.com/K4DQbpO.png)
![](https://i.imgur.com/w75X6sQ.png)
![](https://i.imgur.com/f0ebVnz.png)
![](https://i.imgur.com/B7gHN48.png)
![](https://i.imgur.com/MGvVSK6.png)
![](https://i.imgur.com/so1bZV3.png)
![](https://i.imgur.com/qZOW7Yn.png)

**Curves**

![](https://i.imgur.com/JI82aev.png)
![](https://i.imgur.com/gfhazXA.png)
![](https://i.imgur.com/oqUQPj9.png)

#### Perturbation Coefficient = 0.006

![](https://i.imgur.com/eU7L1WX.png)
![](https://i.imgur.com/JPb19Rk.png)
![](https://i.imgur.com/6qXbqJu.png)
![](https://i.imgur.com/Xx4WPa1.png)
![](https://i.imgur.com/QsGlLP9.png)

**Curves**

![](https://i.imgur.com/AvsTY0d.png)
![](https://i.imgur.com/MJ8kf2p.png)
![](https://i.imgur.com/GGtKz0c.png)

### Experiment 4: Exponential Decay Method With t = 45

- T: 65536.0 * 45

Without perturbation



#### Perturbation Coefficient = 0.004

![](https://i.imgur.com/1se2VX1.png)
![](https://i.imgur.com/nnwrks4.png)
![](https://i.imgur.com/Za3i4F6.png)
![](https://i.imgur.com/ly6QEoT.png)
![](https://i.imgur.com/C1razdp.png)
![](https://i.imgur.com/mbjQQlS.png)

**Curves**
![](https://i.imgur.com/Kxynqiw.png)
![](https://i.imgur.com/h9gKP1c.png)
![](https://i.imgur.com/CAeMjjV.png)

#### Perturbation Coefficient = 0.0045
![](https://i.imgur.com/ACPxRvC.png)
![](https://i.imgur.com/FolAQTC.png)
![](https://i.imgur.com/IGO4i60.png)
![](https://i.imgur.com/hiTyeGP.png)
![](https://i.imgur.com/oLXF7ca.png)
![](https://i.imgur.com/AH71PKp.png)
![](https://i.imgur.com/5kZX69o.png)

**Curves**

![](https://i.imgur.com/L3nDafp.png)
![](https://i.imgur.com/ChQNxpB.png)
![](https://i.imgur.com/vtl2yLm.png)

#### Perturbation Coefficient = 0.005

![](https://i.imgur.com/g8hLSTx.png)
![](https://i.imgur.com/tgs2I4B.png)
![](https://i.imgur.com/SxiAhBd.png)
![](https://i.imgur.com/43kF2Oo.png)
![](https://i.imgur.com/XBNbUQp.png)
![](https://i.imgur.com/GKELNXp.png)
![](https://i.imgur.com/woAXFVN.png)

**Curves**

![](https://i.imgur.com/QD8k4Xu.png)
![](https://i.imgur.com/MflOV5f.png)
![](https://i.imgur.com/ibLCXRM.png)

#### Perturbation Coefficient = 0.0055

![](https://i.imgur.com/JIuYUNL.png)
![](https://i.imgur.com/sQBMKXo.png)
![](https://i.imgur.com/z72uVfg.png)
![](https://i.imgur.com/8I9qnsb.png)
![](https://i.imgur.com/caYRr8o.png)
![](https://i.imgur.com/sMK8ehy.png)
![](https://i.imgur.com/neK6T7Y.png)
![](https://i.imgur.com/ngGbpT2.png)
![](https://i.imgur.com/PahoXaS.png)

**Curves**

![](https://i.imgur.com/uk6iEyc.png)
![](https://i.imgur.com/FmZCpkJ.png)
![](https://i.imgur.com/aiJtJQa.png)

#### Perturbation Coefficient = 0.006

![](https://i.imgur.com/1UHYvCt.png)
![](https://i.imgur.com/GrtcWyc.png)
![](https://i.imgur.com/x1BLWGP.png)
![](https://i.imgur.com/ulqjROH.png)
![](https://i.imgur.com/zFHHMXn.png)
![](https://i.imgur.com/qzASjfC.png)
![](https://i.imgur.com/K9wcllE.png)
![](https://i.imgur.com/n0vrXX3.png)
![](https://i.imgur.com/l0OdfjF.png)
![](https://i.imgur.com/QEtGOPP.png)

**Curves**

![](https://i.imgur.com/bFpfXeO.png)
![](https://i.imgur.com/22ETScz.png)
![](https://i.imgur.com/6UAYqq6.png)

### Experiment 5: Exponential Decay Method With t = 40

- T: 65536.0 * 40

Without perturbation



#### Perturbation Coefficient = 0.004

![](https://i.imgur.com/Vg7TKiP.png)
![](https://i.imgur.com/H0dGa2c.png)
![](https://i.imgur.com/grBphgo.png)
![](https://i.imgur.com/6gE1WXU.png)
![](https://i.imgur.com/Q0nPWSe.png)
![](https://i.imgur.com/YapUMle.png)
![](https://i.imgur.com/NkCUJqh.png)
![](https://i.imgur.com/IDOJQ5p.png)
![](https://i.imgur.com/VkzV6T4.png)
![](https://i.imgur.com/192tUeA.png)
![](https://i.imgur.com/go1D07W.png)

**Curves**

![](https://i.imgur.com/oZSQRHQ.png)
![](https://i.imgur.com/rtwr1WE.png)
![](https://i.imgur.com/oaNXyDW.png)

#### Perturbation Coefficient = 0.0045

![](https://i.imgur.com/s7bhacH.png)
![](https://i.imgur.com/3EWlkDb.png)
![](https://i.imgur.com/xdNKZyB.png)
![](https://i.imgur.com/4mVZZBw.png)
![](https://i.imgur.com/keWZ80J.png)
![](https://i.imgur.com/rmSL0zZ.png)
![](https://i.imgur.com/840apFu.png)
![](https://i.imgur.com/np343Jr.png)
![](https://i.imgur.com/o5tGfoM.png)

**Curves**

![](https://i.imgur.com/9mua6Ld.png)
![](https://i.imgur.com/elfbdSU.png)
![](https://i.imgur.com/O9l8WEd.png)

#### Perturbation Coefficient = 0.005

![](https://i.imgur.com/7sOG6yP.png)
![](https://i.imgur.com/bUdQzNF.png)
![](https://i.imgur.com/OAdFSxP.png)
![](https://i.imgur.com/7mc0njF.png)
![](https://i.imgur.com/0nMzagH.png)
![](https://i.imgur.com/EYL2V4j.png)
![](https://i.imgur.com/R9vCYko.png)
![](https://i.imgur.com/Tpz1ZG9.png)
![](https://i.imgur.com/l0piWGt.png)

**Curves**

![](https://i.imgur.com/EypNEnI.png)
![](https://i.imgur.com/7dkrsmV.png)
![](https://i.imgur.com/RG6vvYF.png)

#### Perturbation Coefficient = 0.0055

![](https://i.imgur.com/rVrdJyz.png)
![](https://i.imgur.com/kCabTCp.png)
![](https://i.imgur.com/Oi3tAEH.png)
![](https://i.imgur.com/cozLiQT.png)
![](https://i.imgur.com/SQKc89A.png)
![](https://i.imgur.com/GpFxHEF.png)
![](https://i.imgur.com/KrGqEwR.png)
![](https://i.imgur.com/oimkKWQ.png)
![](https://i.imgur.com/XTLbzZy.png)

**Curves**

![](https://i.imgur.com/87n7PWI.png)
![](https://i.imgur.com/MS1Rn3v.png)
![](https://i.imgur.com/uA2gLlf.png)

#### Perturbation Coefficient = 0.006

![](https://i.imgur.com/UgIvrnJ.png)
![](https://i.imgur.com/abBY8qb.png)
![](https://i.imgur.com/ubtqJrm.png)
![](https://i.imgur.com/X6zRjNP.png)
![](https://i.imgur.com/yHWBywf.png)
![](https://i.imgur.com/koQwOwG.png)
![](https://i.imgur.com/PjLLYa8.png)
![](https://i.imgur.com/NvCIg3c.png)
![](https://i.imgur.com/IC55OyH.png)
![](https://i.imgur.com/IsiJO8N.png)

**Curves**

![](https://i.imgur.com/hHZFJZi.png)
![](https://i.imgur.com/s05spXo.png)
![](https://i.imgur.com/o5XS6mA.png)

### Experiment 6: Exponential Decay Method With t = 35

- T: 65536.0 * 35

Without perturbation



#### Perturbation Coefficient = 0.004

![](https://i.imgur.com/F7U9bgF.png)
![](https://i.imgur.com/tQqP3hN.png)
![](https://i.imgur.com/9SSIxRb.png)
![](https://i.imgur.com/TjIXB1h.png)
![](https://i.imgur.com/WctI2Mm.png)
![](https://i.imgur.com/z1tSIli.png)
![](https://i.imgur.com/2ybfMo9.png)
![](https://i.imgur.com/05CWh15.png)
![](https://i.imgur.com/xYyrSnj.png)
![](https://i.imgur.com/UxvLKs4.png)
![](https://i.imgur.com/YglRz3i.png)

**Curves**

![](https://i.imgur.com/z78zyrf.png)
![](https://i.imgur.com/dLfPxTg.png)
![](https://i.imgur.com/AuUcYGw.png)

#### Perturbation Coefficient = 0.0045

![](https://i.imgur.com/xBSWJMw.png)
![](https://i.imgur.com/2tP5LOW.png)
![](https://i.imgur.com/Q1WQDYV.png)
![](https://i.imgur.com/zxqM2WW.png)
![](https://i.imgur.com/dREdZVc.png)
![](https://i.imgur.com/RHFiBKO.png)
![](https://i.imgur.com/xSeMVpc.png)
![](https://i.imgur.com/0DA6MlB.png)
![](https://i.imgur.com/RCoJxdF.png)
![](https://i.imgur.com/FgVCIYK.png)

**Curves**

![](https://i.imgur.com/dS4Yfl4.png)
![](https://i.imgur.com/PH8BST1.png)
![](https://i.imgur.com/7BI6h2g.png)

#### Perturbation Coefficient = 0.005

![](https://i.imgur.com/Oc1j9y3.png)
![](https://i.imgur.com/Uh9ShLu.png)
![](https://i.imgur.com/DA7SleH.png)
![](https://i.imgur.com/sKt0EO2.png)
![](https://i.imgur.com/K8XsL2c.png)
![](https://i.imgur.com/kPURoxg.png)
![](https://i.imgur.com/N93cbCR.png)
![](https://i.imgur.com/f6k4amj.png)
![](https://i.imgur.com/rxpxiGM.png)
![](https://i.imgur.com/ecxYbOf.png)
![](https://i.imgur.com/N5ojHQ9.png)

**Curves**

![](https://i.imgur.com/KTU38wo.png)
![](https://i.imgur.com/rUr2tDR.png)
![](https://i.imgur.com/YV9SoKn.png)

#### Perturbation Coefficient = 0.0055

![](https://i.imgur.com/KkD5SD1.png)
![](https://i.imgur.com/vOKhokn.png)
![](https://i.imgur.com/QrLTjRr.png)
![](https://i.imgur.com/u9N6VDL.png)
![](https://i.imgur.com/bmPC9SZ.png)
![](https://i.imgur.com/tdnFrl3.png)
![](https://i.imgur.com/Q5MDZZx.png)
![](https://i.imgur.com/XVLiRj1.png)
![](https://i.imgur.com/tE3yRFS.png)
![](https://i.imgur.com/N4xo5IX.png)
![](https://i.imgur.com/ujqY6cm.png)

**Curves**

![](https://i.imgur.com/ygTXuDH.png)
![](https://i.imgur.com/Xe9ia47.png)
![](https://i.imgur.com/Sw8TgPl.png)

#### Perturbation Coefficient = 0.006

![](https://i.imgur.com/5h4Wx3Y.png)
![](https://i.imgur.com/p2JzE9t.png)
![](https://i.imgur.com/OeB5CYc.png)
![](https://i.imgur.com/XfY1INe.png)
![](https://i.imgur.com/Hdas3D9.png)

**Curves**

![](https://i.imgur.com/yntQUnt.png)
![](https://i.imgur.com/BEs4THL.png)
![](https://i.imgur.com/BqkjqZ2.png)


### Experiment 7: Exponential Decay Method With t = 30

- T: 65536.0 * 30

Without perturbation



#### Perturbation Coefficient = 0.004

![](https://i.imgur.com/CFi9LIQ.png)
![](https://i.imgur.com/GOR5USa.png)
![](https://i.imgur.com/eReW6nc.png)
![](https://i.imgur.com/lNedr1Y.png)
![](https://i.imgur.com/Nn3xd4k.png)
![](https://i.imgur.com/lezCn4B.png)

**Curves**

![](https://i.imgur.com/tQjqQp3.png)
![](https://i.imgur.com/KBxlMyK.png)
![](https://i.imgur.com/Q8vMRnJ.png)

#### Perturbation Coefficient = 0.0045

![](https://i.imgur.com/SgeaENS.png)
![](https://i.imgur.com/sMsb46d.png)
![](https://i.imgur.com/F4857sD.png)
![](https://i.imgur.com/vDGrsCn.png)
![](https://i.imgur.com/uJPunkQ.png)
![](https://i.imgur.com/xSGNpjX.png)
![](https://i.imgur.com/XiBhseN.png)

**Curves**



#### Perturbation Coefficient = 0.005

**Curves**


#### Perturbation Coefficient = 0.0055

**Curves**

#### Perturbation Coefficient = 0.006

**Curves**


# ML 2021/09/17

## Conclusion

1. With fixed batch size, the larger dataset size, the the blurrier image.
2. The effective perturbation coefficient for exponential perturbation is about < 0.007 and > 0.004. When the coefficient is too large, the image would contain some noise and need more epoch to converge. The image would become pure noise after the coefficient > 0.007. 
3. It seems that the perturbation is good at making the fake image a little bit different from the ground truth image, but just a little, for example, changing skin color seems easily to success like [this](#perturbation-coefficient-=-0.004). According to the results, We can speculate that **the acceptance region of the discreminator(means the region taht the discriminator judge as true) is very small and close to the ground truth**. On the other hand, when the perturbation is too large, that is, exceed the acceptance region of the discreminator, the image would become blurry or vague. Guess it is due to the discriminator might not fit well on the manifold out of the acceptance region of the discreminator.
4. Note that, I've tried larger training epoch 30000 and different batch size/dataset size/t combinations, but all of them don't help. The images are still blurry, so we can exclude the underfit/improper dataset size/improper t as the possible resaons.
5. When it comes to larger dataset size, the pertubation tolerance would be lower. That is, it can only accept smaller perturbation.


## Next Goal

Next time, I will try larger batch size and dataset size, it may extend the acceptance region of the the discreminator. By extending the acceptance region, it may help to generate more creative image.

## Results

The way we compute the norm of the `gradient` is the same as the `perturbation`. Both of them are the average of the Euclidean norm.

### Experiment 1: Exponential Decay Method With 256 Batch Size & 256 Dataset Size

- Batch Size: 256
- Dataset Size: 256
- Noise Size: 10
- Epoch: 10000
- T: 65536.0 * 6
- Exponential Decay

Without perturbation

![](https://i.imgur.com/3PCHL1B.png)

#### Perturbation Coefficient = 0.004

No effect

![](https://i.imgur.com/JlwyU5B.png)

**Curves**

![](https://i.imgur.com/iS19Rvd.png)
![](https://i.imgur.com/Jw7xO5D.png)
![](https://i.imgur.com/1xCZJwz.png)

#### Perturbation Coefficient = 0.005

Pretty great creativity

![](https://i.imgur.com/WgKctBk.png)
![](https://i.imgur.com/WDWE1Eu.png)

**Curves**

![](https://i.imgur.com/poTaomf.png)
![](https://i.imgur.com/dWgCad5.png)
![](https://i.imgur.com/KZfiR3g.png)

#### Perturbation Coefficient = 0.0055

![](https://i.imgur.com/UrfK6AO.png)
![](https://i.imgur.com/1y4htYh.png)

**Curves**

![](https://i.imgur.com/r0D7glI.png)
![](https://i.imgur.com/GKO29MU.png)
![](https://i.imgur.com/sSHFEhX.png)

#### Perturbation Coefficient = 0.006

![](https://i.imgur.com/ervBVe0.png)
![](https://i.imgur.com/kqQpjb7.png)
![](https://i.imgur.com/JibxtZT.png)

**Curves**

![](https://i.imgur.com/kvb4GZK.png)
![](https://i.imgur.com/Cinc2Xb.png)
![](https://i.imgur.com/w4gTyfi.png)

#### Perturbation Coefficient = 0.0065

![](https://i.imgur.com/f6XedAR.png)
![](https://i.imgur.com/YPuu5Eb.png)
![](https://i.imgur.com/JzCOk8e.png)
![](https://i.imgur.com/Thjli5x.png)
![](https://i.imgur.com/xj0XcsP.png)
![](https://i.imgur.com/crr31tr.png)

**Curves**

![](https://i.imgur.com/nOVdEgt.png)
![](https://i.imgur.com/KpOej4r.png)
![](https://i.imgur.com/Saumupk.png)

#### Perturbation Coefficient = 0.007

![](https://i.imgur.com/Jc988iz.png)

**Curves**

![](https://i.imgur.com/Sf5wjpo.png)
![](https://i.imgur.com/oRsHn5S.png)
![](https://i.imgur.com/IFYgr4h.png)

#### Perturbation Coefficient = 0.008

Totally blury

![](https://i.imgur.com/YJkPSjM.png)

**Curves**

![](https://i.imgur.com/LSnN7Xe.png)
![](https://i.imgur.com/lFDqeFN.png)
![](https://i.imgur.com/a0WoAbl.png)

### Experiment 2: Exponential Decay Method With 256 Batch Size & 300 Dataset Size

- Batch Size: 256
- Dataset Size: 300
- Noise Size: 10
- Epoch: 10000
- T: 65536.0 * 6
- Exponential Decay

#### Perturbation Coefficient = 0.004

![](https://i.imgur.com/qYWP9DJ.png)
![](https://i.imgur.com/iHPKRKo.png)
![](https://i.imgur.com/Q6PptCX.png)

**Curves**

![](https://i.imgur.com/wNRF7f6.png)
![](https://i.imgur.com/nC9PvWN.png)
![](https://i.imgur.com/MhSogF9.png)

#### Perturbation Coefficient = 0.005

![](https://i.imgur.com/ACFHlC1.png)

**Curves**

![](https://i.imgur.com/9pE0KJk.png)
![](https://i.imgur.com/dTI44UC.png)
![](https://i.imgur.com/Lm3tywD.png)

#### Perturbation Coefficient = 0.0055

![](https://i.imgur.com/WB86ZDe.png)

**Curves**

![](https://i.imgur.com/GljOINP.png)
![](https://i.imgur.com/rYUxQmF.png)
![](https://i.imgur.com/jEzXMAR.png)

#### Perturbation Coefficient = 0.006

![](https://i.imgur.com/h3ppdEK.png)
![](https://i.imgur.com/AcrCkhk.png)
![](https://i.imgur.com/Ey0ghJt.png)
![](https://i.imgur.com/QM70Ou0.png)
![](https://i.imgur.com/d89gILf.png)

**Curves**

![](https://i.imgur.com/T6miAxK.png)
![](https://i.imgur.com/kCzts4R.png)
![](https://i.imgur.com/3BAm6Uv.png)

#### Perturbation Coefficient = 0.0065

![](https://i.imgur.com/0v8cAQM.png)

**Curves**

![](https://i.imgur.com/L8oQOw4.png)
![](https://i.imgur.com/PE91TqK.png)
![](https://i.imgur.com/6JV2xH0.png)

### Experiment 4: Exponential Decay Method With 256 Batch Size & 400 Dataset Size

- Batch Size: 256
- Dataset Size: 400
- Noise Size: 10
- Epoch: 10000
- T: 65536.0 * 6
- Exponential Decay

#### Perturbation Coefficient = 0.004

![](https://i.imgur.com/17Opb2t.png)
![](https://i.imgur.com/F7zWnYQ.png)
![](https://i.imgur.com/HhGQ3Nt.png)
![](https://i.imgur.com/grI2l55.png)
![](https://i.imgur.com/pHcVNg9.png)
![](https://i.imgur.com/fkeHiFN.png)

**Curves**

![](https://i.imgur.com/pf47dzL.png)
![](https://i.imgur.com/ggO4opQ.png)
![](https://i.imgur.com/UgW6ANQ.png)

#### Perturbation Coefficient = 0.005

![](https://i.imgur.com/FgR52Bh.png)

**Curves**

![](https://i.imgur.com/vl8Xpso.png)
![](https://i.imgur.com/pffg5j6.png)
![](https://i.imgur.com/hCv05xe.png)

#### Perturbation Coefficient = 0.0055

![](https://i.imgur.com/k7wBiXx.png)
![](https://i.imgur.com/y2uplrn.png)
![](https://i.imgur.com/MJcJFSP.png)
![](https://i.imgur.com/gl7exKo.png)
![](https://i.imgur.com/SP508vZ.png)

**Curves**

![](https://i.imgur.com/iBALA3z.png)
![](https://i.imgur.com/j02Rhh2.png)
![](https://i.imgur.com/J8ioGOL.png)

#### Perturbation Coefficient = 0.006

![](https://i.imgur.com/vGPuQ0Z.png)
![](https://i.imgur.com/5AXtR4U.png)
![](https://i.imgur.com/nUtYY2T.png)

**Curves**

![](https://i.imgur.com/h0koFVG.png)
![](https://i.imgur.com/Pd5SvDE.png)
![](https://i.imgur.com/taCc7wy.png)

#### Perturbation Coefficient = 0.0065

![](https://i.imgur.com/HAfXzTS.png)

**Curves**

![](https://i.imgur.com/iiF07hS.png)
![](https://i.imgur.com/o2VYvaw.png)
![](https://i.imgur.com/WIIxQPw.png)

### Experiment 5: Exponential Decay Method With 256 Batch Size & 512 Dataset Size

- Batch Size: 256
- Dataset Size: 512
- Noise Size: 10
- Epoch: 10000
- T: 65536.0 * 6
- Exponential Decay

Without perturbation

![](https://i.imgur.com/uDsHgF9.png)


#### Perturbation Coefficient = 0.003

![](https://i.imgur.com/FD079ow.png)
![](https://i.imgur.com/KYRSrHL.png)
![](https://i.imgur.com/HxzCpGk.png)
![](https://i.imgur.com/XKD9Dhw.png)
![](https://i.imgur.com/LyLoRG6.png)
![](https://i.imgur.com/vnq7bDp.png)
![](https://i.imgur.com/BdnqK2u.png)
![](https://i.imgur.com/sKA3FCC.png)
![](https://i.imgur.com/ixWwyDd.png)

**Curves**

![](https://i.imgur.com/Om1r7mw.png)
![](https://i.imgur.com/K7k75Oa.png)
![](https://i.imgur.com/defoGty.png)

#### Perturbation Coefficient = 0.004

![](https://i.imgur.com/wOT7FIg.png)
![](https://i.imgur.com/6J3i7D6.png)
![](https://i.imgur.com/gid0gIv.png)
![](https://i.imgur.com/RgaxSNf.png)
![](https://i.imgur.com/a0HdPN0.png)
![](https://i.imgur.com/HCKvrhr.png)
![](https://i.imgur.com/W0q80Ep.png)
![](https://i.imgur.com/Eb1l5aj.png)
![](https://i.imgur.com/Y4rsDM3.png)
![](https://i.imgur.com/tU25ChF.png)
![](https://i.imgur.com/lC8r61d.png)

**Curves**

![](https://i.imgur.com/sq2Byw4.png)
![](https://i.imgur.com/vQV1MwS.png)
![](https://i.imgur.com/0tP4g6l.png)

#### Perturbation Coefficient = 0.005

![](https://i.imgur.com/9wZqWyl.png)
![](https://i.imgur.com/IOwZ99n.png)
![](https://i.imgur.com/Ce0rQ5s.png)
![](https://i.imgur.com/q3tL7ne.png)
![](https://i.imgur.com/JRICAnJ.png)

**Curves**

![](https://i.imgur.com/5cvAoVT.png)
![](https://i.imgur.com/GGHLaWF.png)
![](https://i.imgur.com/biX86Nu.png)

#### Perturbation Coefficient = 0.0055

![](https://i.imgur.com/NxQugkx.png)
![](https://i.imgur.com/3wCdINV.png)
![](https://i.imgur.com/v0icgRh.png)
![](https://i.imgur.com/5ecqsi3.png)
![](https://i.imgur.com/e9jQRpf.png)
![](https://i.imgur.com/zVm3Tnf.png)

**Curves**

![](https://i.imgur.com/lOlUDgR.png)
![](https://i.imgur.com/QHdw0HV.png)
![](https://i.imgur.com/KYyVFeR.png)

#### Perturbation Coefficient = 0.006

![](https://i.imgur.com/ZNQ7XfH.png)

**Curves**

![](https://i.imgur.com/3DVotxJ.png)
![](https://i.imgur.com/FmDVv2S.png)
![](https://i.imgur.com/sThGykS.png)

### Experiment 6: Fixed Exponential Decay Method With Different 'training_t_rate'

- Batch Size: 256
- Dataset Size: 256
- Noise Size: 10
- Epoch: 10000
- T: training_t_unit * training_t_rate = 65536.0 * [6, 5, 4, 3, 2]
- Exponential Decay, Perturbation Coefficient = 0.005

#### training_t_rate = 6.0

Without perturbation

![](https://i.imgur.com/xbSHaAR.png)

With perturbation

![](https://i.imgur.com/zp6iMmD.png)
![](https://i.imgur.com/oGW8t1E.png)

**Curves**

![](https://i.imgur.com/Au6NUYw.png)
![](https://i.imgur.com/sxz9wV6.png)
![](https://i.imgur.com/6PXUkZA.png)

#### training_t_rate = 5.0

Without perturbation

![](https://i.imgur.com/uvH1K0H.png)

With perturbation

![](https://i.imgur.com/A5KTokP.png)
![](https://i.imgur.com/PzTkwuy.png)
![](https://i.imgur.com/S4ChnXT.png)

**Curves**

![](https://i.imgur.com/LTHCofe.png)
![](https://i.imgur.com/qkjh7C8.png)
![](https://i.imgur.com/7TzQax1.png)

#### training_t_rate = 4.0

Without perturbation

![](https://i.imgur.com/UhfCFln.png)

With perturbation

![](https://i.imgur.com/UuzulEs.png)
![](https://i.imgur.com/y6zYeuG.png)
![](https://i.imgur.com/Xy2E2TZ.png)
![](https://i.imgur.com/fdsP4AX.png)
![](https://i.imgur.com/jjpkB41.png)
![](https://i.imgur.com/leCpaZu.png)

**Curves**

![](https://i.imgur.com/3X8beUX.png)
![](https://i.imgur.com/eZF2Iid.png)
![](https://i.imgur.com/kx9tLnC.png)

#### training_t_rate = 3.0

Without perturbation

![](https://i.imgur.com/3xQ7oXR.png)

With perturbation

![](https://i.imgur.com/1waifd5.png)
![](https://i.imgur.com/bXrbGMi.png)
![](https://i.imgur.com/wq0cTDB.png)

**Curves**

![](https://i.imgur.com/LqMzpTT.png)
![](https://i.imgur.com/jH14Tuy.png)
![](https://i.imgur.com/zZb9xRW.png)

#### training_t_rate = 2.0

Without perturbation



With perturbation

![](https://i.imgur.com/CdieCDL.png)

**Curves**

![](https://i.imgur.com/lg56QZF.png)
![](https://i.imgur.com/GljQIi8.png)
![](https://i.imgur.com/6dyRM2a.png)

# ML 2021/09/02

## Goal

Add perturbation on the gradient to make the generator more creative.

## Method

Denote $g$ as the gradient that update on image, and $\eta$ as the learning rate. 

$$
grad \_ accumulator = gradient + \eta \cdot perturbation
$$

With Yurong's momentum update

$$
gradint = \eta \cdot grads + momentum \cdot  grad \_ accumulator
$$

Update the image

$$
x \_noise = x \_noise - grad \_ accumulator
$$

Denote the **perutrbation coefficient** as $\gamma$ in the follow formulas. The `gradient` is the momentum gradient, so the implementation of the perturbation is

```python=1
# add to momentum
perturb = noise_generator.generate(shape=x_noise.shape)
grad_accumulator = grads * learning_rate + momentum * grad_accumulator + learning_rate * perturb
```

### Method 1: Static Noise

$$
perturbation = \mathcal{N}(0, 1) \cdot \gamma
$$

### Method 2: Exponential Decay

Decay along the epochs.

$$
perturbation = \mathcal{N}(0, 1) \cdot \gamma^{epoch}
$$

### Method 3: Anneal Decay

$$
perturbation = \frac{\mathcal{N}(0, 1)}{(1 + epoch)^{\gamma}}
$$

Refers to [Adding Gradient Noise Improves Learning for Very Deep Networks](https://arxiv.org/abs/1511.06807)

## Perturbation Norm & Gradient Norm



## Results

The way we compute the norm of the `gradient` is the same as the `perturbation`. Both of them are the average of the Euclidean norm.

Common Settings

- Batch Size: 256
- Dataset Size: 256
- Noise Size: 10
- Epoch: 10000
- T: 65536.0 * 6

### Exponential Decay

#### Perturbation Coefficient = 0

Same as no-perturbation

![](https://i.imgur.com/Qt0Qo4z.png)
![](https://i.imgur.com/Lg8HCnH.png)
![](https://i.imgur.com/plu96cr.png)

#### Perturbation Coefficient = 0.009

The image become pure noise.

![](https://i.imgur.com/0iEt4y6.png)
![](https://i.imgur.com/H45FqSU.png)
![](https://i.imgur.com/EbNcu9n.png)
![](https://i.imgur.com/EQJKQtK.png)


#### Perturbation Coefficient = 0.09

The image become pure noise.

![](https://i.imgur.com/P9BEoKk.png)
![](https://i.imgur.com/IikA8WA.png)
![](https://i.imgur.com/DUC7eY1.png)
![](https://i.imgur.com/V9ZpejL.png)

#### Perturbation Coefficient = 0.9, 0.99, 0.999

The image become pure noise.

### Static

#### Perturbation Coefficient = 1e-7

Some images get blurry.

![](https://i.imgur.com/WKh7QtR.png)
![](https://i.imgur.com/b1S6tgF.png)
![](https://i.imgur.com/3ygJfIU.png)

#### Perturbation Coefficient = 1e-6, 1e-5, 1e-3, 

The image become pure noise.




