GAN project weekly report

--- title: GAN project weekly report tags: GAN, project --- # Weekly report [TOC] [GAN](https://hackmd.io/iHQIBWpFSS2GigHoDufO3Q?both#Deep-learning-Flow) --- ## 待討論 1. 我們也要每週禮拜二中午12. 前寫完ㄇ？？ >sure 2. 要訂看完論文的期限ㄇ？？ >yes, 還是每週訂一個進度嗎? --- ## July 22 - 28 ### Joann - PCA - method - Model Collapse >[read more...](https://hackmd.io/iHQIBWpFSS2GigHoDufO3Q?both#Model-Collapse) - dimensionality reduction - PCA - t-SNE - Obtain Optimal $z^*$ >$z^∗$- search & BEGAN-CS - Disentangled Reprepesentation Learning >由某一個較低維度的latent representation z所產生，而資料也可以藉由控制這個z所生成 - VAE - InfoGAN - What is CelebA dataset? > a large-scale face attributes dataset - Why unsupervised learning? >allow the model to work on its own to discover information and perform more complex processing tasks compared to supervised learning - VAE(Variational Autoencoder) - InfoGAN - Distributed vs Disentangled Representation - Auto-encoder >輸入一筆input data經過類神經網路後，也要得到跟 input data一模一樣的data (可以拆成Encoder 和 Decoder 兩個neural network) - What is a Per-Pixel Loss Function? >Used as a metric for understanding differences between images on a pixel level. - What is Disentangled Representation Learning? >Breaks down, or disentangles, each feature into narrowly defined variables and encodes them as separate dimensions. - What is latent space? >abstract multi-dimensional space - What is Back-Propagation(BP)? > a widely used algorithm in training feedforward neural networks for supervised learning. - [完整筆記在這ㄦ~全部打進來太多惹QQ](https://hackmd.io/MiYVyg6uQvu_4n9MWzt7Iw?both) Q: 除了寫進度之外要不要確切讀到了些什摩也大概寫一寫ㄚ？？這樣如果有一樣的問題就不用重複再查還可以討論 >補上了~但太多還是先放個大概，完整版附上了 :crying_cat_face: --- ### Joycelyn's - Thesis: - **latent vector**: meaningful feature - use Principle Component Analysis(**PCA**): essential technique in order to find the latent vector. - **disentangled representations**: 把整張高維圖片拆解降維成 meaningful characteristics, like a nose or a 肉圓. - Suggested loss function for - discriminator: ReLU function - generator: sigmoid function - Linear Algebra: - [**Singular Value Decomposition**](https://www.youtube.com/watch?v=TX_vooSnhm8&list=PLE7DDD91010BC51F8&index=31) （看一半就快完了啦:eyes: - Topic brainstorming - GAN + facial recognition + NLP -> personal 小護士 :female-doctor: - To be continued: - [A Gentle Introduction to Generative Adversarial Networks (GANs)](https://machinelearningmastery.com/what-are-generative-adversarial-networks-gans/) - [thispersondoesnotexist.com](https://www.thispersondoesnotexist.com) --- ### Yang - CelebA(CelebFaces Attribute 名人人臉屬性數據集) - BEGAN - 新的簡單的GAN結構，使用標準的訓練方式不加訓練trick也能很快且穩定的收斂 - 對於G及D的能力提出均衡的概念 - 提供一個超參數，可在圖像的多樣性和生成質量做均衡 - 提出一種收斂程度的估計，這個機制只在WGAN出現過(靈感來自WGAN) - loss function - Equilibrium - WGAN - 提出Wasserstein距離代替JS Divergence - DCGAN - 網路結構 - Deconvolution - 訓練方法 - GAN改進 - PCA - [完整筆記](https://hackmd.io/ej9lseR4QzaROxy6WPv0eg?both) --- ## July 28 - Aug 4 ### Joycelyn's - Basics: - [GAN implementation](https://www.youtube.com/watch?v=DQNNMiAP5lw) - [Machine Learning](https://www.youtube.com/watch?v=CXgbekl66jc&list=PLJV_el3uVTsPy9oCRY30oBPNLCo89yu49) - 1, 2, 3, 4 - **What is Machine Learning?** - Looking for a function ==from data== - **Machine Learning Framework (流程)** ![](https://i.imgur.com/bDbf2hl.jpg) > **This is supervised learning.** > Step1: Give a set of function (ex. f(cat image) = "cat") > Step2: 讓 machine 可以去衡量這個 function 是好還是不好 > Step3: 讓 machine 自己去找到最好的 function - **Machine Learning Task** - Regression: The output of the target function f is "scalar" - Thesis: chapter 3 ~ 5 - So like if we actually do a research using GAN we would have to invent a new GAN-branch like dear嘉哲？？ --- ### Joann #### thesis : - [x] chapter 4~5 - experiment - k value suddenly drop, BEGAN collapse shortly - $\alpha$ : controls the important of the constraint loss (set to 1 as the default value) - k value : how much attention is paid on $\mathcal{L}_G$ - M value : value of the convergence measure - when use M value, the figure seems not observation - constraint loss - effective - preventing from mode collapse - very good performance in generating high-quality images - mode collapsing - the precise timing is unpredicted - highly related to : 1. the resolution of generated images 2. th size of training dataset - suppressed after adding the constraint loss - small sized dataset - BEGAN-CS can generated high-quality result than original - BEGAN collapse earlier than original - keep BEGAN from converging to an optimal state - BEGAN can **learn salient** and **high-quality disentangled representation** in an unsupervised setting - advantages of BEGAN-CS - approximate $z^*$ on-the-fly - $G(z)$ looks similar to $x$ - exhibit the adjustable disentangled properties #### basis - [Machine learning](https://www.youtube.com/watch?v=CXgbekl66jc&list=PLJV_el3uVTsPy9oCRY30oBPNLCo89yu49) : 5/35 episodes - [**ML note**](https://hackmd.io/3JJW0_70SL-fhwavFcmRhg) - [deep learning theory](https://www.youtube.com/watch?v=KKT2VkTdFyc&list=PLJV_el3uVTsOh1F5eo9txATa4iww0Kp8K) : 0/10 episodes - [GAN introduction](https://www.youtube.com/watch?v=DQNNMiAP5lw&list=PLJV_el3uVTsMq6JEFPW35BCiOQTsoqwNw) : 0/10 episodes - supervised(predictive) v.s. unsupervised(descriptive) - supervised : to learn a mapping from inputs x to outputs y, given a labeled set of input-output pairs - unsupervised : to find “interesting patterns” in the data - much less well-defined problem - no obvious error metric to use #### news [Recognising fake images using frequency analysis](https://news.rub.de/english/press-releases/2020-07-16-information-technology-recognising-fake-images-using-frequency-analysis) #### terms - measure of convergence - [Convergence of measures](https://encyclopediaofmath.org/wiki/Convergence_of_measures) - not to be confused with **convergence in measure** - spectral normalization - [SN-GAN簡介](https://medium.com/@xiaosean5408/sn-gan%E7%B0%A1%E4%BB%8B-spectral-normalization-for-generative-adversarial-networks-f8fd784f2ad) - 新的 weight normalization 的方式防止GAN在training時不穩 - 透過 Lipschitz constant 對每層的輸出做限縮 - 控制 Lipschitz constant，不需複雜的參數調整 - 實作簡單並且計算需求少 - 數學式太多好難理解...(會用到SVD) --- ### yang - thesis : [learning note](https://hackmd.io/Q5SfSlibTnmuK57jMGRs3w) --- ## Aug 4 - 11 ### Joycelyn's - Basic: - Machine Learning lecture: 5 6 7 8 9 --- ### Joann - basic - [Machine learning](https://www.youtube.com/watch?v=CXgbekl66jc&list=PLJV_el3uVTsPy9oCRY30oBPNLCo89yu49) : 6-12/35 episodes - [Machine Learning note](https://hackmd.io/3JJW0_70SL-fhwavFcmRhg) - [deep learning note](https://hackmd.io/xv5OKCzJSjChHfhpqemORg) --- ### yang --- ## Aug 11 - 18 ### Joycelyn's - basic - [Machine learning lecture](https://www.youtube.com/watch?v=CXgbekl66jc&list=PLJV_el3uVTsPy9oCRY30oBPNLCo89yu49): 10 11 12 13 14 15 --- ### Joann - basic - machine learning lecture : 13-20, 21(14'47) - [deep learning note](https://hackmd.io/xv5OKCzJSjChHfhpqemORg) - semi-supervised的一半筆記轉到ML --- --- ## Aug 18 - 25 ### Joycelyn's - basic - [Machine learning lecture](https://www.youtube.com/watch?v=CXgbekl66jc&list=PLJV_el3uVTsPy9oCRY30oBPNLCo89yu49): 16 - [Deep Learning Homework](https://drive.google.com/drive/folders/1D0WtaDQWFKiXL9zXNEu3h1aa2V2Z5Dx5?usp=sharing) ### Joann - basic - machine learning lecture : 25 - [Deep learning lecture](https://www.youtube.com/watch?v=KKT2VkTdFyc&list=PLJV_el3uVTsOh1F5eo9txATa4iww0Kp8K) : 1-3 - [deep learning note](https://hackmd.io/xv5OKCzJSjChHfhpqemORg) - deep learning hw - hw1 practice(pytorch) - [pytorch 語法tutorial note](https://hackmd.io/pY7ABMeuTfqY4cRGdLGfuQ) --- --- ## Aug 25 - Sep 1 ### Joycelyn's - Topic brainstorming - [Rewriting the rules of machine-generated art](https://news.mit.edu/2020/rewriting-rules-machine-generated-art-0818) - An artificial intelligence tool lets users edit generative adversarial network models with simple copy-and-paste commands. - [GAN Paint](http://gandissect.res.ibm.com/ganpaint.html?project=churchoutdoor&layer=layer4), a tool that lets users add and remove features like doors and clouds from a picture. - OMG this is f**king cool !! - "If we can rewrite the rules of a GAN directly, the only limit is human imagination." - Big datasets are not essential, you can edit the numerical weights in its layers to get the behavior you desire, even if no literal example exists. - "It's like inserting the DNA of a firefly into a plant to make it glow in the dark." - “The neural network has different memory banks that function as a set of general rules, relating one set of learned patterns to another, if you could identify one line of memory, you could write a new memory into it.” --- ### Joann - basic - [deep learning lecture](https://www.youtube.com/watch?v=KKT2VkTdFyc&list=PLJV_el3uVTsOh1F5eo9txATa4iww0Kp8K) : 4 - 7 - [keras toutorial](https://hackmd.io/JWMXlWCFQQCspIxLlJSKjQ) - deep learning hw - hw1 pytorch practice - [pytorch語法tutorial note](https://hackmd.io/pY7ABMeuTfqY4cRGdLGfuQ) - problem : I cannot use the code `cuda`, e.g. ```python= args.cuda: device = torch.device('cuda') ``` because my GPU is not the product by NVIDIA - [上網找的回答(1) 但我看不懂@@](https://www.thinbug.com/q/58408143) - pytorch官網, 這個不知道是不是可切換成cpu?? - ![](https://i.imgur.com/MTdQ7Y2.png) --- --- ## Sep 1 - 8 --- ### Joann - basic: - [keras](https://keras.io/getting_started/intro_to_keras_for_engineers/) - [keras note](https://hackmd.io/JWMXlWCFQQCspIxLlJSKjQ) - machine learning lecture : 23, 30 - (word embedding, RNN part 1) ### Joycelyn's - [Deep Learning for NLP: ANNs, RNNs and LSTMs explained!](https://towardsdatascience.com/deep-learning-for-nlp-anns-rnns-and-lstms-explained-95866c1db2e4) --- ## Sep 8 - 15 ### Joann - basic : - deep learning lecture : 8 - machine learning lecture : 31 (26'19) - RNN ### Joycelyn's - Topic brainstorming - [AI researchers use heartbeat detection to identify deepfake videos](https://venturebeat.com/2020/09/03/ai-researchers-use-heartbeat-detection-to-identify-deepfake-videos/) - “We propose a deepfake source detector that **predicts the source generative model** for any given video. To our knowledge, our approach is the first to conduct a deeper analysis for source detection that interprets residuals of generative models for deepfake videos. Our key finding emerges from the fact that we can **interpret these biological signals as fake heartbeats** that contain a signature transformation of the residuals per model. Thus, it gives rise to a new exploration of these biological signals for not only determining the authenticity of a video, but also classifying its source model that generates the video.” - [Kaggle competition](https://www.kaggle.com/competitions) # Topic branstorming - [Rewriting the rules of machine-generated art](https://news.mit.edu/2020/rewriting-rules-machine-generated-art-0818) - An artificial intelligence tool lets users edit generative adversarial network models with simple copy-and-paste commands. - [GAN Paint](http://gandissect.res.ibm.com/ganpaint.html?project=churchoutdoor&layer=layer4), a tool that lets users add and remove features like doors and clouds from a picture. - "If we can rewrite the rules of a GAN directly, the only limit is human imagination." - Big datasets are not essential, you can edit the numerical weights in its layers to get the behavior you desire, even if no literal example exists. - "It's like inserting the DNA of a firefly into a plant to make it glow in the dark." - “The neural network has different memory banks that function as a set of general rules, relating one set of learned patterns to another, if you could identify one line of memory, you could write a new memory into it.” - [AI researchers use heartbeat detection to identify deepfake videos](https://venturebeat.com/2020/09/03/ai-researchers-use-heartbeat-detection-to-identify-deepfake-videos/) - Russian troll farm: 俄羅斯政治網軍 - Deepfakes can make epic memes or put Nicholas Cage in every movie, but they can also undermine elections. - Existing deepfake detection models focus on traditional media forensics methods, like tracking unnatural eyelid movements or distortions at the edge of the face. - “We propose a deepfake source detector that **predicts the source generative model** for any given video. To our knowledge, our approach is the first to conduct a deeper analysis for source detection that interprets residuals of generative models for deepfake videos. Our key finding emerges from the fact that we can **interpret these biological signals as fake heartbeats** that contain a signature transformation of the residuals per model. Thus, it gives rise to a new exploration of these biological signals for not only determining the authenticity of a video, but also classifying its source model that generates the video.” - The first study for detection of [unique GAN fingerprints](https://arxiv.org/abs/1811.08180) was introduced in 2018. - GANs have shown increasing success in generating photorealistic images （寫實派影像） - learning GAN fingerprints towards **image attribution** and using them to classify an image as real or GAN-generated - GAN fingerprints: - are stable - minor differences in GAN training can result in different results. - persist across different image frequencies and patches (like smaller blocks). - finetuning (調整) is effective in immunizing five different types of adversarial image perturbations (擾動). - [Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints](https://arxiv.org/pdf/1811.08180.pdf) - photoplethysmography PPG - [Kaggle competition](https://www.kaggle.com/competitions)