---
# System prepended metadata

title: 'Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving 論文共筆'

---

# Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving 論文共筆

---

[TOC]

---

## ABSTRACT
- 討論GAN在自動駕駛中的應用問題
    - 包括諸如高級數據增強
    - 損失函數學習
    - 半監督
## INTRODUCTION
- 自動駕駛模塊
    - Sense
    - Perceive & Localize
    - Abstract
    - Plan
    - Control
- 作法
    - 獨立設計模塊(傳統)
    - 端到端
- GAN在自動駕駛的優勢
    - Discriminative models & Generative models
    - Autonomous driving systems requires training the model with all possible scenarios which can happen in real life.

## OVERVIEW OF GAN
- 背景
    - GAN were introduced in 2014 and were immediately recognized as a perspective direction of upcoming deep learning research.
        - unsupervised learning
        - semi-supervised learning
        - advanced data augmentation.
- 問題
    - stabilization of the complicated GAN learning
        - 原因
            - the generated data do not reflect the diversity of the under-lying data distribution
        - 結果
            -  the discriminator is fooled to believe in unrealistic samples
            -   degeneration of the generator

### Vanilla GAN
- 方法
    - 一個生成器(network)，生成器的任務是生成樣本，這些樣本與真實數據樣本盡可能相似
    - 一個鑑別器(network)，鑑別器的任務是將真實樣本與生成的樣本區分開
    - 平衡點(結束),鑑別器應輸出等於0.5的概率
- 重點
    - G must not be trained too much without updating D
- 優點
    - only back-propagation is needed to compute the gradients
    - a wide variety of functions can be incorporated in the model
    - 
- 缺點
    - absence of explicit representation of $p_g(x)$
    - simultaneous optimization of the discriminator D with the generator G

### Prominent Derivatives of GAN
#### Conditional Generative Adversarial Nets (CGAN)
- 特點
    - conditioning the model on additional information one can influence the data generation process
- 優點
    - we can exploit the conditioning to generate samples of a required label.
- 驗證資料
    - MNIST
- 類似GAN
    - FC-GAN: 快速收斂

#### Wasserstein GAN (WGAN)
- 特點
    - WGAN focus solely on the learning of GAN.
    - use the maximum likelihood estimation (MLE) over Wasserstein distance
- 優點
    - train the critic till the optimality
    - prevent collapsing modes

#### Improved WGAN
- 特點
    -  the authors propose to penalize the norm of the gradient of the critic with respect to its input 
- 優點
    - 改進WGAN flawed learning(where only poor quality samples are generated)

#### Boundary-Seeking Generative Adversarial Networks(BGAN)
- 方法
    -  training the generator in order to produce the samples lying on the decision boundary of the current discriminator
- 特點
    -  training the generator in order to produce the samples lying on the decision boundary of the current discriminator
- 優點
    - proved to be more stable against the mode collapse.
    - a definition of a unified learning frame- work for both discrete and continuous variables

### GAN:Recent Advances
####  BigGAN
- 優點
    - achieved a new level of performance
    - fine control over the trade-off between sample fidelity and variety

#### Self-Attention Generative Adversarial Network (SAGAN)
- 優點
    -  allows attention-driven modeling for image generation where details are generated using cues from all feature locations

## GAN Applications for autonomous driving

### Advanced Data Augmentation
- GAN create realitic looking images
    - from a black and white image to a colored one
    - areal image to map
    - edges to a photo-realistic images of the sketched objects
    - day to night
    - summer to winter
    - context-aware object placement
-  However, the task is much more difficult thanks to the temporal information, which also has to remain consistent.
- CGAN、CycleGAN+UNIT、AC-GAN.........未完的待續
- 
#### CGAN
- 功能
    - Image-to-Image translation as an instance
- 方法
    - mixed with a traditional L1 loss

#### CycleGAN
- 方法
    - Briefly, the authors are learning a mapping G : X → Y , such that the distribution of images from G(X ) is indistinguishable from the distribution Y . Because such mapping is highly under-constrained, they couple it with an inverse mapping F : Y → X and introduce a cycle consistent loss enforcing F(G(X)) ≈ X, and vice versa.
- 優點
    - operate without a specific supervision
- 缺點
    - the generated images, after a careful inspection, show the same artifacts as the previous work

#### Synthesis
- visual perception
- fix noisy input
#### 2D Synthesis
-  Image-to-Image translation can be approached based on two main directions : paired or unpaired ; unimodal or mutilmodal
- unimodal paired image translation
    - 特點
        - the model learns to map images where the training data is organized in pairs of input and output samples
        - In many cases, the paired training data could not be available.
    - 範例
        - Pix2Pix
        - SRGAN
- unimodal unpaired
    - 特點
        - the image translation is conducted on unpaired data from two domains, where it learns a mapping between the two domains without supervision
    - 範例
        - CycleGAN
        - DiscoGAN
        - StarGAN
        - UNIT
- multimodal image translation
    - 特點
        - generate several images of different styles based on a single source image
    - 範例
        - Pix2PixHD
        - BicycleGAN
        - unpaired MUNIT
        - Augmented GAN
#### 3D Synthesis
- 目的
    - LiDAR can perceive accurate depth and to produce 3D point clouds
    - Most of GAN approaches are not applicable to 3D point clouds
- 範例
    - Point Cloud GAN (PC-GAN) 
        - proposed a two fold modification to GAN algorithm for learning to generate point clouds
    - 3D-GAN framework
        - map from a low-dimensional probabilistic space to the space of 3D objects
    - PrGAN
        -  investigated the task of generating a distribution over 3D structures given 2D views of multiple objects taken from unknown viewpoints.
#### Video Synthesis
- 目的
    - create new interactive 3D virtual worlds for different domains
- 範例
    - Temporal GAN(TGAN)
        - learns a semantic representation of unlabeled videos and generates videos, using a temporal generator and an image generator.
    - from early GAN network for video with a spatio-temporal convolutional architecture
        - scene is separated from the background and generating small one second  videos.
#### Domain adaptation from simulation to real
- 目的
    - using simulated environments enables much easier collection
    - simulated environments often fail to generalize on real environments
- GraspGAN
    - 功能
        -  extended the pixel-level domain adaptation to reduce the number of real world samples needed by up to 50 times for vision-based grasping system
- Reinforcement Learning
    - 方法
        - Two image-to-image translation networks are used
        -  The first network translates virtual images to their segmentation, the second network translates segmented images into their realistic counterpart
#### Object Detection
- 目的
    - inferring the occluded objects is essential for scene understanding and taking decisions.
- SeGAN
    - 功能
        - an approach for both segmentation and generation of the occluded parts of objects
    - 方法
        - the proposed network has three parts: segmentor, generator, and discriminator
- Perceptual-GAN
    - 功能
        - narrows representation difference of small and large objects
    - 方法
        - the generator learns to transfer the small objects representations large ones
#### Super Resolution
- 目的
    - enable and enhance the systems that were trained on high resolution inputs
- SRGAN
    - able to infer photo-realistic natural images for 4x upscaling factors
#### Inpainting
- 目的
    -  sensors may read noisy data or may suffer from failures causing incomplete readings, and Inpainting can provide a solution
### Semi-supervised/Unsupervised Learning
- $a$-GAN
    - 方法
        - combines VAE(Variational Autoencoders) and GAN
            - VAE $\rightarrow$ one of the most popular approaches to unsupervised
    -  目的
        -  the best of both worlds is used, and the limitations of both methods are mitigated
        -  沒說互補了哪些優缺點
-  unsupervised pre-training is beneficial for deep learning in general
### Learned Loss Functions
- DAN
    - 目的
        - semi-supervised learning and loss function learning
    - 方法
        - uses two discriminators
            - Predictor(P):receives a data point x on input and outputs a prediction p(x)
            - Judge(J):receives a data point x together with a label y,produces a single scalar J(x, y) representing the probability that x, y came from the labeled training data, rather than being predicted by P. 
    - 特點
        - P does not make use of labels, so the semi-supervised learning is pretty straightforward within this framework
        - concentrates on learning loss functions for discriminative models
### Adversarial training/testing
- 特點
    - attacks to weaken the performance of CNN by addition of noise
    - can also be interpreted as loss function learning
    - we can use adversarial loss for improving the final classifier robustness
- EL-GAN
    - Since there are very stringent requirements on safety in AD, the adversarial examples generation might be used as a tool for testing corner cases and robustness

## Our Results
- 問題
    - The image deterioration by soiling and adverse weather is caused either by presence of some “soiling categories”. So we have to enhance the image quality 
    - obtaining the relevant data is both very problematic and expensive
- 方法
   - CycleGAN
       - sorted our images to two categories : clean、soiled
       - recognize which parts of the image are soiled
       - desoiling
       - “desoiling” generator
           - learned to introduce shadow of the car body to the image
           - the vast majority of images in the “clean” category contained shadow of the car body
       - “soiling” generator 
           - learned that the weather was usually cloudy on our images from the “soiled” category.
   - MUNIT
       - ability to split content from the style, which would help our intention to possess the control over generated images and therefore ease the further classifiers training. 
       - Fail
## Discussion
-  discuss the main challenges of GAN
### Quantitative Evaluation
- 作法
    - generative models evaluation is based on the model likelihood.
- 方法
    -  Inception Score
        -  conditional label distribution of samples containing meaningful objects should have low entropy and the variability of the samples should be high
        -  優點
            -  well correlated with scores from human annotators
        -  缺點
            -  IS is found to be insensitive to the prior distribution over labels
    -  Fre ́chet Inception Distance
        -  方法
            1. the samples are embedded into a feature space given by a specific layer of the Inception Net.
            2. these are modeled as a continuous multivariate Gaussian distribution
            3. quantify the mean and covariance which is estimated for the generated and the real data and the Fre ́chet distance is evaluated
        - 優點
            - FID score showed to be consistent with human judgment
            - FID can detect intra-class mode dropping
    - 比較
        - IS mainly captures precision 
        - FID captures both precision and recall
### Adversarial examples and Safety
- 問題
    - inputs to machine learning models that have been intentionally modified to fool the model

- Defensive Distillation mechanism
    - 方法
        - trains a model whose surface is smoothed in the directions an attacker will typically try to exploit
    - 目的
        - making it difficult to discover adversarial examples
### Optimization Stability
- designed to minimize loss function
- 方法
    - minibatch discrimination
    - identifying the Kullback-Leibler (KL) divergence minimization task $\rightarrow$ distributions supported by low-dimensional manifolds $\rightarrow$ KL not defined or simply infinite $\rightarrow$ propose to use a different distance function(earth-mover, or Wasserstein, distance)

## Conclusions
- GAN have a potential for high impact for autonomous driving applications
- discussed the main challenges and open problems which have to be resolved in order for it to be more practically used

要來一段一段討論怎麼報告嗎?? 嗯嗯
我們要用那個簡報軟體？ office google hackmd icloud?

好問題
我其實不太知道老師這次報告是希望我們在論文中學到東西 <-其次
還是希望我們可以練習報告 <- 我覺得是這個
還是都有~~

如果要練習報告的話 感覺座簡報比較好一點  google 嗯嗯 好啊
如果要論文學東西  就hackmd直接上   <--不然也是可以hackmd<----做一個重點摘要版的就好，拿這一篇來刪減
也是可以，之前做起來有比office這類的快，因為不用準備背景之類的圖
OK
我先開個範本，在看看要不要
office google hackmd icloud 這幾個都是簡報或有簡報功能


這裏 https://hackmd.io/@NtutShare/SJ34R9EPw/edit





<!---

---

### Semi-supervised/Unsupervised Learning
- $a$-GAN
    - combines VAE(Variational Autoencoders) and GAN
    - ![](https://i.imgur.com/6uydAp0.png)

---

### Learned Loss Functions
- DAN
    - ![](https://i.imgur.com/nqCyg9o.png)

--->

