Notes on "Unsupervised Intra-domain Adaptation for Semantic Segmentation through Self-Supervision"

tags: notes unsupervised domain-adaptation segmentation self-supervised cvpr20

Author: Akshay Kulkarni

Note: CVPR '20 Oral, Code

Brief Outline

They propose a two-step self-supervised DA approach to minimize the inter-domain and intra-domain gap together.

1. They conduct inter-domain adaptation and from this, they separate the target domain into an easy and hard split using an entropy-based ranking function.
2. For the intra-domain adaptation, they propose a self-supervised adaptation technique from the easy to the hard split.

Introduction

• Target data collected from real world have diverse scene distributions, caused by various factors such as moving objects, weather conditions, which leads to a large gap within the target (intra-domain gap).
• Previous DA works focus more on inter-domain gap, so this paper presents a 2-step DA approach to minimize the inter-domain and the intra-domain gaps.
• Their model consists of 3 parts
  • An inter-domain adaptation module to close the inter-domain gap between labeled source data and unlabeled target data.
  • An entropy-based ranking system to separate target data into an easy and hard split.
  • An intra-domain adaptation module to close intra-domain gap between the easy and hard split (using pseudo labels from the easy domain).

Methodology

• Let
  $\mathcal{S}$ denote a source domain consisting of a set of images
  $\in {\mathbb{R}}^{H\times W\times 3}$ with their associated ground-truth
  $C$-class segmentation maps
  $\in (1,C{)}^{H\times W}$. Similarly, let
  $\mathcal{T}$ denote a target domain containing a set of unlabeled images
  $\in {\mathbb{R}}^{H\times W\times 3}$.
• The first step is inter-domain adaptation, based on common UDA approaches (Tsai et. al. 2018 and Vu et. al. 2019). Then, the pseudo labels and predicted entropy maps are used by an entropy-based ranking system to cluster the target data into the easy and hard split.
• The second step is intra-domain adaptation, which consists of aligning the pseudo-labeled easy split with the hard split. The full procedure is illustrated in the figure above.
• The proposed network consists of the inter-domain generator and discriminator
  $\{G_{inter},D_{inter}\}$, and the intra-domain generator and discriminator
  $\{G_{intra},D_{intra}\}$.

Inter-Domain Adaptation

• A sample
  $X_s\in {\mathbb{R}}^{H\times W\times 3}$ is from the source domain with it's associated map
  $Y_s$. Each entry
  $Y_s^{(h,w)}=[Y_s^{(h,w,c)}{]}_c$ of
  $Y_s$ provides a label of a pixel
  $(h,w)$ as a one-hot vector.
• The network
  $G_{inter}$ takes
  $X_s$ as input and generates a "soft segmentation map"
  $P_s=G_{inter}(X_s)$.
  $G_{inter}$ is optimized in a supervised way by minimizing the CE loss

$$\begin{array}{}\text{(1)}& {\mathcal{L}}_{inter}^{seg}(X_s,Y_s)=-\sum _{h,w}\sum _c{Y}_s^{(h,w,c)}\log (P_s^{(h,w,c)})\end{array}$$

• ADVENT (Vu et. al. 2019) assumes that trained models tend to produce over-confident (low entropy) predictions for source-like images, and under-confident (high entropy) predictions for target-like images. Based on this, they propose to utilize entropy maps to align the distribution shift of the features.
• This paper adopts ADVENT for inter-domain adaptation due to it's simplicity and effectiveness. The generator
  $G_{inter}$ takes a target image
  $X_t$ as input and produces the segmentation map
  $P_t=G_{inter}(X_t)$, and the entropy map
  $I_t$ is formulated as

$$\begin{array}{}\text{(2)}& I_t^{(h,w)}=\sum _c-P_t^{(h,w,c)}\log (P_t^{(h,w,c)})\end{array}$$

• To reduce the inter-domain gap,
  $D_{inter}$ is trained to predict the domain labels for the entropy maps while
  $G_{inter}$ is trained to fool
  $D_{inter}$, and the optimization is achieved via the loss function

$$\begin{array}{}\text{(3)}& {\mathcal{L}}_{inter}^{adv}(X_s;X_t)=\sum _{h,w}\log (1-D_{inter}(I_t^{(h,w)}))+\log (D_{inter}(I_s^{(h,w)}))\end{array}$$

• Here,
  $I_s$ is the entropy map of
  $X_s$. The loss functions 2 and 3 are optimized to align the distribution shift between source and target domains.

Entropy-based Ranking

• Some target prediction maps are clean (confident and smooth) while others are noisy, despite being generated from the same model. Since this intra-domain gap exists among target images, a straightforward solution is to decompose the target domain into small subdomains.
• To build these splits, they use entropy maps to determine the confidence levels of target predictions. They rank the predictions using the mean value of the entropy map
  $I_t$ given by

$$\begin{array}{}\text{(4)}& R(X_t)=\frac{1}{HW}\sum _{h,w}{I}_t^{(h,w)}\end{array}$$

• Let
  $X_{te}$ and
  $X_{th}$ denote a target image assigned to the easy and hard splits respectively. For domain separation, they define
  $\lambda =\frac{|X_{te}|}{|X_t|}$ where
  $|X_{te}|$ is the cardinality (number of elements) of easy split, and
  $|X_t|$ is the cardinality of the whole target set.
• Note that a threshold value for separation is not used since it would be specific to a dataset. They choose the ratio as a hyperparameter, which shows strong generalization to other datasets.

Entropy Normalization

• Complex scenes (containing many objects) might be categorized as hard.
• For a more representative ranking, they adopt a new normalization by dividing the mean entropy with the number of predicted rare classes in the target image.
• Note that rare classes are pre-defined from the set of all classes (see results section in the paper for definition).
• The entropy normalization helps to move images with many objects to the easy split.

Intra-domain Adaptation

• They propose to use the predictions from
  $G_{inter}$ as pseudo labels for the easy split. Given image from easy split
  $X_{te}$, the prediction map
  $P_{te}=G_{inter}(X_{te})$ is a soft-segmentation map, which is converted to
  ${\mathcal{P}}_{te}$ where each entry is a one-hot vector.
• Using these pseudo-labels,
  $G_{intra}$ is optimized by minimizing the CE loss

$$\begin{array}{}\text{(5)}& {\mathcal{L}}_{intra}^{seg}(X_{te})=-\sum _{h,w}\sum _c{\mathcal{P}}_{te}^{(h,w,c)}\log (G_{intra}(X_{te}{)}^{(h,w,c)})\end{array}$$

• An image
  $X_{th}$ from the hard split gives the segmentation map
  $P_{th}=G(X_{th})$ (note that this is the
  $G$ being trained and not the fixed
  $G_{inter}$) and the entropy map
  $I_{th}$.
• To close the intra-domain gap, the intra-domain discriminator
  $D_{intra}$ is trained to predict the split labels of
  $I_{te}$ (easy split) and
  $I_{th}$ (hard split), and
  $G$ is trained to fool
  $D_{intra}$. The adversarial loss can be formulated as

$$\begin{array}{}\text{(6)}& {\mathcal{L}}_{intra}^{adv}(X_{te},X_{th})=\sum _{h,w}\log (1-D_{intra}(I_{th}^{(h,w)}))+\log (D_{intra}(I_{te}^{(h,w)}))\end{array}$$

• The complete loss function
  $\mathcal{L}$ is the sum of all 4 equations (2, 3, 5, 6) and the objective is to learn a target model
  $G$ according to

$$\begin{array}{}\text{(7)}& G^{\ast }={\text{argmin}}_{G_{intra}}\underset{\begin{array}{c}{G}_{inter}\\ G_{intra}\end{array}}{min}\underset{\begin{array}{c}{D}_{inter}\\ D_{intra}\end{array}}{max}\mathcal{L}\end{array}$$

• Since the proposed method is a 2-step self-supervised approach, it is difficult to train in a single stage. They choose to minimize it in 3 stages as follows
  1. Train the inter-domain adaptation model to optimize
     $G_{inter}$ and
     $D_{inter}$.
  2. Generate target pseudo labels using
     $G_{inter}$ and rank all target images based on
     $R(X_t)$.
  3. Train the intra-domain adaptation model to optimize
     $G_{intra}$ and
     $D_{intra}$.

Note: See Section 4.3 of the paper for theoretical analysis (not able to understand yet).

Conclusion

• A self-supervised DA approach is proposed to minimize the inter-domain and intra-domain gaps simultaneously.