Notes on "FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation" [Paper]

tags: notes unsupervised domain-adaptation

Notes Author: Rohit Lal

---

Brief Outline

• propose a UDA method that effectively handles large domain discrepancies.
• introduce a fixed ratio-based mixup to augment multiple intermediate domains between the source and target domain.

Introduction

• an augmented domain close to the source domain has more reliable label information, but it has a lower correlation with the target domain.
• apply self penalization, which penalizes its own model to improve performance through self-training.
• to prevent divergence of the augmented models generated in different domains, we propose a consistency regularization using an augmented domain with the same ratio of source and target samples.

Methodology

FixBi algorithm

Fixed Ratio based Mixup

• Given a pair of input samples and their corresponding one-hot labels in the source and target domain:
  $(x_i^s,y_i^s)$ and
  $(x_i^t,{\hat{y}}_i^t)$, mixup settings are defined as follows:
  
  Note:
  ${\hat{y}}_i^t$ obtained from pseudo labels got from DANN
  
  $\lambda \in \{{\lambda }_{sd},{\lambda }_{td}\}$ such that
  ${\lambda }_{sd}+{\lambda }_{td}=1$
• Taking advantage of the fixed ratio-based mixup, we construct two network models that act as bridges between the source and target domain
• The source-dominant model has strong supervision for the source domain but relatively weak supervision for the target domain. By contrast, the target-dominant model has strong target supervision but weak source supervision.

Confidence based Learning

• one model teaches the other model using the positive pseudo-labels or teach itself using the negative pseudo-labels.

Bidirectional Matching with positive pseudo-labels

• when one network assigns the class probability of input above a certain threshold
  $\tau$ , we assume that this predicted label as a pseudo-label.
• Then we train the peer network to make its predictions match these positive pseudo-labels via a standard cross-entropy loss.
  

Self-penalization with negative pseudo-labels.

• the negative pseudo-label indicates the most confident label (top-1 label) predicted by the network with a confidence lower than the threshold
  $\tau$
• Since the negative pseudo-label is unlikely to be a correct label, we need to increase the probability values of all other classes except for this negative pseudo-label. Therefore, we optimize the output probability corresponding to the negative pseudo-label to be close to zero.
  
• $\tau$ changes adaptively by the sample mean and standard deviation of a mini-batch, not a fixed one.

Consistency Regularization

• assume that the well-trained models should be regularized to have consistent results in the same space.