Try   HackMD

2. DEIT (Data Efficient Image Transformers)

  • Trained on ImageNet only (1.2 million images)
  • Competitive performance on ImageNet (84.4%)
  • In addition to CLS token, also added distillation token => responsible for predicting output of a CNN model (RegNetY-16GF)
  • To make it work on a smaller amount of data, three techniques were used:
    • data augmentation: repeated augmentation, auto-augment, rand-augment, random erasing, mixup, cutmix
    • optimization
    • regularization: trained on (224,224) then finetuned on (384,384) (Question: HOW???)
      • it was ensured that the L2 norm of enlarged patches was the same as the L2 norm of regular patches

Implementation

Model

  • DistilledVisionTransformer is defined here

It simply adds a self.dist_token and updates the number of positional embeddings to 2 + num_patches.

  • 2D image to patch embedding

  • An image (B,C,H,W) in transformed into a tensor of size (B,2+num_patches,embed_dim)

  • We only care about the CLS token and distillation token outputs as shown here.

  • During inference, we take the mean of both tokens as shown here.

  • A Block is defined here.

Losses

  • Implemented in this file: pretty straight forward - weighted loss of base_criterion and distillation loss.
tags: vit
Last changed by 
 
vinsis
0
189

Read more

How to implement a GNN

import torch from torch_scatter import scatter We will use this graph: num_nodes = 5 num_edges = 6 num_edge_types = 3 edge_index = torch.LongTensor([

Dec 20, 2022
Metaformer is Actually What You Need for Vision [CVPR 2022]

Paper: https://arxiv.org/abs/2111.11418 Key idea: abstract the network architecture from high performing models like Transformers, MLP-Mixers etc. It is this network that gives good performance. They replace transformer, MLP-mixer etc with pooling to prove this statement. The main thing to understand is how pooling works: class Pooling(nn.Module): """ Implementation of pooling for PoolFormer --pool_size: pooling size """

Dec 9, 2022
Patches are all you need?

Main claim: patches are what lead to an improved performace at least to a certain extent Stem implementation self.stem = nn.Sequential( nn.Conv2d(in_chans, dim, kernel_size=patch_size, stride=patch_size), activation(), nn.BatchNorm2d(dim) ) Blocks implementation

Dec 9, 2022
MLP-Mixer: An All-MLP Architecture for Vision

Mostly the same people behind ViT paper. Adequate (84.15 top 1 on ImageNet by Mixer-L/16) but not SOTA. Benefits much more from scaling up. Common part with ViT: Divide an image into NxN patches, unroll each patch and do a linear transform. Some simple Mlp = Linear -> Activation -> Dropout -> Linear -> Dropout style MLP layers implemented here class Mlp(nn.Module): """ MLP as used in Vision Transformer, MLP-Mixer and related networks """ def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0.): super().__init__()

Dec 9, 2022
Read more from vinsis
Published on HackMD