Example : If the model has never seen tricycle, it still should give a plausible prediction as vehicle.

They take each class in ADE20K dataset and relate it with a synset(synonym set) from WordNet, end up with 2019 unique synsets forming a DAG with entity being the common root.

A max-margin loss is used to learn the embedding function \(f(\cdot)\) for mapping the concept space to joint embedding space.

They argue that since label retrieval is a ranking problem, negative labels should be introduced to push scores of positive labels to be larger than those of negative.

Initially, they use a max-margin loss for learning the mapping \(g(\cdot)\) from pixel feature space to the joint embedding space, but find that using softmax in the form of a triplet loss performs better.

\[ \begin {align} \newcommand\ddfrac[2]{\frac{\displaystyle #1}{\displaystyle #2}} \mathcal{L}_{image}(x_{i,j}) &= -\log\bigg(\ddfrac{e^{S_{image}(f(y_{i,j}), g(x_{i,j}))}}{e^{S_{image}(f(y_{i,j}), g(x_{i,j}))} + \sum_{y'_{i,j}}{e^{S_{image}(f(y'_{i,j}), g(x_{i,j}))}}}\bigg) \\ where,\ x_{i,j} &= pixel\ features\ of\ the\ (i,j)^{th}\ pixel\\ y_{i,j} &= label\ of\ the\ (i,j)^{th}\ pixel\\ y'_{i,j} &= negative\ labels\ for\ the\ (i,j)^{th}\ pixel \end{align} \]

Their 'Image Stream' uses an adapted version of VGG-16 (to make the embeddings have a dimension equal to the word concept embeddings).

Also, (in the latent space), they fix the norms of the image embedding pixels to 30 to improve numerical stability (since pixel embeddings are the most specific concept in the joint embedding space).

The 'Concept Stream' is trained first and the trained word embeddings are used as initializations for training loop.

Zero-shot: Here, however, they were able to consistently perform better than the baselines.

They also find that using the asymmetric scoring function gives a significant improvement.

Only hierarchical metrics and information content ratio were used for comparisons here.

Qualitatively, they show that in places where the model is unsure of the specific object, it correctly predicts a more general concept.

For example, in a rocking chair, the top part looks like a chair so it classifies that correctly, but the bottom part is not like a normal chair, and since it hasn't seen that particularly, it classifies it as ' furniture', which is plausible and human-like.

They also do a 'concept search' in the embedding space to show that though baseline models can learn specific objects equally well, when more abstract terms are 'searched' for in the joint embedding space, their model is still able to detect them in images whereas baseline models aren't.

They also show that because objects like chair and bench are close in the joint embedding space, so by looking in the vicinity of chair, they hypothesize that they will find sittable objects.

They do this by maximizing the agreement (via cosine similarity) of the \(N\) correct pairs, and, pushing away/reducing agreement between the \(N^2 - N\) negative pairs.

Image Encoder: Because of the wide variety of architectures and designs available, they ended up choosing two architectures.

• One is based on ResNet50, with a modifications to the layers and replacing global average pooling with 'attention pooling'. They mention

  'transformer-style' multi head QKV attention, query is conditioned on the global average-pooled representation of the image

• The other one is based on the recent Vision Transformer (ViT). They make only minor changes to this architecture.

• They argue that for the ResNet based encoders, increasing one dimension alone (either depth, width or resolution) is less beneficial than increasing all dimensions together (keeping the computing resources same).

Text Encoder: The text encoder is taken as a transformer with some previously published modifications.
They only scale the width of this encoder as they find that CLIP is less sensitive to the text encoder.

Training Details

ResNets: They train 5 models (ResNet50,ResNet101,"Efficient-Net" style RN50x4,RN50x16,RN50x64)
ViT: They train 3 models (ViT-B/32,ViT-B/16,ViT-L/14)

The largest ResNet model, RN50x64, took 18 days to train on 592 V100 GPUs while the largest Vision Transformer took 12 days on 256 V100 GPUs.

\(F\) is calculated as the dot product of the image embeddings \(I\) and label embeddings \(T\).
\[ \begin {align} F_{i,j,k} &= I_{i.j}\cdot T_{k} \\ dimensions:\ I_{i,j} &\in \mathbb{R}^{C},\ \{i,j\}\ represent\ pixels\\ \ T_{k} &\in \mathbb{R}^{C},\ k \in \{1..N\} \\ \ F_{i,j} &\in \mathbb{R}^{N} \end {align} \]
So, they want to maximize the dot product \(F_{i,j,k}\) for those pixels \(\{i,j\}\) where \(y_{i,j} = k\) (GT label). They do this by applying softmax over \(k\) on \(F_{i,j,k}\) and taking a CrossEntropy loss.

For the final step, The softmaxed feature block \(F\) (equivalent to predictions) is then 'spatially regularized' using a DepthwiseBlock(Depthwise Conv) or a BottleneckBlock(Depthwise Conv augmented with max-pooling), and is upsampled to the input image's resolution using bilinear interpolation.

Training Details: They use pretrained weights on ImageNet for ResNet and ViT image encoders, and take random initialization for DPT. They freeze the text encoder(the ViT-B/32 from CLIP) while training.

They show results that are comparable with 1-shot state-of-the-art(HSNet) results, and significantly higher than previous zero-shot models on PASCAL-5i and COCO-20i. They outperform HSNet on FSS-1000.

They use different text encoders from CLIP and compare them. (The text encoder is always a simple Transformer, the difference is the image encoder it is co-trained with in the CLIP pretraining step).

Related but unseen labels

They show that LSeg is able to predict objects belonging to unseen classes close to the points in embedding space.

They show the same behavior with hierarchical unseen labels (i.e. being able to predict correctly when a parent category is present in label set instead of the specific object).

Failure Cases

They mention that since LSeg is trained only on positive examples of classes (unlike CLIP which had a contrastive objective), it can give wrong predictions sometimes. For example,

In this image, it predicts the dog as toy (when only toy and grass are provided) because a dog is probably closer to a toy than grass visually and semantically.

Framework

They make region descriptions by filling 'object concepts'(from concept pool) into prompts and then, using a teacher model \(\mathcal{V}_t\) (from CLIP), and sees which region (proposed by the Region Proposal Network, RPN, pretrained) aligns with the region description the most, and assigns it to that.
Once these region-text pairs are generated, the new encoder can be contrastively trained on these, similar to CLIP's contrastive language-image pretraining.

They use RoIAlign to extract the region's visual features from the encoder \(\mathcal{V}\), which pools regional features from the image's feature map using interpolation.

\(\mathcal{V}\) takes initial weights from \(\mathcal{V}_t\) for a good start in the visual-semantic space.

Details

The CC3M (contextual captions dataset) was used for training.

The region descriptions are made by filling the concepts from concept pool into prompts, i.e. kite is filled into the prompt a photo of a .... to make the description a photo of a kite. These are then passed through the pretrained language encoder (CLIP) to get the semantic text embedding.

Cosine Similarity is used as the metric of how much the region proposed aligns with some region description for the contrastive loss between region-text pairs \(L_{cntrst}\).

Losses

They use a distillation loss \(L_{dist}\) in addition to a contrastive loss, which is defined as:
\[ L_{dist} = \frac{1}{N} \sum_{i}{L_{KL}(q_i^t, q_i)} \]
where, \(q_i^t\) is a 'soft target' = \(softmax_j(distance(v_i^t, l_j))\),
\(v_i^t\) is region's visual features from teacher \(\mathcal{V}_t\)
and, \(v_i\) is region's visual features from \(\mathcal{V}\)

They also added the contrastive loss at image level \(L_{cntrst-img}\) (with negative samples being labels of different images), like CLIP to their final loss. So, final loss is
\[ L = L_{cntrst} + L_{dist} + L_{cntrst-img} \]

They show that the closed set accuracy is highly correlated to the open set performance.

• Performed multiple experiments using a variety of models: ViT, ResNet, EfficientNet, VGG.

• ViT doesn't overfit its representation to the training classes and outperforms other methods.

Good closed-set performance => Better OSR

To enhance the closed set performance, they leverage existing techniques from image recognition:

• label smoothing
• longer training times
• better augmentations
• better LR schedules

They also try changing the open set scoring rule to Maximum Logit Score (MLS).

Using MLS gives better performance in OSR but softmax normalization is better in combined (OSCR) (because softmax normalization cancels the effect of the feature norm)

Using CLIP features for dense prediction

• Failure: Fine-tuning the image encoder of CLIP for segmentation tasks.
• Performance is good on seen classes but modified DeepLabv2 in conjunction with CLIP's text fails to segment novel classes.
• Reasons:
  • The visual-language association of CLIP features should remain intact for best performance.
  • Loss of generality => Additional mapper trained on seen classes.

MaskCLIP

Doesn't modify the CLIP feature space

Base Class Performance

Image	Ground Truth	Ours (PSPNet)	MaskCLIP (w/o PD and KS)	MaskCLIP (w/ PD and KS)

Novel Class Performance

Image	Ground Truth	Ours (PSPNet)	MaskCLIP (w/o PD and KS)	MaskCLIP (w/ PD and KS)

Summary: