# Rebuttal ICML 2024

# General comment
We want to thank all of the reviewers for their careful reviews of our manuscript. We are pleased with the overall positive feedbacks on the effectivity of the method (R-wQuf, R-Vo7f), the motivation of the paper (R-wQuf, R-RMT7j) and the novelty of the idea (R-Vo7f), the strong performance on zero-shot classification and retrieval (R-wQuf, R-Vo7f).

While we address each reviewer questions in their respective thread, we would like to highlight key results of general interest.

**Llip significantly and consistently outperforms the baselines.** To further demonstrate the significance of Llip, we trained a ViT-H/14 backbone with Llip and show that it **outperforms CLIP with a ViT-G/14 backbone trained on the same dataset while being more efficient.** We trained Llip with 64 mixtures tokens on a ViT-H/14 backbone and obtain **82.7%** accuracy on 0-shot ImageNet. This number is **0.6%** higher than MetaCLIP's reported number on a ViT-G while being **32%** faster to train when controlling for the hardware (256 A100 GPUs) and having an inference time **55%** lower when evaluating ImageNet. Note that Llip uses the same pre-trainining data as MetaCLIP and our implementation of Llip is based on the MetaCLIP's codebase.

**Comparison with SigLIP when controlling for the training dataset.** We trained a VIT-L/14 using SigLIP with the MetaCLIP dataset. When controlling for the training dataset, SigLIP obtains **79.4%** 0-shot ImageNet accuracy with a VIT-L/14. This performance is similar to MetaCLIP's performance with the same backbone (**79.2%**). Meanwhile, Llip obtains **80.9%** 0-shot ImageNet accuracy when trained with a ViT-L/14. Additional comparison for a VIT-B/32 and VIT-B/16 can found in Table 8 in the paper where we draw the same conclusions.

# Reviewer wQuf
We thank the reviewer for their feedback. We address the reviewer's questions and provide new exeprimental analysis.

**Does the visual mixing happens over L layers?** The mixing happens only at the last layer once after the image encoding. We thank the reviewer for pointing out a potentially confusing notation and based on the feedback we improved the clarity of our notation and use "H" instead of “L” to denote the number of cross-attention head.

**SigLIP number in Figure 4 is taken from SigLIP's paper.** This number is the outcome of a model trained with a larger private dataset. We present an apples-to-apples comparison of Llip and SigLIP, using our training dataset, with a ViT-B/32 and a ViT-B/16 in Table 8 and find that Llip outperforms SigLIP by **3.1%** and **3.0%** respectively on IN1K 0-shot evaluation.

**We additionally trained SigLIP with a ViT-L/14 with the MetaCLIP dataset** and obtain **79.4%** 0-shot IN1k accuracy which Llip outperforms by **1.5%** for the same backbone. We find that SigLIP's performance is comparable to CLIP's performance when we control for the training dataset and Llip consistently outperforms both CLIP and SigLIP for a given backbone.

**Inference time of Llip and CLIP's.** Following the reviewer's feedback, we will report the inference time in the paper. We ran additional measurements and report the inference time for IN1K's 0-shot (1000 prompts per image) in Table R1 below. Llip's inference time is slightly higher than CLIP for the same model size, while having **1.7%** improvement on 0-shot IN1K with a ViT-L/14 and **2.2%** improvement on 0-shot IN1k with a ViT-H/14

Additionally Llip outperforms larger CLIP models while requiring a significantly lower inference time. Compared to CLIP, Llip's inference time are:
* 40% lower on a Llip ViT-L/14 vs CLIP ViT-H/14 with 0.4% improvement on 0-shot IN1K.
* 55% lower on a Llip ViT-H/14 vs CLIP ViT-G/14 with 0.6% improvement on 0-shot IN1K.

**Training time of Llip and CLIP.** Table R1 also shows that Llip outperforms CLIP models with larger backbones which necessitate more training compute. Specifically, Llip outperforms a:
* CLIP ViT-H/14 with a ViT-L/14 by 0.4% on 0-shot IN1K while training 35% faster.
* CLIP ViT-G/14 with a ViT-H/14 by 0.6% on 0-shot IN1K while training 32% faster.

**Inference time as a function of the number of prompts.** We test the inference time of Llip with a ViT-L/14 for number of prompts in {10, 100, 1000, 10K}. We report the numbers in Table R2 and find that even when running an inference evaluation that requires 10K prompts, Llip's inference with a ViT-L/14 is faster than CLIP's inference time on a ViT-H/14.

**The length of the prompts does not influence the inference time of Llip** since we only use the last text token as text representation, as done in CLIP. Thus, the cross-attention is always performed with only one text query.

**Llip trained on half the number of iterations also reduces the number of sample seen**. For that reason, we argue that comparing Llip with larger scale baselines that require more compute is a more fair comparison.

**The difference between Figure 3 and Figure 4 is the backbone scale.** The number of samples seen is the same for models in Figures 3 and 4. Figure 3 reports numbers on a ViT-B/32 and Figure 4 reports numbers for a ViT-L/14. We will clarify the captions.

**SigLIP's numbers are obtained with MetaCLIP dataset instead of the Webli dataset.** We will put SigLIP numbers obtained by training with the MetaCLIP dataset in a different table and we will report SigLIP's number in Table 4.

**Table 1-3 compares Llip with numbers reported in the literature** and SigLIP does not report most of these numbers. We will add a row with the number SigLIP obtains when trained with a ViT-L/14 and the MetaCLIP dataset to Table 8.

**Clarity**: We thank the reviewer for pointing out these issues. Following your comment, we will clarify the introduction and Figure 1's caption.

We are happy to answer remaining questions.

**Table R1: Inference time, training time and resulting top-1 0-shot IN-1k accuracy for CLIP and Llip models. Inference times are obtained using 1 A100.**
|| Inference time (s/sample) for IN1k | Training time (s/iteration) | Top-1 0-shot IN1k accuracy |
|-|-|-|-|
| CLIP (ViT-L/14) | 0.009 | 1.11 (128 A100) | 79.2 |
| Llip (ViT-L/14; K=32)| 0.011 | 1.43 (128 A100) | 80.9 |
| CLIP (ViT-H/14) | 0.020 | 2.20 (256 A100) | 80.5 |
| Llip (ViT-H/14; K=64) | 0.024 | 3.10 (256 A100) | **82.7**|
| CLIP (ViT-G/14)| 0.054 | 4.55 (256 A100) | 82.1 |

**Table R2: Inference time of Llip with a ViT-L/14 for various prompt lengths. All experiments are run with 1 A100.**
|# of prompt | Inference time (s/sample) |
|-|-|
| 10 | 0.0107 |
| 100| 0.0109 |
| 1000 | 0.0111 |
| 10000 | 0.0118 |

# Reviewer Vo7f
We thank the reviewer for their comments. We are pleased to read that they appreciated the simplicity of the idea. We address the reviewer's questions below.

**What are the numbers of SigLIP in Table 1, Table 3, and Table 4?** We report SigLIP's number with ViT-B/32 and ViT-B/16 backbones in Table 8. When controlling for the dataset, Llips outperforms SigLIP by **3.2%** and **3.0%** on ViT-B/32 and ViT-B/16 respectively. Following the reviewers feedback, we additionally trained SigLIP with a ViT-L/14 on the MetaCLIP dataset and obtain **79.4%** on 0-shot IN1k accuracy. Meanwhile, Llip obtains **80.9%** accuracy when trained with a ViT-L/14.

**Have the authors tried to re-implement CLIP using the new datasets?** MetaCLIP's numbers in Table 1-4 and CLIP's number from Figure 1 and Table 8 are produced using the same dataset that we used to produce Llip's number. We also reproduced MetaCLIP's number using our codebase and present the result in the second row of Table 8 showing that we achieve comparable numbers as what was reported.

**Adding SigLIP loss definition.** We thank the reviewer for pointing this out. We will add a definition of the SigLIP objective in the paper to clarify the difference with Llip's objective.

**Ablation result of the InfoNCE loss.** We performed such an ablation in Table 8 in the Appendix and found that CLIP and SigLIP performed comparably. Moreover, SigLIP ViT-L/14 numbers are similar to MetaCLIP's ViT-L/14 reported number. We will refer to this experiment in the main paper.

**Additional classification results.** Thank you for suggesting this experiment! We use the standard linear probe experiment [0] on IN1k to assess the quality of the visual representations. We found that Llip outperforms SigLIP by **1.3%** for a ViT-B/32 model and present the results in Table R1 below.

**Qualitative analysis of the latent vectors**
We thank the reviewer for inquiring this qualitative analysis. Beyond learning a more expressive representation, we find that the different latent vectors learn different concepts and the contextualization allows to mix these concepts. For example, given this image [1] and two captions: ("A photo of an apple", "A photo of an ipod"), we find that that the visual representation of Llip given the first caption is closer to the first caption's representation and the visual representation of Llip given the second caption is closer to the second caption's representation. Meanwhile, CLIP can only output a single representation leading to the commonly known *typographic attack*. Thanks to your suggestion, we will add this discussion along with more qualitative examples to the paper.

<!-- How to interpret the semantics of the mixture tokens visually and qualitatively remains an open question. In this work we scope our research on the effect of modeling the caption's expressivity and have a qualitative analysis with the spectrum of the feature covariance matrix (Figure 5) which is a common way to assess representation expressivity in SSL. We find that more a more expressive representation leads to better downstream performance.

In our downstream experimental results and spectral analysis of the feature covariance matrix we show that having visual mixture tokens allows Llip to learn more expressive representations. In particular, we hypothesize that different latent vectors focus on different semantic aspects of the image. We will add qualitative analysis on downstream inference of individual examples in the next revision of the paper.
For example, for a fixed image $x_i$ when extracting visual representation with two different captions $t_1$ and $t_2$, we observe that
the weighting coefficient of the visual mixtures differ and the resulting visual representations $v_1$ and $v_2$ are closer to the corresponding caption representations. -->

**Table R1: Linear probe top-1 accuracy of SigLIP and Llip.**
|| IN-1K accuracy |
|-|-|
| SigLIP (B/32)| 76.8 |
| Llip (B/32; K=32) | **78.5** |
| Llip (B/32; K=64) | x |

[0] https://github.com/LAION-AI/CLIP_benchmark
[1] https://ibb.co/XpPp1By

# Reviewer MT7j
We appreciate the reviewer’s comments and thoughtful questions. Following the reviewer’s questions, we will update the discussion in the paper to improve the clarity of the manuscript. Inspired by reviewer's feedback, we also ran additional experiments during the rebuttal period which we discuss below: comparing models using linear probes and including image patch tokens in Llip's cross-attention.

**Llip's performance gains are consistent compared to existing methods.** When trained on the same dataset and the same backbone scale, Llip consistently demonstrates a significant improvement over SigLIP and CLIP as demonstrated in Table 8. We find that Llip outperforms SigLIP by **3.2%** on ViT-B/32 and **3.0%** on ViT-B/16 on 0-shot ImageNet accuracy. We also ran an additional pre-training of SigLIP with a ViT-L/14 and obtained **79.4%** vs **80.9%** obtained with Llip. Furthermore, we demonstrate that Llip outperforms CLIP models trained with larger backbones. Specifically, as denoted in Table R1 below, on 0-shot IN1K Llip outperforms:
* CLIP ViT-H/14 with a ViT-L/14 by 0.4% while training 35% faster.
* CLIP ViT-G/14 with a ViT-H/14 by 0.6% while training 32% faster.

**Inference time.** From Table R1, we find that Llip's inference time is only slightly higher than CLIP's for the same model size, while having a 1.4% improvement on 0-shot IN1K with a ViT-L/14 and a 2.2% improvement on 0-shot IN1k with a ViT-H/14.

Additionally, Llip outperforms larger CLIP models while having a significantly lower inference time. Compared to CLIP, Llip's inference time are:
* 40% lower on a Llip ViT-L/14 vs CLIP ViT-H/14, while simultaneously achieving a 0.4% improvement on 0-shot IN1K.
* 55% lower on a Llip ViT-H/14 vs CLIP ViT-G/14, while simultaneously achieving a 0.6% improvement on 0-shot IN1K.

**How does Llip compare with other methods on linear probe experiments?** Thank you for suggesting this analysis! Based on your suggestion, we ran linear probe experiments and report the results in Table R2. The experiments are based off [0] and we use the same linear probe procedure as described in Appendix A.3 of CLIP. For Llip, we probe the mixture tokens directly (without contextualization). We find that Llip outperforms the baselines. Specifically, it outperforms SigLIP by **1.7%** on a ViT-B/32 backbone.

<!-- **Why is there a need for the K mixture tokens?** Thanks to the reviewer suggestion, we find that Llip also works with the patch tokens of the image. Table R2 reports the number of Llip trained on patch tokens and mixture tokens and report a slight improvement. We find that increasing the amount of mixture tokens also improves the downstream 0-shot performance when we use the patch tokens. -->
**Using image patch tokens in Llip's cross-attention.** Inspired by reviewer's question, we ran additional experiments where we included image patch tokens at the last layer together with $K$ visual mixture tokens in the cross-attention. For this experiment, we use Llip with ViT-B/32 for which we have $P=49$ image patch tokens and we report results varying $K$ in Table R3. As we can see from the Table:
* Using a smaller number of additional visual mixture tokens $K=32$ is more effective than using $P=49$ image patch tokens + $K=1$ mixture token. We hypothesize that additional learnable tokens enable learning more expressive features.
* With the same number of visual mixture tokens $K$, adding image patch tokens in the cross-attention additionally slightly increases the performance.

<!-- Also note that for a fixed visual encoder model there is only a fixed number of image patch tokens that can be used in the cross-attention, while we have flexibility in how many visual mixture tokens we want to add, and in Figure 6 in the paper and Table R3 in the rebuttal we show that the higher number of mixture tokens $K$ leads to increased accuracy.
-->

**How 0-shot inference is implemented and how it avoids information leakage:**
Following the reviewer's feedback, we will add more details explaining how the 0-shot inference is implemented in the updated paper. In summary, the implementation of the 0-shot inference of Llip is similar to CLIP’s implementation. For a given image $x_i$, we have $N$ possible labels $t_k, k\in[N]$ for which we compute the similarity. The predicted label is the one with the highest similarity. The main difference between Llip's 0-shot inference and CLIP's is that we have to encode each image $x_i$ with each label $t_k$. Thus, for a given image $x_i$, we compute the cosine similarity between the normalized visual features $I_{ik}$ and text features $T_k$ for all $k\in[N]$ and we define the predicted label as the one that maximizes the cosine similarity.

There is no information leakage because the information of the ground truth label is not used during the prediction of the label in the above procedure.

**Why is $I_{ij}\cdot T_j, i\neq j$ optimized as a negative sample?** In the updated version, we will clarify that we assume access to a dataset of image, text pairs: $(x_i, t_i) \in \mathcal D$. An image and a caption are a positive pair when $x_i$ and $t_i$ share the same index and they are a negative pair when an image and text pair $(x_i, t_j)$ have a different index $i\neq j$. Since we optimize a contrastive objective between positive and negative sample pairs, we optimize $I_{ij}\cdot T_j$ as a negative because $I_{ij}$ is the visual representation of a negative pair $(x_i, t_j)$ and $T_j$ is the text representation of the negative caption.

**Was there any attempt (or ablation) to introduce $K$ latent tokens on the text side?** We believe that modeling the text representation as $K$ latent tokens is an interesting follow-up direction, however in the scope of this paper we focus on modeling semantic diversity of image representations.

**Pre-training on smaller scale data.** While we leave smaller-scale pre-training for future work due to the limited rebuttal timeline, we agree with the reviewer on the importance of research accessibility, and we will publicly release our codebase along with the public release of this paper.

We hope that we have clarified the reviewer's question and are happy to answer any follow-up questions.

**Table R1: Inference time, training time and resulting top-1 0-shot IN-1k accuracy for CLIP and Llip models. Inference times are obtained using 1 A100.**
|| Inference time (s/sample) for IN1k | Training time (s/iteration) | Top-1 0-shot IN1k accuracy |
|-|-|-|-|
| CLIP (ViT-L/14) | 0.009 | 1.11 (128 A100) | 79.2 |
| Llip (ViT-L/14; K=32)| 0.011 | 1.43 (128 A100) | 80.9 |
| CLIP (ViT-H/14) | 0.020 | 2.20 (256 A100) | 80.5 |
| Llip (ViT-H/14; K=64) | 0.024 | 3.10 (256 A100) | **82.7**|
| CLIP (ViT-G/14)| 0.054 | 4.55 (256 A100) | 82.1 |

**Table R2: Linear probe top-1 accuracy of SigLIP and Llip.**
|| IN-1K accuracy |
|-|-|
| SigLIP (B/32)| 76.8 |
| Llip (B/32; K=32) | **78.5** |
| Llip (B/32; K=64) | x |

**Table R3: 0-shot Top-1 accuracy on IN-1K for a Llip model with the patch tokens and K additional mixture tokens.**
|| Llip with image patch tokens | Llip w/o image patch tokens |
|-|-|-|
| K=1 | 68.2 | N/A |
| K=32 | 70.4 | 70.1 |
| K=64 | 70.7 | 70.4 |
| K=128 | 71.5 | 71.2 |

[0] https://github.com/LAION-AI/CLIP_benchmark