# SimCSE: Simple Contrastive Learning of Sentence Embeddings

## Introduction:

1\. SimCSE, a simple contrastive learning framework that greatly advances the state-of-the-art sentence embeddings.

2\. Dropout acts as minimal data augmentation and removing it leads to a representation collapse.

## Method:

### The architecture of a spatial transformer module</div>


### Background: Contrastive Learning</div>

Aims to learn effective representation by pulling semantically close neighbors together and pushing apart non-neighbors

**InfoNCE Loss**

**Positive instances**
simply using standard dropout

**Alignment and uniformity**
**Alignment**: two samples forming a positive pair should be mapped to nearby features, and thus be (mostly) invariant to unneeded noise factors.

**Uniformity**: feature vectors should be roughly uniformly distributed on the unit hypersphere Sm−1, preserving as much information of the data as possible.

model collapse
The data are extremely unevenly distributed to the same point on the hypersphere. which means all the data converge to the same constant solution after the feature representation mapping process.

### Unsupervised SimCSE</div>

1. In standard training of Transformers, there are dropout masks placed on fully-connected layers as well as attention probabilities (default p = 0.1).
2. feed the same input to the encoder twice and get two embeddings with different dropout masks z, z0


**Dropout noise as data augmentation**

**Why does it work?**
### Supervised SimCSE</div>


result of Supervised SimCSE

## Results:</div>