---
# System prepended metadata

title: Quadruplet self-supervised using SwinT.

---

# Quadruplet self-supervised using SwinT.

Here, I am using MoCoV3 to try to perform the quadruplet self-supervised learning

![image](https://hackmd.io/_uploads/SJYRP3sKa.png)

Code dir: /home/compu/doanhbc/SSL_project

To train self-supervised learning:

```
cd /home/compu/doanhbc/SSL_project/SSL
bash train_mocov3_swint.sh
```

I mainly define the architecture at: /home/compu/doanhbc/SSL_project/SSL/moco/builder.py, class `MoCo_SwinT_Chessboard`.

As my understanding, the objective is that performing self-supervised learning such that sub-sequence within a possible quadruplet (the same image) to be close to each other, help to enhance the representation ability. Then, for downstream tasks, we only use one sub-sequence and pass it to SwinT for decision making, we hope to obtain better performance and faster inference.

At this stage, I used colon KBSMC to perform self-supervised learning on its training set first, then I perform the downstream task for classification.

When training downstream task, I perform the training as below figure (shared SwinT backbone for four sub-sequences, but seperate linear classifiers):

![image](https://hackmd.io/_uploads/SyHlsnstp.png)

When inference, I perform Entropy-weighted sum for four probabilities:

![image](https://hackmd.io/_uploads/SJewIjnotp.png)

Here is the result:

![image](https://hackmd.io/_uploads/HJBRonoYp.png)


- SwinT: original SwinT, with pre-trained ImageNet loaded, process with full tokens.
- SwinT-Quad w SSL (MoCov3): train SwinT with quadruplet self-supervised learning first, then train the downstream task.

Some disadvantage points at this stage:
- When training self-supervised learning, I load the pretrained ImageNet, but it should not be loaded, we should train from scratch to prove the effective self-supervised learning approach.
- When training down-stream task, I use four different classiers. This is not suitable, because after performing self-supervised learning, we hope four sub-sequences should be close to each others, then we target to only use one sub-sequence.

On-going:
- This approach I think should be performed self-supervised learning on general datasets, such as CIFAR100 first to see the effectiveness, then expand to train it on ImageNet.