## JANuS-1000: Technical Notes

This document is intended to serve as a high-level overview of the systems and frameworks you'll need to familiarize yourself with in order to run the experiments for the JANuS-1000 paper.

## greene

Greene is NYU's HPC cluster. It's where you'll be training models.

### Links

https://sites.google.com/nyu.edu/nyu-hpc

### Properties

* No sudo / root
* No apt installs (no Linux package management) AFAIK
* Limited to 1 million files (called inodes) per user
* You need to request access to the ImageNet images by emailing HPC
* Don't run anything on login nodes except for basic io and file transfer
* Scratch files are deleted every 60 days (you can run find + chmod to prevent files from being deleted)
* If you're not using something, back it up to /archive/USER as a tar archive (archive has a very low inode allowance but a fairly large 2TB space allowance)
* /scratch/projects/hegdelab is our "research project space"; this also has a 1 million file inode limit, but files on it are not deleted. This should be used for shared archival storage. You can save to the folder with your username.
* Most Greene jobs time out after 2 days, so you often need to resume long training jobs multiple times. You can request a small number of GPUs for as many as 4 days. If you need to run for longer than that, you have to email HPC and ask for permission

### environment management

The recommended way to manage Greene environments is by storing them on singularity containers.

### Parameters

#### SIF Containers

SIF containers are the base disk images on which Singularity environments run. The ones we use usually contain a copy of Ubuntu Linux and some CUDA frameworks. These are read-only.

CUDA 11.3 + Ubuntu 20.04
/scratch/work/public/singularity/cuda11.3.0-cudnn8-devel-ubuntu20.04.sif

ROCM 5.1.1 + Ubuntu 20.04
/scratch/work/public/singularity/rocm5.1.1-ubuntu20.04.4.sif

CUDA 11.7 + Ubuntu 22.04
/scratch/work/public/singularity/cuda11.7.99-cudnn8.5-devel-ubuntu22.04.2.sif

#### Overlays

Overlay containers are like mountable file-systems for your SIF container. They come in at least two flavors, EXT3 and SQF.

SQF containers are compressed, read-only containers; they're ideal for storing large amounts of small files that you never need to modify (like datasets).

EXT3 containers are uncompressed and can be written to; they're typically used for storing environment-specific packages (conda, pip, etc).

You can only mount an image in rw mode on one node at a time.

EXT3 containers have a limited amount of space, and they can fill up pretty quickly, so bear that in mind if you run into trouble.

read-only
--overlay PATH/TO/IMAGE:ro

read-write
--overlay PATH/TO/IMAGE:rw

#### Some overlays we use

IMAGENET 21k and IMAGENET 1k
$(for sqf in /vast/work/public/ml-datasets/imagenet/winter21_whole/*.sqf; do echo "--overlay $sqf:ro"; done) \
--overlay /vast/work/public/ml-datasets/imagenet/imagenet-train.sqf:ro \
--overlay /vast/work/public/ml-datasets/imagenet/imagenet-val.sqf:ro \

YFCC-15m
$(for sqf in /vast/work/public/ml-datasets/yfcc15m/data/*.sqf; do echo "--overlay $sqf:ro"; done) \

IN100
--overlay /vast/work/public/ml-datasets/bf996/CaptionNet/in100.sqf:ro \

LAION100
--overlay /vast/work/public/ml-datasets/bf996/CaptionNet/laion100.sqf:ro \

OPENIMAGES
--overlay /vast/work/public/ml-datasets/bf996/CaptionNet/openimages1000.sqf:ro \

INATURALIST 2021 VALIDATION
--overlay /vast/work/public/ml-datasets/bf996/inat_2021_valid.sqf:ro \

IMAGENET SHIFT TEST SETS
--overlay /vast/work/public/ml-datasets/bf996/imagenet-r.sqf:ro \
--overlay /vast/work/public/ml-datasets/bf996/imagenet-a.sqf:ro \
--overlay /vast/work/public/ml-datasets/bf996/imagenet-sketch.sqf:ro \
--overlay /scratch/projects/hegdelab/bf996/datasets/objectnet.sqf:ro \

#### Making your own SQF Overlays

Example; converting the folder insecta/train to a squashfs

find insecta/train -type d -exec chmod 755 {} \; && find insecta/train -type f -exec chmod 644 {} \; && mksquashfs insecta/train/ insecta/insecta_train_small.sqf -keep-as-directory

#### Loading your environment

Once your Singularity container is launched, you have to load the conda environment you want to use. If you set up your singularity container according to the instructions on HPC's website, you can do this by simply typing:

source /ext3/env.sh

If you have opened your container in rw mode, you can now install and modify packages on your container.

#### Example Commands

ROCM

singularity \
    exec --rocm \
    --bind $tmp:$HOME/.config/miopen \
  $(for sqf in /vast/work/public/ml-datasets/yfcc15m/data/*.sqf; do echo "--overlay $sqf:ro"; done) \
  $(for sqf in /vast/work/public/ml-datasets/imagenet/winter21_whole/*.sqf; do echo "--overlay $sqf:ro"; done) \
  --overlay /scratch/bf996/singularity_containers/openclip_env_rocm.ext3:ro \
  --overlay /vast/work/public/ml-datasets/bf996/CaptionNet/in100.sqf:ro \
  --overlay /vast/work/public/ml-datasets/bf996/CaptionNet/laion100.sqf:ro \
  --overlay /vast/work/public/ml-datasets/bf996/CaptionNet/openimages1000.sqf:ro \
  --overlay /vast/work/public/ml-datasets/bf996/imagenet-r.sqf:ro \
  --overlay /vast/work/public/ml-datasets/bf996/imagenet-a.sqf:ro \
  --overlay /vast/work/public/ml-datasets/bf996/imagenet-sketch.sqf:ro \
  --overlay /scratch/projects/hegdelab/bf996/datasets/objectnet-imagenetonly.sqf:ro \
  --overlay /vast/work/public/ml-datasets/imagenet/imagenet-train.sqf:ro \
  --overlay /vast/work/public/ml-datasets/imagenet/imagenet-val.sqf:ro \
  /scratch/work/public/hudson/images/rocm4.5.2-ubuntu20.04.3.sif \
  /bin/bash

CUDA

singularity \
  exec --nv \
  $(for sqf in /vast/work/public/ml-datasets/yfcc15m/data/*.sqf; do echo "--overlay $sqf:ro"; done) \
  $(for sqf in /vast/work/public/ml-datasets/imagenet/winter21_whole/*.sqf; do echo "--overlay $sqf:ro"; done) \
  --overlay /PATH/TO/openclip_env_cuda_n.ext3:ro \
  --overlay /vast/work/public/ml-datasets/bf996/CaptionNet/in100.sqf:ro \
  --overlay /vast/work/public/ml-datasets/bf996/CaptionNet/laion100.sqf:ro \
  --overlay /vast/work/public/ml-datasets/bf996/CaptionNet/openimages1000.sqf:ro \
  --overlay /vast/work/public/ml-datasets/bf996/imagenet-r.sqf:ro \
  --overlay /vast/work/public/ml-datasets/bf996/imagenet-a.sqf:ro \
  --overlay /vast/work/public/ml-datasets/bf996/imagenet-sketch.sqf:ro \
  --overlay /vast/work/public/ml-datasets/imagenet/imagenet-train.sqf:ro \
  --overlay /vast/work/public/ml-datasets/imagenet/imagenet-val.sqf:ro \
  /scratch/work/public/singularity/cuda11.3.0-cudnn8-devel-ubuntu20.04.sif \
  /bin/bash

## timm

Timm, or pytorch-image-models, is a popular Huggingface framework for training and evaluating computer vision models.

### Links

https://github.com/huggingface/pytorch-image-models

### Uses

* Training vision backbones
* ImageNet-1k evaluation

### Individual Model Info (example)

https://huggingface.co/timm/vit_base_patch16_rpn_224.sw_in1k

### Results Sheets

https://github.com/huggingface/pytorch-image-models/blob/main/results/

### Parameters

timm stores parameters in YAML config files. There are a huge number of possible parameters to modify in timm -- here are a few we use frequently.

resume: "PATH/TO/CHECKPOINT.pt"

### Example Commands

EVALUATE A TIMM-TRAINED MODEL USING VLHUB

python src/training/main.py --batch-size=128 --workers=8 --imagenet-val "/imagenet/val/" --imagenet-v2 "/scratch/projects/hegdelab/bf996/datasets" --imagenet-s "/imagenet-sketch" --imagenet-a "/imagenet-a" --imagenet-r "/imagenet-r" --model="resnet50" --zeroshot-frequency=1 --linear-probe=True --image-size=224 --resume "/scratch/bf996/pytorch-image-models/output/train/20231223-105639-resnet50-176/model_best.pth.tar"

## VLHub

VLHub is an extension of OpenCLIP which adds support for a lot of things that are useful for the experiments we run.

### Uses

* Training VL models
* Training vision backbones
* Getting extended metrics, such as per-class accuracy and confusion matrices
* Evaluating VL and image-only models on on many different kinds of distribution shifts

### Properties

* After you load your singularity container and before you run a VLHub job, you have to run a setup command; you need to do this every time you re-launch your container, so you may want to add it to your env.sh launch command.

export PYTHONPATH="$PYTHONPATH:PATH/TO/VLHUB/src";

### Parameters

#### Inference

IMAGENET AND SHIFT EVALS
--imagenet-val "/imagenet/val/" 
--imagenet-v2 "/scratch/projects/hegdelab/bf996/datasets" --imagenet-s "/imagenet-sketch" 
--imagenet-a "/imagenet-a" 
--imagenet-r "/imagenet-r"
--objectnet "/objectnet-1.0/images"

IMAGENET-REAL (run separately)

--imagenet-real "/scratch/bf996/vlhub/metadata/imagenet1k_val.csv" --multiclass

DATACOMP EVAL (run separately)

--dc_eval --dc_eval_data_dir "/PATH/TO/datacomp_ds"

100-CLASS EVALS
--caption-subset="in100"
--caption-subset="in100-dogs"
--caption-subset="in100_random_01"

EXTENDED METRICS
--extended-metrics=True

#### Datasets

CSV
--train-data="~/Train_GCC-training_output.csv"       --csv-img-key filepath     --csv-caption-key title

WEBDATASET
--train-data '/vast/work/public/ml-datasets/laion400m/{7500..22500}.tar' --train-num-samples 50000000 --dataset-type webdataset

#### Debugging

--debug

#### Logging

Save results to weights and biases

--report-to wandb

#### Model Selection

Load a pretrained OpenCLIP model (VL)
--model=RN50 --pretrained=openai

Resume from a local checkpoint (VL)
--model=RN50 --resume "/scratch/bf996/open_clip/logs/yfcc_cars_01/checkpoints/epoch_32.pt"

Load a pretrained timm vision backbone
--model="resnetv2_50x1_bit.goog_in21k_ft_in1k" --linear-probe=True

Resume from a local checkpoint (vision backbone)

--integer-labels

### Example Commands

These commands presume that your terminal working directory is /PATH/TO/vlhub.

Inference on a pretrained OpenAI CLIP model

python src/training/main.py --batch-size=32 --workers=8 --imagenet-val "/imagenet/val/" --zeroshot-frequency=1  --model=RN50  --pretrained=openai

Inference on a pytorch-image-models checkpoint, with wandb logging and extended metrics

python src/training/main.py --batch-size=32 --workers=8 --imagenet-val "/imagenet/val/" --imagenet-v2 "/scratch/projects/hegdelab/bf996/datasets" --imagenet-s "/imagenet-sketch" --imagenet-a "/imagenet-a" --imagenet-r "/imagenet-r" --objectnet "/objectnet-1.0/images" --model="resnet50" --zeroshot-frequency=1 --linear-probe=True --image-size=224 --resume "/scratch/bf996/pytorch-image-models/output/train/20230926-101310-resnet50-224/model_best.pth.tar" --extended-metrics=True --report-to wandb