---
# System prepended metadata

title: 'he[a]t AI'

---

# he[a]t AI

This document supports **markdown** syntax! 

- Hi, I am Peter!
- 
- Hi, Yue Yao here for test
- Hi, I am Mert.


## setup on JUWELS

- dump your environment modules
`ml list`
    - for example
```
$ ml list

Currently Loaded Modules:
  1) Stages/2022 (S)   2) GCCcore/.11.2.0 (H)   3) zlib/.1.2.11 (H)   4) binutils/.2.37 (H)   5) StdEnv/2022

  Where:
   S:  Module is Sticky, requires --force to unload or purge
   H:             Hidden Module
```

- to setup the correct cuda libs, please do:
```
$ ml load 
```

## setup on Conda on JEWELS

- Work in the login node
wget command does not work in the devbooster and we could not find a way to download it, but the command works in the login node.

- Follow the instructions here
https://docs.jupyter-jsc.fz-juelich.de/github/FZJ-JSC/jupyter-jsc-notebooks/blob/master/001-Jupyter/Create_JupyterKernel_conda.ipynb

- Create your Conda environment with Python 3.8
By default, it creates Conda environment with Python 3.9 then it is hard to change once when you created

- Download modules for your environment
`/bin/bash/ ./setup.sh`
`pip install -r requirements.txt` 

## setup without conda

- set up the environement
    - default is python 3.8
    ```
    $ ml Python/3.8.5 CUDA/11.3
    ```
    - what also works (but requires a `requirements.txt` without matplotlib): 
    ```
    $ ml Stages/2022 GCC/11.2.0 CUDA/11.5
    $ ml Python/3.9.6
    ```

- go to your repo 
```
$ cd /path/to/repo
$ git clone http://psteinb@github.com/mertkeser/detr_simple.git
$ cd detr_simple
```

- set up `venv` folder:
```
$ python -m venv local-py38
$ source local-py38/bin/activate
(local-py38) [steinbach1@jwlogin24 detr_simple]$
```

- download requirements:
```
$ python -m pip install -U pip #important: update pip first, proved essential!
$ python -m pip install --extra-index-url https://download.pytorch.org/whl/cu113 -r requirements.txt
```

- create folders for data:
```
$ mkdir -p data/sets/nuimage
$ cd data/sets/
$ wget -N https://www.nuscenes.org/data/nuimages-v1.0-mini.tgz
$ tar xf nuimages-v1.0-mini.tgz -C nuimage
```

## setup without conda and not using torch from pip

- set up the environement
    ```
    $ ml PyTorch/1.11-CUDA-11.5
    ```
    
- go to your repo 
```
$ cd /path/to/repo
$ git clone http://psteinb@github.com/mertkeser/detr_simple.git
$ cd detr_simple
```

- set up `venv` folder:
```
$ python -m venv local-pyt38
$ source local-pyt38/bin/activate
(local-pyt38) [steinbach1@jwlogin24 detr_simple]$
```

- download requirements:
```
$ python -m pip install -U pip #important: update pip first, proved essential!
$ python -m pip install --extra-index-url https://download.pytorch.org/whl/cu115 -r notorch-requirements.txt
```

the requirements file used is the following:
```
$ cat notorch-requirements.txt
#matplotlib
#numpy
torchvision
scikit-image
nuscenes-devkit>=1.1.9
#pillow
```


## devel booster GPU properties

outputs of `deviceQuery`:
`/p/software/juwelsbooster/stages/2020/software/CUDA/11.3/samples/1_Utilities/deviceQuery`

```
Device 0: "NVIDIA A100-SXM4-40GB"
  CUDA Driver Version / Runtime Version          11.4 / 11.3
  CUDA Capability Major/Minor version number:    8.0
  Total amount of global memory:                 40536 MBytes (42505273344 bytes)
  (108) Multiprocessors, (064) CUDA Cores/MP:    6912 CUDA Cores
  GPU Max Clock rate:                            1410 MHz (1.41 GHz)
  Memory Clock rate:                             1215 Mhz
  Memory Bus Width:                              5120-bit
  L2 Cache Size:                                 41943040 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total shared memory per multiprocessor:        167936 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 5 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Managed Memory:                Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 3 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
```

# parallelisation

- bare pytorch https://pytorch.org/tutorials/intermediate/ddp_tutorial.html
- option: pytorch-lightning
    - https://william-falcon.medium.com/pytorch-lightning-vs-deepspeed-vs-fsdp-vs-ffcv-vs-e0d6b2a95719
    - https://www.pytorchlightning.ai/
    - https://pytorch-lightning.readthedocs.io/en/latest/common/production_inference.html?highlight=inference

# Goals

- Effect of parallelization on heat dissipation of GPU at Inference
- Optimization of power consumption (use case : Edge computing and car PC)
- Enhancement of DNN Operation
- Training bottlenecks search and elimination

# To-Do List

1. Integrate GPU for Training (Georgii)
- [X] Should be able to select Gpu or Gpus
    - measure training CPU-only
    - port to GPU (mind the `.to(device)` clause)
    - measure training GPU-only
- [X] Should save the trained model -> `torch.save` / `.pth`

2. Write a test.py (Mert)
- [X] Should be able to select Gpu or Gpus
- [X] Should be able to read trained model
    - GPU versus CPU model?
- Able to read images in 30 FPS
    - baseline measurement: CPU-only inference, GPU-only inference
- Visualisation of Results

3. Download Nuscenes (Yue Yao)
- [X] - How to download full version of Nuscenes 
    - *shared folder* for the entire team: `/p/scratch/training2203/heatai`
    > data are automatically deleted (files after 90 days by modification and access date, empty directories after 3 days)
    > taken from https://apps.fz-juelich.de/jsc/hps/juwels/filesystems.html?highlight=scratch
    - test if you can access this: `touch /p/scratch/training2203/heatai/hi-from-${USER}`
- [X] - Search on training methods for Transformers (Epoch, batch size ?)
    - Original Settings according to https://arxiv.org/pdf/2005.12872.pdf
        - Short schedule (3 Days)
            - 300 epochs with a learning rate drop by a factor of 10 after 200 epochs
            - Batch Size 64 (4 images per GPU X 16 V100 GPUs)
        - Long schedule (Approximately 5 Days)
            - 500 epochs with a learning rate drop by a factor of 10 after 400 epochs
            - Batch Size 64 (4 images per GPU X 16 V100 GPUs)
    - With our setup (with CPU in Login Node):
        - The training with 1 GPU with whole "train" data (67280 images) takes: ca. 196min for 1 epoch
        - 3364 batches (20 images per batch) with avg training time 3.5s for one batch
        
- Look into Dataset and come up with strategy for dataset validation of NuSense. 
    - Based on how many images we are going to use. 
        - Train (67280 Images), Val (16450 Images), Test (9760 Images without Annotation)
        - We can use train + val dataset in our project. Test data might not be useful since there is no annotation
    - consider stratification on `number of boxes x size of boxes x class label`
        - ~~According to the plot below, Train and Val data are quite balanced and without big bias.~~
        - I made a mistake in the previous program and updated the plots below.
        - It seems the bounding box size (width, height, area) in train and val data is not similar for some classes such as trailer. (But we do have very few trailers in our train and val data)
        - For the most objects (barrier, car, pedestrain, traffic_cone, truck), their bounding box sizes are quite similar.
            - PS: let's be clear on wording here: You suggest that the plots confirm that there are no large shifts visible between training and val set. The balance of object classes appears similar.
        - ![](https://i.imgur.com/NlZEMzf.png)
        - ![](https://i.imgur.com/owG50cg.png)
        - ![](https://i.imgur.com/Pv5biWT.png)
        - ![](https://i.imgur.com/n0D2k4O.png)

            - PS: would be cool to see the same without normalizing the size. A  measure like say `number of pixels in the box` might be directly applicable.
            - Done :)
        - ![](https://i.imgur.com/fsauGVf.png)
        - ![](https://i.imgur.com/UwGt3Bu.png)



4. How to save Heat or Energy Consumption of GPU (Kumar)
- Check `nvidia-smi` and find out how to record measurements (temperature or total energy consumption)
    - look at a benchmark application that heats GPU -> take something from `which nvcc`/../samples/*
- Should be able to gpu spesific or whole gpus
- needs to be integrated into slurm script
- ask nvidia dev techs how to do it?
    - Andreas Herten suggested job reports for any sbatch job longer than 5min
    - Markus from Nvidia suggested: `nvidia-smi dmon -s pucvmt -o T |& tee nvidia_smi_dmon.log` in order to log the power consumption and temperature
    
bonus. parallelize training/inference
- technology to use?
    - pytorch lightning
    - plain pytorch
    - anything else: `horovod`
- how to measure speed-up? What is a good speed-up?
    - CPU only inference time measurement
    - Inference time with GPU only. (Optionally we can compare the impact of one GPU inference with multiple GPU at once inferece)


# DETR?

I talked to some folks at the Helmholtz Imaging Platform. Michael Baumgartner to be precise. He is the main author of nnDetection: https://github.com/MIC-DKFZ/nnDetection

He shared the following view of DETR with me:

----

From some very early we are currently conducting and some Issues from Github, DETR has two "big-ish" problems: 

1) Convergence Issues: the original paper used a pre-trained backbone network (ResNet).  There were some Issues on Github which tried training DETR from scratch on COCO and failed. This might have been a problem with hyperparameters or similar but in general it seems quite difficult to train it from scratch even on big datasets. Since this Issue is potentially also associated with the next point, I don't think there was particular literature on this problem specifically.

2) Very Long Training Duration: As the original paper itself declares, they require a very long training schedule of 500 epochs on COCO. For comparison, the usual training schedule of Faster R-CNN on COCO is 12 Epochs, the "long" schedule is usually set to 36 epochs. Since this was quite a huge problem several works tried to solve this in various ways:

Some of the earlier works:
"Deformable DETR: Deformable Transformers for End-to-End Object Detection" https://arxiv.org/abs/2010.04159 uses custom module to speed up training and improve performance

"Rethinking Transformer-based Set Prediction for Object Detection" https://arxiv.org/abs/2011.10881
Re-introduced some of the heuristics commonly used in Anchor based detectors to speed up the training.

Somewhat recent papers which build on each other and use an "Anchor Like" design for the Queries inside the transformer part of DETR-> achieved some nice results and speed up during training:
https://arxiv.org/abs/2203.03605   "DINO DETR" -> https://github.com/IDEACVR/DINO
https://arxiv.org/abs/2201.12329   "DAB DETR"
https://arxiv.org/abs/2203.01305   "DN-DETR"
https://arxiv.org/abs/2109.07107   "Anchor DETR"
https://arxiv.org/abs/2108.06152   "Conditional DETR for Fast Training Convergence"

The most recent one "DINO" currently sets the SOTA on COCO.

----

# Re: DETR?

The whole point was getting an experience on the training a network like a DETR and answer some real-world application questions. Therefore, we will have experience in the future use cases. The inference may be slowed by other processes like data loading. However, these bottlenecks are usually not detected in many cases.

For the hackathon the size of the model was reduced:
 - E.g. the number of neurons in hidden layers reduced by half (quadratically reduces the time for self-attention calculation),
 - number of encoder/decoder blocks was also reduced,
 - resnet50 was replaced with resnet18

The convergence in this case will be quicker and the performance lower. During the hackathon the performance is only a pivot metric for us, which is only used as a reference point.

Training the reduced model for, say, 100 epochs will be enough.


# Nuscene Dataset

- Based on how many images we are going to use. 
    - Train (67280 Images), Val (16450 Images), Test (9760 Images without Annotation)
    - We can use train + val dataset in our project. Test data might not be useful since there is no annotation
- consider stratification on `number of boxes x size of boxes x class label`
        - ~~According to the plot below, Train and Val data are quite balanced and without big bias.~~
        - I made a mistake in the previous program and updated the plots below.
        - It seems the bounding box size (width, height, area) in train and val data is not similar for some classes such as trailer. (But we do have very few trailers in our train and val data)
        - For the most objects (barrier, car, pedestrain, traffic_cone, truck), their bounding box sizes are quite similar.
            - PS: let's be clear on wording here: You suggest that the plots confirm that there are no large shifts visible between training and val set. The balance of object classes appears similar. 
    - ![](https://i.imgur.com/NlZEMzf.png)
    - ![](https://i.imgur.com/owG50cg.png)
    - ![](https://i.imgur.com/Pv5biWT.png)
    - ![](https://i.imgur.com/n0D2k4O.png)
        - PS: would be cool to see the same without normalizing the size. A  measure like say `number of pixels in the box` might be directly applicable.
    - ![](https://i.imgur.com/fsauGVf.png)
    - ![](https://i.imgur.com/UwGt3Bu.png)

# Day 1

## Tasks

PS: I suggest to make a note on who is working on what.

- port code to GPU
    - find optimal `batch_size`
        - run also with 1024 an the processig time stays the same ( around 50ms) 
        - this problem has been solved ( batch_size dataloader)
    - measure runtime for single-GPU mode(Georgii)
        - copy&paste
        ```
        (0.050s) Batch [809/16819] loss: 2.3066082
        (0.051s) Batch [810/16819] loss: 2.5847437
        (0.049s) Batch [811/16819] loss: 2.237958
        ```
    
- measure code runtime with multiple-CPUs (Yue)
    - measure runtime for multi-CPU mode

- setup how to profile the code: (Yue Yao)
    - https://pytorch.org/tutorials/beginner/profiler.html
    - alternatively: use `nvprof`

- would be cool to extend CLI arguments