he[a]t AI - HackMD

# he[a]t AI This document supports **markdown** syntax! - Hi, I am Peter! - - Hi, Yue Yao here for test - Hi, I am Mert. ## setup on JUWELS - dump your environment modules `ml list` - for example ``` $ ml list Currently Loaded Modules: 1) Stages/2022 (S) 2) GCCcore/.11.2.0 (H) 3) zlib/.1.2.11 (H) 4) binutils/.2.37 (H) 5) StdEnv/2022 Where: S: Module is Sticky, requires --force to unload or purge H: Hidden Module ``` - to setup the correct cuda libs, please do: ``` $ ml load ``` ## setup on Conda on JEWELS - Work in the login node wget command does not work in the devbooster and we could not find a way to download it, but the command works in the login node. - Follow the instructions here https://docs.jupyter-jsc.fz-juelich.de/github/FZJ-JSC/jupyter-jsc-notebooks/blob/master/001-Jupyter/Create_JupyterKernel_conda.ipynb - Create your Conda environment with Python 3.8 By default, it creates Conda environment with Python 3.9 then it is hard to change once when you created - Download modules for your environment `/bin/bash/ ./setup.sh` `pip install -r requirements.txt` ## setup without conda - set up the environement - default is python 3.8 ``` $ ml Python/3.8.5 CUDA/11.3 ``` - what also works (but requires a `requirements.txt` without matplotlib): ``` $ ml Stages/2022 GCC/11.2.0 CUDA/11.5 $ ml Python/3.9.6 ``` - go to your repo ``` $ cd /path/to/repo $ git clone http://psteinb@github.com/mertkeser/detr_simple.git $ cd detr_simple ``` - set up `venv` folder: ``` $ python -m venv local-py38 $ source local-py38/bin/activate (local-py38) [steinbach1@jwlogin24 detr_simple]$ ``` - download requirements: ``` $ python -m pip install -U pip #important: update pip first, proved essential! $ python -m pip install --extra-index-url https://download.pytorch.org/whl/cu113 -r requirements.txt ``` - create folders for data: ``` $ mkdir -p data/sets/nuimage $ cd data/sets/ $ wget -N https://www.nuscenes.org/data/nuimages-v1.0-mini.tgz $ tar xf nuimages-v1.0-mini.tgz -C nuimage ``` ## setup without conda and not using torch from pip - set up the environement ``` $ ml PyTorch/1.11-CUDA-11.5 ``` - go to your repo ``` $ cd /path/to/repo $ git clone http://psteinb@github.com/mertkeser/detr_simple.git $ cd detr_simple ``` - set up `venv` folder: ``` $ python -m venv local-pyt38 $ source local-pyt38/bin/activate (local-pyt38) [steinbach1@jwlogin24 detr_simple]$ ``` - download requirements: ``` $ python -m pip install -U pip #important: update pip first, proved essential! $ python -m pip install --extra-index-url https://download.pytorch.org/whl/cu115 -r notorch-requirements.txt ``` the requirements file used is the following: ``` $ cat notorch-requirements.txt #matplotlib #numpy torchvision scikit-image nuscenes-devkit>=1.1.9 #pillow ``` ## devel booster GPU properties outputs of `deviceQuery`: `/p/software/juwelsbooster/stages/2020/software/CUDA/11.3/samples/1_Utilities/deviceQuery` ``` Device 0: "NVIDIA A100-SXM4-40GB" CUDA Driver Version / Runtime Version 11.4 / 11.3 CUDA Capability Major/Minor version number: 8.0 Total amount of global memory: 40536 MBytes (42505273344 bytes) (108) Multiprocessors, (064) CUDA Cores/MP: 6912 CUDA Cores GPU Max Clock rate: 1410 MHz (1.41 GHz) Memory Clock rate: 1215 Mhz Memory Bus Width: 5120-bit L2 Cache Size: 41943040 bytes Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384) Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 49152 bytes Total shared memory per multiprocessor: 167936 bytes Total number of registers available per block: 65536 Warp size: 32 Maximum number of threads per multiprocessor: 2048 Maximum number of threads per block: 1024 Max dimension size of a thread block (x,y,z): (1024, 1024, 64) Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535) Maximum memory pitch: 2147483647 bytes Texture alignment: 512 bytes Concurrent copy and kernel execution: Yes with 5 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Enabled Device supports Unified Addressing (UVA): Yes Device supports Managed Memory: Yes Device supports Compute Preemption: Yes Supports Cooperative Kernel Launch: Yes Supports MultiDevice Co-op Kernel Launch: Yes Device PCI Domain ID / Bus ID / location ID: 0 / 3 / 0 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > ``` # parallelisation - bare pytorch https://pytorch.org/tutorials/intermediate/ddp_tutorial.html - option: pytorch-lightning - https://william-falcon.medium.com/pytorch-lightning-vs-deepspeed-vs-fsdp-vs-ffcv-vs-e0d6b2a95719 - https://www.pytorchlightning.ai/ - https://pytorch-lightning.readthedocs.io/en/latest/common/production_inference.html?highlight=inference # Goals - Effect of parallelization on heat dissipation of GPU at Inference - Optimization of power consumption (use case : Edge computing and car PC) - Enhancement of DNN Operation - Training bottlenecks search and elimination # To-Do List 1. Integrate GPU for Training (Georgii) - [X] Should be able to select Gpu or Gpus - measure training CPU-only - port to GPU (mind the `.to(device)` clause) - measure training GPU-only - [X] Should save the trained model -> `torch.save` / `.pth` 2. Write a test.py (Mert) - [X] Should be able to select Gpu or Gpus - [X] Should be able to read trained model - GPU versus CPU model? - Able to read images in 30 FPS - baseline measurement: CPU-only inference, GPU-only inference - Visualisation of Results 3. Download Nuscenes (Yue Yao) - [X] - How to download full version of Nuscenes - *shared folder* for the entire team: `/p/scratch/training2203/heatai` > data are automatically deleted (files after 90 days by modification and access date, empty directories after 3 days) > taken from https://apps.fz-juelich.de/jsc/hps/juwels/filesystems.html?highlight=scratch - test if you can access this: `touch /p/scratch/training2203/heatai/hi-from-${USER}` - [X] - Search on training methods for Transformers (Epoch, batch size ?) - Original Settings according to https://arxiv.org/pdf/2005.12872.pdf - Short schedule (3 Days) - 300 epochs with a learning rate drop by a factor of 10 after 200 epochs - Batch Size 64 (4 images per GPU X 16 V100 GPUs) - Long schedule (Approximately 5 Days) - 500 epochs with a learning rate drop by a factor of 10 after 400 epochs - Batch Size 64 (4 images per GPU X 16 V100 GPUs) - With our setup (with CPU in Login Node): - The training with 1 GPU with whole "train" data (67280 images) takes: ca. 196min for 1 epoch - 3364 batches (20 images per batch) with avg training time 3.5s for one batch - Look into Dataset and come up with strategy for dataset validation of NuSense. - Based on how many images we are going to use. - Train (67280 Images), Val (16450 Images), Test (9760 Images without Annotation) - We can use train + val dataset in our project. Test data might not be useful since there is no annotation - consider stratification on `number of boxes x size of boxes x class label` - ~~According to the plot below, Train and Val data are quite balanced and without big bias.~~ - I made a mistake in the previous program and updated the plots below. - It seems the bounding box size (width, height, area) in train and val data is not similar for some classes such as trailer. (But we do have very few trailers in our train and val data) - For the most objects (barrier, car, pedestrain, traffic_cone, truck), their bounding box sizes are quite similar. - PS: let's be clear on wording here: You suggest that the plots confirm that there are no large shifts visible between training and val set. The balance of object classes appears similar. - ![](https://i.imgur.com/NlZEMzf.png) - ![](https://i.imgur.com/owG50cg.png) - ![](https://i.imgur.com/Pv5biWT.png) - ![](https://i.imgur.com/n0D2k4O.png) - PS: would be cool to see the same without normalizing the size. A measure like say `number of pixels in the box` might be directly applicable. - Done :) - ![](https://i.imgur.com/fsauGVf.png) - ![](https://i.imgur.com/UwGt3Bu.png) 4. How to save Heat or Energy Consumption of GPU (Kumar) - Check `nvidia-smi` and find out how to record measurements (temperature or total energy consumption) - look at a benchmark application that heats GPU -> take something from `which nvcc`/../samples/* - Should be able to gpu spesific or whole gpus - needs to be integrated into slurm script - ask nvidia dev techs how to do it? - Andreas Herten suggested job reports for any sbatch job longer than 5min - Markus from Nvidia suggested: `nvidia-smi dmon -s pucvmt -o T |& tee nvidia_smi_dmon.log` in order to log the power consumption and temperature bonus. parallelize training/inference - technology to use? - pytorch lightning - plain pytorch - anything else: `horovod` - how to measure speed-up? What is a good speed-up? - CPU only inference time measurement - Inference time with GPU only. (Optionally we can compare the impact of one GPU inference with multiple GPU at once inferece) # DETR? I talked to some folks at the Helmholtz Imaging Platform. Michael Baumgartner to be precise. He is the main author of nnDetection: https://github.com/MIC-DKFZ/nnDetection He shared the following view of DETR with me: ---- From some very early we are currently conducting and some Issues from Github, DETR has two "big-ish" problems: 1) Convergence Issues: the original paper used a pre-trained backbone network (ResNet). There were some Issues on Github which tried training DETR from scratch on COCO and failed. This might have been a problem with hyperparameters or similar but in general it seems quite difficult to train it from scratch even on big datasets. Since this Issue is potentially also associated with the next point, I don't think there was particular literature on this problem specifically. 2) Very Long Training Duration: As the original paper itself declares, they require a very long training schedule of 500 epochs on COCO. For comparison, the usual training schedule of Faster R-CNN on COCO is 12 Epochs, the "long" schedule is usually set to 36 epochs. Since this was quite a huge problem several works tried to solve this in various ways: Some of the earlier works: "Deformable DETR: Deformable Transformers for End-to-End Object Detection" https://arxiv.org/abs/2010.04159 uses custom module to speed up training and improve performance "Rethinking Transformer-based Set Prediction for Object Detection" https://arxiv.org/abs/2011.10881 Re-introduced some of the heuristics commonly used in Anchor based detectors to speed up the training. Somewhat recent papers which build on each other and use an "Anchor Like" design for the Queries inside the transformer part of DETR-> achieved some nice results and speed up during training: https://arxiv.org/abs/2203.03605 "DINO DETR" -> https://github.com/IDEACVR/DINO https://arxiv.org/abs/2201.12329 "DAB DETR" https://arxiv.org/abs/2203.01305 "DN-DETR" https://arxiv.org/abs/2109.07107 "Anchor DETR" https://arxiv.org/abs/2108.06152 "Conditional DETR for Fast Training Convergence" The most recent one "DINO" currently sets the SOTA on COCO. ---- # Re: DETR? The whole point was getting an experience on the training a network like a DETR and answer some real-world application questions. Therefore, we will have experience in the future use cases. The inference may be slowed by other processes like data loading. However, these bottlenecks are usually not detected in many cases. For the hackathon the size of the model was reduced: - E.g. the number of neurons in hidden layers reduced by half (quadratically reduces the time for self-attention calculation), - number of encoder/decoder blocks was also reduced, - resnet50 was replaced with resnet18 The convergence in this case will be quicker and the performance lower. During the hackathon the performance is only a pivot metric for us, which is only used as a reference point. Training the reduced model for, say, 100 epochs will be enough. # Nuscene Dataset - Based on how many images we are going to use. - Train (67280 Images), Val (16450 Images), Test (9760 Images without Annotation) - We can use train + val dataset in our project. Test data might not be useful since there is no annotation - consider stratification on `number of boxes x size of boxes x class label` - ~~According to the plot below, Train and Val data are quite balanced and without big bias.~~ - I made a mistake in the previous program and updated the plots below. - It seems the bounding box size (width, height, area) in train and val data is not similar for some classes such as trailer. (But we do have very few trailers in our train and val data) - For the most objects (barrier, car, pedestrain, traffic_cone, truck), their bounding box sizes are quite similar. - PS: let's be clear on wording here: You suggest that the plots confirm that there are no large shifts visible between training and val set. The balance of object classes appears similar. - ![](https://i.imgur.com/NlZEMzf.png) - ![](https://i.imgur.com/owG50cg.png) - ![](https://i.imgur.com/Pv5biWT.png) - ![](https://i.imgur.com/n0D2k4O.png) - PS: would be cool to see the same without normalizing the size. A measure like say `number of pixels in the box` might be directly applicable. - ![](https://i.imgur.com/fsauGVf.png) - ![](https://i.imgur.com/UwGt3Bu.png) # Day 1 ## Tasks PS: I suggest to make a note on who is working on what. - port code to GPU - find optimal `batch_size` - run also with 1024 an the processig time stays the same ( around 50ms) - this problem has been solved ( batch_size dataloader) - measure runtime for single-GPU mode(Georgii) - copy&paste ``` (0.050s) Batch [809/16819] loss: 2.3066082 (0.051s) Batch [810/16819] loss: 2.5847437 (0.049s) Batch [811/16819] loss: 2.237958 ``` - measure code runtime with multiple-CPUs (Yue) - measure runtime for multi-CPU mode - setup how to profile the code: (Yue Yao) - https://pytorch.org/tutorials/beginner/profiler.html - alternatively: use `nvprof` - would be cool to extend CLI arguments

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.