# GPU benchmarking

We are looking to buy some new GPU nodes for the Argus Kubernetes cluster. Chris asked us to run some checks to find the cheapest type of GPU which is viable for running our desired program. We want to run AreTomo on these nodes because then we could scale the tomo_align service; at the moment we can only scale using the tomo_align_iris service, which doesn't have as high guarantees as Argus (and no SLA).

AreTomo licensing was mentioned as a possible issue but I think this should be fine.
Test A100 V100s - how to test in container?
Mihai said to temporarily install Docker on Hopper - containerd, singularity is daemonless.
Will it make a difference running in a Singularity container vs running directly on the node or in a Docker container? Chris says every system call is marshalled by the container process, so it could make a difference. Mihai seems to think it shouldn't make a difference. Run everything through Singularity containers.
Take one node out to test - Hamilton?
Test V100 on Argus
Test A100 on Hopper/Hamilton
Test other, lower spec GPUs on Pollux
Could we have access to RTX cards? - Mihai talk with Di - I don't think we need these, just use Pollux's GPUs
Make sure to test GPU calulation not disc, put files into memory first.
Put the test dataset in /dev/shm in memory filesystem.
Test the time-only execution of GPU part of code

Steps:
1. Build AreTomo Singularity image.
2. Run jobs with Singularity on Hopper/Hamilton.
3. Then use RAM disk and see if that makes a difference
4. Then test on other types of GPU (Argus and Pollux)

To build the Singularity container:
`export SINGULARITY_CACHEDIR=/tmp`

`singularity pull docker://gcr.io/diamond-pubreg/em/tomo_align:latest`

`singularity exec --nv /dls/ebic/data/staff-scratch/AnnaH/tomo_align_latest.sif AreTomo`

To get access to filesystem locations add:
--home ${PWD}
or
-B mount/path/here:mount/path/container for more specific locations

The AreTomo manual says:
"Since the current implementation has not optimized yet for GPU, any GPU card that is as good as GTX 1080 card or better is sufficient",
we expect the Pollux nodes to be enough.