# Best Practices for HPC @ VSC User Day

## Meta

### Sync meeting Alex + Kenneth

- Fri 13 Oct 10:00 CEST
- goal: slide deck draft ready, to be sent to VSC on feedback
- Thu 19 Oct 10:00 CEST
- have slide deck basically ready to go
- Mon 23 Oct: last changes to slides

### 2023-10-13

- contact info change
- https://docs.vscentrum.be/contact_vsc.html => https://docs.vscentrum.be/contact
- should also mention [e-mail] contacts for Tier-1
- https://docs.vscentrum.be/user_support.html should point to https://docs.vscentrum.be/contact

### Slides

https://docs.google.com/presentation/d/1L3zYaWLv_b0VOXdOcnlBwo6aXlhoitF_tpq5gWWWIoM

## Documentation (Alex)

Working in the HPC is not same as with a personal computer. Knowing the systems in place will make your life much easier.

* read the documentation
- give URL of VSC docs
- show revamped VSC docs
- ask KUL and UA about the URL to their docs
* follow our trainings
- give URL to VSC page about trainings
- some sessions are sort of specific to a particular VSC sites
- mainly for HPC intro sessions
- https://www.vscentrum.be/vsctraining
- ask GJ for link to EuroCC trainings

## Help us help you (Kenneth)

Things will go wrong, that's normal. All VSC clusters have dedicated user support teams that can help you.

* support is specific per cluster/service
- Tier-2 or VSC account trouble => local site
- Tier-1 compute => [e-mail]
- Tier-1 cloud => [e-mail]
- see docs

* understanding and troubleshooting issues takes time
- be polite & patient
- if something is urgent for you, make that clear and provide deadlines
- limit your support tickets to a single issue
- if your request gets stalled, don't hesitate to ping for an update
- priority: global problems vs very specific to own use case

* how to open a good support request: the more information you give in your request, the faster it will be tackled
* connection issues:
* where are you connecting from
* how do you connect to the internet: wifi, vpn?
* what OS and client software (PuTTy, MobaXterm, SSH) do you use
* use troubleshooting checklist
* UGent: https://docs.hpc.ugent.be/troubleshooting/#sec:connecting-issues
* VUB: https://hpc.vub.be/docs/faq/troubleshooting/#issues-connecting-to-hydra
* VSC docs: ... (under https://docs.vscentrum.be/faq.html ?)
* job/software issues:
* your VSC ID
* ID of your job
* path to job script
* path to output of job
* any error messages in the output of the job
* providing a small script to reproduce the issue is vey helpful
* avoid screenshots (except fot issues on web portals)
- example of good & bad support request

- Tips to google error messages
- READ the error messages! (don't panic)
- cfr. HHGTTG
- use a good search query
- use search operators: "..." and - (negate)
- check the date of any answer found
- be careful with ChatGPT...

## Workflows on the HPC (Alex)

Designing your workflow on the HPC is critical. Think in advance about the type of simulations you need to execute and plan how will you carry out those on the cluster.

Clusters can be busy and in such cases your jobs will wait in queue, do not wait until to run your jobs. Murphy's Law of HPC: the cluster is always full the day before your deadlines.

* Parallel computational jobs in batches

* stays the main target of the HPC platform
* simulations should run beginnning to end autonomously, from a non-interactive script (aka job)
* jobs will be scheduled and executed at a future time
* suitable for single big parallel simulation or a collection of independent non-parallel tasks
* **not** suitable for interactivity or graphical visualization
* distributed on the cluster (aka job arrays, task farming)
* main interface for this type of workflow is the Linux terminal (interactive terminal and job scripts)
* steep learning curve but most powerful interface
* good for automatization of tasks
* learning the shell scripting always pays off

* Task farming: worker, GNU parallel

* Interactive workflows

- most clusters provide portals designed for interactive workflows (JupyterHub, R Studio, Remote desktop with VNC)
- Tier-1 Hortense (OoD)
- HPC-UGent Tier-2 (OoD)
- Tier-2 Leuven (OoD)
- Tier-2 Brussels (JupyterHub)
- not yet for Tier-2 UAntwerp
- access without needing any terminal client (or SSH)
- suitable for data analysis and visualization, simulations requiring user interaction, code development
- **not** suitable for long simulations, sessions are limited to a day of work (check if this is true everywhere)
- coordinate with MT on this w.r.t. preceding session
-
* Test/Debug workflows

- VSC clusters have in most cases a debug queue, or partition, or cluster for quickly prototyping and test job scripts
- KUL: max. 2 nodes for 30min
- UGent: debug/interactive cluster (max. 8 cores, max 3 jobs running, 5 queued, oversubscribed)
- UA: auto-routing of small jobs to dedicated nodes
- VUB: testing phase of cluster with oversubscription

## Tips and tricks of the Bash shell (Kenneth)

* shell capabilities
- tab completion
- Ctrl-R and `history`
- ls options (ls -lrt, colors, ...)
- search with `grep` and `rg` (if available)
- pipelines (example with ls, grep, sed, cut, xargs)
- screen & tmux

* make your own commands
- define aliases as shorthands for commonly used commands (less typos)
- add custom functions to your `.bashrc`
- add custom scripts to `~/bin`

* software modules
- ml vs module command
- module spider

* see Linux training sessions

* docs:
* https://docs.vscentrum.be/jobs/basic_linux_usage.html (NEEDS UPDATE)
* https://docs.vscentrum.be/jobs/how_to_get_started_with_shell_scripts.html (FIX QUOTES)
* https://docs.hpc.ugent.be/linux-tutorial/

## Organising your data in the system (Alex)

Not all directories in the HPC clusters are equal. There are different storage systems with different capabilities

* VSC_HOME: very small, don't use it
* VSC_DATA: shared accross sites, place to keep important data but it is _slowish_
* VSC_SCRATCH: very fast storage, jobs should use the scratch as much as possible

The storage is not all powerful, there are limits, beware of them:
- in general: I/O is slow, impacts all jobs/users
- show how to check quota on the storage
- number of files matter:
- be careful with lots of small files
- organize large numbers of files in subdirectories
- rule of thumb: 1000s of files per directory (not a million)
- create tarballs of directories with large number of files
- alternatives: HDF5, ...
- move data used in jobs to scratch on-the-fly
- cfr. VUB docs (should be copied to VSC docs?)
- Globus for quickly copying files to other systems (better than scp, rsync, etc.)

## Managing your software stack (Kenneth)

As general rule, software performs best in the cluster if it is installed by the support teams. We know the optimizations that are needed for the different hardware platforms available in the cluster.

* Requesting software installation centrally:

- Software that can be optimized on installation by VSC support teams:
- all compiled software (C, C++, Fortran, Rust)
- software written in some interpreted languages (Python, Julia) that bundle compiled software libraries in (C, C++, Fortran, Rust)
- software relying on MPI
- Software that cannot be optimized on installation by VSC support teams:
- software purely written in interpreted languages (R, Python, Java)

In cases where optimization is negligible or for software under active development, users are encouraged to install any needed packages on their own

- Installing (pure) Python packages on top of centrally installed software (Python venv example)
- on top of TensorFlow or PyTorch available via modules
- makes sense for "pure" Python packages (only Python code)
- cfr. VUB docs:
- https://hpc.vub.be/docs/software/additional_software/#installing-additional-python-packages
- https://hpc.vub.be/docs/software/additional_software/#python-virtual-environments
- cfr. VSC docs
- https://docs.vscentrum.be/software/python_package_management.html

- Also cover self-installation of R packages
- cfr. VUB docs:
- https://hpc.vub.be/docs/software/additional_software/#installing-additional-r-packages
- cfr. VSC docs:
- https://docs.vscentrum.be/software/r_package_management.html

- Compiling from source code
- be careful where you compile, caveats with compiler options like `-march=native`
- $VSC_ARCH_LOCAL, $VSC_OS_LOCAL
- cfr. VUB docs:
- https://hpc.vub.be/docs/software/additional_software/#compiling-and-testing-your-software-on-the-hpc

- docs
- https://docs.hpc.ugent.be/easybuild => move to VSC docs?

## Don’t waste compute resources! Start small, scale up. (Alex)

Determining in advance the optimal amount of resources (CPU cores, memory, time) needed by your jobs is not trivial. Do not be afraid to run test jobs to establish the best combination of resources for your job. Using the right amount will result in less queue time and faster execution time.

- first test on small scale and then go bigger
- single job before launching a large array job
- smaller input with fewer resources and shorter walltime
- check scaling behaviour: stop adding more cores/nodes if you don't see significant speedups

- only ask for more resources if your job will actually use them
- adding more cores doesn't automagically make programs run faster, s sequential software will only use 1 core and multiproces software might need specific options to use more core
- adding more nodes will not make any difference if your program unless it supports technology such as MPI
- adding GPUs doesn't make software software that can not use GPUs faster
- adding more memory will not make any difference unless your job needs it (i.e. it fails due to OOM errors)

- check on job effiency post-hoc (decrease in walltime vs total # cores, avg CPU usage, memory usage, ...)
- slurm_jobinfo (from https://github.com/hpcugent/vsc-jobs, https://github.com/rug-cit-hpc/slurm-jobinfo)
- sacct (incl. with Slurm)
- seff output in job email notification (@ UA)
- Show/demo XDMoD to get overview of usage?

- Avoid oversubscribing available cores (the N^2 threads issue)
- this problem is getting worse as there are more cores per node
- both R and OpenBLAS starting one thread per core...
- $OMP_NUM_THREADS, $OPENBLAS...
- in your code (thread pool, multiprocess in Python code)
- cfr. VUB docs
- give example of dramatic speedup
- how can a user monitor this?
- check load when job is running (when using full nodes)
- load should be ~equal to # cores
- Likwid? perf? pstree and "ps xfu", eyeballing via top or htop
- high system CPU time (> 50%) due to context switching
- CPU time should be close to # cores x walltime
- VUB docs:
- https://hpc.vub.be/docs/faq/troubleshooting/#my-jobs-run-slower-than-expected-what-can-i-do
- https://hpc.vub.be/docs/faq/advanced/#how-to-run-python-in-parallel
- https://hpc.vub.be/docs/faq/advanced/#how-can-i-run-r-in-parallel (maybe not relevant)
- https://hpc.vub.be/docs/software/usecases/#pytorch

- How can I know job (memory) requirements?
- rely on default to start with (amount of memory is proportional to # requested cores)
- trial and error, increase (double) memory request on each attempt, scale back down if it makes sense
- can use sacct to check consumed memory
- control # cores used in job script
- differences across sites
- cfr. VUB docs:
- https://hpc.vub.be/docs/faq/basic/#how-can-i-check-my-resource-usage
- https://hpc.vub.be/docs/faq/troubleshooting/#how-can-i-monitor-the-status-of-my-jobs
- https://hpc.vub.be/docs/faq/troubleshooting/#why-is-my-job-not-starting

## Tricks to get your jobs started faster (Kenneth)

- consider all available resources (multiple clusters per site, consider other VSC sites)
- if possible fill gaps (use a quarter node in terms of cores is a good one I always use)
- faster runtime vs shorter queue time => total turnaround time
- submitting independent tasks
- try to distribute them (worker, etc.) and fill gaps
- avoid submitting many tiny jobs, pack them together in a larger jobs
- balancing act
- job runtime
- single-digit minutes is too short, aim for closer to 1h at minimum
- job count
- up to couple of thousand is OK, over 10k is too much
- single user policy @ UA on some clusters (no single user policy at UGent, KUL, VUB)

## Extra topics (optional)

- SSH config file
- agent forwarding, jump hosts, X11 forwarding, ...