Julia in HPC (BoF) -- JuliaCon 2021
===
[](https://hackmd.io/MUu0ri0aQOu8FwKZBKvCEg)

###### tags: `BoF` `JuliaCon` `HPC`

:::info
- **Date:** July 29th, 2021 4:30 PM (UTC)
- **Agenda**
0. User poll
- Vote here: https://tinyurl.com/julia-hpc-voting

1. Brief presentations
- ClimateMachine
> [name=Simon Byrne]
- CESMIX (cesmix.mit.edu)
> [name=Valentin Churavy]
- GPU4GEO ([pasc-ch.org](https://www.pasc-ch.org/projects/2021-2024/gpu4geo/)) & Julia at ETH Zurich
> [name=Ludovic Raess/Sam Omlin]
- Julia+HPC @ NERSC
> [name=Johannes Blaschke]
2. Roundtable discussion
- Julia in the DOE ([US Department of Energy]([energy.gov](https://www.energy.gov/)))
- [MPI.jl](https://github.com/JuliaParallel/MPI.jl)
- Challenges of running Julia at scale
- Deploying Julia (Sysimages/Pkg/Depots/Artifacts/...)
- Teaching HPC
- Julia on PowerPC
- ???
- **Participants:**
- Valentin Churavy, MIT, JuliaLab (VC)
- Carsten Bauer (CB)
- Ludovic Räss (LR)
- Samuel Omlin (SO)
- Michael Schlottke-Lakemper (MSL)
- Jane Herriman (JH)
- Michael Herbst (RWTH Aachen University)
- Tuomas Koskela, UCL (TK)
- Lucas Wilcox (LCW)
- Erik Schnetter (Perimeter)
- Mosè Giordano
- Julian Samaroo
- William F Godoy (ORNL)
- Johannes Blaschke (LBNL/NERSC)
- (Add yourself...)
:::

Feel free to edit this document, add your name above, etc...

## Presentations (Notes)

### CliMA

https://clima.caltech.edu

- A collaboration (Caltech, MIT, NPS, JPL) building a climate model in Julia
- Targeting multiple architectures (CPU & GPU), with distributed parallelism (MPI)
- Main development repositories: https://github.com/CliMA/
- ClimateMachine.jl, Oceananigans.jl, ClimaCore.jl, RRTMGP.jl + many more
- https://github.com/CliMA/slurm-buildkite (for running CI on your cluster)
- Significant contributions to
- MPI.jl
- KernelAbstractions.jl

#### Significant challenges

- Race conditions in Pkg.jl
- Downloading packages, Precompilation, Downloading Artifacts
- Both within job (e.g. MPI processes), and between jobs (e.g. multiple jobs instantiating at the same time)
- Parallel I/O
- Printing and logging with distributed programs
- MPI.jl annoyances
- Combining non-blocking MPI and Julia tasks
- Binary dependencies (can we make use of wi4mpi?)

#### Some ideas

- Interactive parallelism: https://github.com/simonbyrne/SPMD.jl
- Use Distributed.jl as an orchestration mechanism for SPMD programming
- Inspired by mpi4py + IPython
- How to define a standard Distributed Object API?
- Usability/getting started
- Distributed docs are a bit of a mess
- https://juliaparallel.github.io/ is a 404

### CESMIX
1. cesmix.mit.edu
2. Multi-year DOE founded project (PSAAP-III)
3. Multi-scale material science
4. Julia in the loop for UQ and AD
- Enzyme
6. Working on better integration for
- LLNL/flux
- LLNL/spindle
- Spack
- WI4MPI/Switchable MPI
- JuliaParallel/UCX.jl
7. https://github.com/cesmix-mit
- Interface for LAMMPS
- much more...

### GPU4GEO & Julia at ETH Zurich

#### GPU4GEO
1. GPU4GEO https://www.pasc-ch.org/projects/2021-2024/gpu4geo/
- Frontiers GPU solvers for ice dynamics and geodynamic applications
- 3 years project with ETHZ (VAW, GFD, CSCS), JGU Mainz, MIT JuliaLab
2. Objectives (_LR_)
- Develop high-performance multi-GPU building blocks
- Develop extensible solvers to build on these blocks
- Develop 2 production ready multi-GPU supercomputing applications
- Run on CSCS' ["ALPS"](https://www.cscs.ch/computers/alps/) (Nvidia GPUs + ARM CPUs)
- Run on EU (Finland) ["LUMI"](https://www.lumi-supercomputer.eu/lumi_supercomputer/) (AMD Epyc CPUs + Instinct GPUs)
3. Portable high-performance, massive scaling (_SO_)
- [AMDGPU.jl](https://github.com/JuliaGPU/AMDGPU.jl) and [KernelAbstractions.jl](https://github.com/JuliaGPU/KernelAbstractions.jl) backends in [ParallelStencil.jl](https://github.com/omlins/ParallelStencil.jl)
- GPU RDMA and communication hiding with [ImplicitGlobalGrid.jl](https://github.com/eth-cscs/ImplicitGlobalGrid.jl) and ParallelStencil.jl
- Parallel I/O, on-the-fly data compression and in-situ visualization with [ADIOS2](https://adios2.readthedocs.io) (c.f. [concise ADIOS2 tutorial](https://github.com/omlins/adios2-tutorial) by SO)

#### Julia teaching at ETH Zurich
1. Teaching in ETH Master programm (LR, SO & Mauro Werder)
- [Solving Partial Differential Equations in parallel on GPUs](http://www.vvz.ethz.ch/Vorlesungsverzeichnis/lerneinheit.view?semkez=2021W&ansicht=KATALOGDATEN&lerneinheitId=155538&lang=en)
2. Julia GPU course at [CSCS](https://user.cscs.ch/tools/interactive/julia/) - Swiss National Supercomputing Centre (Tim Besard & SO)

## General Notes

:::info
Monthly HPC Call, on the last Tuesday of the Month!

See the Julia Community calendar https://julialang.org/community/#events
:::

## Resources

### Training

- https://github.com/ornl-training/julia-basics
- https://github.com/luraess/geo-hpc-course

### Deployment

- https://docs.nersc.gov/development/languages/julia/
- https://github.com/JBlaschke/NERSCJuliaInstall
- https://github.com/easybuilders/CSCS/blob/master/easybuild/easyblocks/julia.py