# Julia for HPC: CSC/ENCCS collab *Location* - Online (Zoom) *Schedule* - Onboarding: 2023-10-09 for 1-2 hours - Training: 2023-10-10 to 2023-10-13 - Sweden: 9:30-11:30 CEST, 12:30-14:30 CEST - Finland: 10:30-12:30 EEST, 13:30-15:30 EEST - October has lower probability for LUMI to be on maintainence break than September. Thanks for Heidi Reiman for figuring this out! *Lecturers and support* - Thor Wikfeldt (ENCCS) - Jaan Tollander de Balsch (CSC) - Yonglei Wang (ENCCS) - Zoom host: Thor, Yonglei *Participants* - Sweden: 20 - Finland: 20 - Outside: 10 *Tools* - LUMI : HPC cluster (Leonardo as backup) - LUMI OOD : Jupyter notebook for Julia for coding and demoing (if available) - GitHub : Training material - Zoom : Streaming video - HedgeDoc (https://siili.rahtiapp.fi/) : Collaborative note taking - Excalidraw (https://excalidraw.com/) : Collaborative whiteboard and scetching *Funding* - EURO CC TASK 2 *Training material and content* - https://enccs.github.io/julia-for-hpc/ - https://github.com/enccs/julia-for-hpc - https://docs.csc.fi/apps/julia/ - https://docs.csc.fi/support/tutorials/julia/ *Notes* - https://hackmd.io/@enccs/rk64gg3Zh - https://wiki.eduuni.fi/calendar/spacecalendar.action?spaceKey=CSCLUSTLO - https://www.csc.fi/asiakaskoulutus - https://enccs.se/events/10-2023-julia-for-hpc/ *Checklist* - [ ] Create registration form and add people the LUMI project (ENCCS) - [ ] Install and test software for the course material - [ ] Reserve a video conference room - [ ] Test programming exercises on LUMI - [ ] Test AV-tech and zoom for online presentation - [ ] Good microphone - [ ] Good webcam - [ ] Prepare backups if there are problem with LUMI - [ ] Test HedgeDoc - [ ] Populate HedgeDoc with standard info - [ ] Prepare post-workshop survey - [ ] Communicate with accepted participants - installation instructions and LUMI accounts - [ ] Explore whether we could use Leonardo as backup - are guest accounts now possible? ### Meeting 18 Sep 2023 Command Line (CL): ``` mkdir julia cd julia module use /appl/local/csc/modulefiles module load julia module load julia-amdgpu julia ``` Julia: ``` using MPI using AMDGPU MPI.install_mpiexecjl() ``` then: `exit()` CL: ``` touch Project.toml vi Project.toml ``` ``` pwd julia --project=/path/to/Project.toml or in my case julia --project=/users/andriiev/julia/Project.toml ``` Julia: ``` using Pkg Pkg.instantiate() ``` - explore using VSCode with remote connection - but also fine to just use an editor on LUMI and VSCode for all local work - Dagger.jl - reserves one core per node for scheduling - we need an episode on this! - who teaches what - day 1 - julia-intro - Special features: Jaan - syntax: Yonglei - developing in Julia: Thor - motivation: Thor - day 2 - julia-HPC - motivation: Thor - writing performant Julia: Yonglei - multithreading: Jaan - day 3 - distributed computing: Anastasiia - message passing: Thor - day 4 - running on a cluster: Jaan - GPU programming: Thor - summary: ? - new episodes: - dagger (day 3): Jaan - interfacing to C/Fortran/Python: Yonglei and Jaan - helpers / hedgeDoc answerers: - Thor all days - Yonglei all days - Jaan all days - we will record workshop ### Meeting 29 May 2023 Agenda - CSC LUST calendar: https://wiki.eduuni.fi/display/CSCLUSTLO/calendars - CSC training calendar: https://www.csc.fi/asiakaskoulutus - Thor didn't manage to get it to work on LUMI following Jaan's protocol - how to use rocBLAS, rocFFT etc? - finalising dates - 11-15/9 bad for ENCCS due to RISE event - 19-22nd possible? - 9:30-11:30 CEST, 12:30-14:30 CEST - 10:30-12:30 EEST, 13:30-15:30 EEST - **A LUMI service break is currently planned to the second half of August**. The schedule might still change, though (as nothing is certain as long as the hardware hasn't been shipped). - We should have a backup plan in case of LUMI is not available on the course dates. In Sweden there is a KTH machine (Dardel?) that is similar to LUMI, could we think of it as a backup? - possible to run interactively on compute node? - Thor has problems with https://github.com/Lumi-supercomputer/utility-tools - `srun --account=project_462000007 --partition=small --nodes=1 --cpus-per-task=1 --mem-per-cpu=1750 --time=00:15:00 --pty bash` - lesson material - `depends_on("PrgEnv-gnu-amd", "gcc/11", "rocm/5.2")` - `setenv("JULIA_AMDGPU_DISABLE_ARTIFACTS", "1")` - splitting out intro part - first 4 episodes go into "Introduction to Julia" - on day before the workshop we should have a LUMI onboarding session, 1-2 hours - practicalities - ENCCS runs registration, communications with participants etc - ENCCS and CSC have web pages - both do dissemination - 20 from SE, 20 FI, 10 outside - https://docs.lumi-supercomputer.eu/runjobs/scheduled-jobs/julia/ - consider using Jupyter if Open on Demand is ready in time - but could also use https://github.com/Lumi-supercomputer/utility-tools if we really want Jupyter - https://docs.lumi-supercomputer.eu/software/installing/easybuild/ --- ### Meeting 6 April 2023 Notes: - use Distributed.jl for multi-node - show one example at least - benchmark latency, compare with MPI.jl - use pingpong example? - inspiration from MPI benchmark suite for C? - is it possible to run these lessons in Jupyter Lab via Binder from the lesson website? - wrapper in Julia to set MPI backend - Preferences.jl - MPIPreferences.jl - we use LUMI during the workshop - test GPU-aware MPI! include in lesson - AMDGPU.jl has test suite, run it - thoroughly check AMDGPU.jl examples - from AMDGPU.jl README: *Julia 1.7, 1.9, and above. Julia 1.8 is not supported and should not be used with AMDGPU.jl, as Julia 1.8 cannot properly handle the code generated by AMDGPU.jl.* d settings are in settings.sh Here you can set the timeouts, toggle if ssh to compute nodes is enabled and change the nam - Anastasiia to test material on LUMI - need to automate adding users via puhuri - should lessons be converted to jupyter notebooks? - many advantages to VS-Code with Julia extension, so don't drop it completely - creating programs in VS is advanced in comparison to run Jupyter notebook in prepared environment, thus .ipynb provides inclusivity and increasing community. - CSC to consider getting involved in Julia-for-HPDA lesson development - installing Julia yourself on cluster can have issues: e.g. Plots.jl installs ~10k files - mention this in the lesson and point to work-arounds - centralised location vs other - JULIA_DEPOT_PATH - add to Profiling section, perhaps include section after parallelisation episodes to show how to profile parallel code - include episode to interfacing to C/C++, Fortran, Python (issue #9) - e.g. matplotlib example - perhaps as an extra episode - have core material, for new users - have a route through material for more experienced users, skipping intro episodes - figure out if ClusterManagers.jl works with multi-node - possibly some issue with inter-node communication - when workshop? - September, maybe ~~4-8th~~ or 11-15th? - CodeRefinery 19-21 and 26-29 - BioExcel 11-15th, but maybe not much overlap (summer school) - 9:30-11:30 CEST, 12:30-14:30 CEST - 10:30-12:30 EEST, 13:30-15:30 EEST - 20 participants from SE, 20 from FI - who teaches? - Thor, Jaan, maybe others - helpers from CSC and ENCCS, and NAISS? - make it into a CASTIEL Twinning/Collaboration - event pages on both ENCCS and CSC - registration via prace-indico - use high max-number to balance number of participants from SE and FI - can we charge for it?? - money can not go into NCC project - we keep workshop free - use https://hedgedoc.org/ for workshop, hosted by CSC Rahti - https://siili.rahtiapp.fi/ - GitHub: @jaantollander
Last changed by  
 
ENCCS
EuroCC National Competence Center Sweden

Read more

RISE bootcamp planning

Event description: https://hackmd.io/@enccs/RISE-software-bootcamp

9/20/2023
Performance programming training material

planning Karl-Filip Faxen The workshop is interactive in the sense that students read text, write programs, run and measure programs and so on. We assume Linux, since that is what almost all supercomputers run. C is the primary language with as much material as possible duplicated in Fortran. I will add Fortran code when the C code has come along a bit since that will be important to set the structure. There is an issue with what machines to use. If the students run on their local machines as the workshop is given, the videoconferencing software (Teams, Zoom,...) will affect measurements quite dramatically. We will use a minimum of tools, with a focus on tools that are readily available in Linux like the bash command time as well as the perf tool.

9/20/2023
ENCCS - Python for High Performance Data Analytics

September 28 - 29, 2023 General information This hackMD: https://hackmd.io/@enccs/instructor-training-sept2023 Backup hedgeDoc: https://johank-enccs-hedgedoc.icedc.se/GAzvF4MJSm2AAVVX1i6KxQ Training material: https://enccs.github.io/instructor-training/ Direct link: https://rise.zoom.us/j/69209747774?pwd=bVhiWXJTT25ocHI0NFRZQkVTOVJTZz09 Workshop feedback form: TBA Follow ENCCS Upcoming events: https://enccs.se/events/

9/20/2023
Julia for HPDA - lesson development

General introduction to Julia, packages, syntax etc…

9/15/2023
Read more from ENCCS
Published on HackMD