Try   HackMD

Archive of LUMI Coffee Break 2022

This is the archive document of the sessions of 2022 of the public LUMI coffee break.
The archive document for 2023 can be found here: https://hackmd.io/@lust/coffeearchive

Link to document for next coffee break: https://md.sigma2.no/lust-coffee-break

Past sessions:

2022-04-06 13:00 – 14:00 CEST

  • Can you use easyconfigs from official easybuild github repository?

    • We use different toolchains and often also other parameters, so most unmodified ones don't work.
    • Often only few modifications needed but needs some experience with writing easyconfigs
    • Easyconfigs prepared by LUST
      • these are available on the system via eb -S (after loading LUMI/21.12 and EasyBuild-user/LUMI)

  • How much of the LUMI software stack is stable?

    • Goal is not to have complete software stack but offer simple way of installing software yourself
    • Often also not possible because of license issues (especially for commercial ones)
    • LUMI/LUST doesn't have resources to maintain a big software stack
    • Users are expected to be able to use easybuild or compile their own software
    • If you need help either your local organisation or LUST can help

  • CP2K using OpenMP: Is it possible to run it on GPUs?

    • Not yet. We only have early access platform ready. GPU partition is not yet installed.
    • Will be one of the first software being made available to install on LUMI-G when it's available
    • Note that there is no Open MPI on LUMI, only MPICH. Given the specific interconnect of LUMI, we use Cray MPICH as that is the only MPI implementation for which we can really guarantee support and a level of performance.

  • Job dependencies

  • Why low quota on number of files in project folder?

    • To guarantee good file system performance as Lustre FS is susceptible to slow down if too many files are accessed
    • It is possible to raise the quota for special projects (mostly development work) but we don't do that often. We are more likely to increase size quota than inode quota(number of files)
    • Especially for python/conda workflows, we provide a beta version tool to wrap the environment in a container. See our doc.
    • For compiling codes: In most build procedures it is possible to have sources and object files on scratch while doing the final install on /project (and then clean up scratch). This is actually also what we do in all but one of our EasyBuild builds (with the difference that then we even use a RAM disk, which may be needed for some packages as building on Lustre/GPFS/BeeGFS fails for some packages).

2022-05-25 13:00 – 13:45 CEST

Updates:

  • LUMI news
    • Installation of the LUMI-G cabinets
    • Integration of LUMI-C and LUMI-G in June
    • 2nd phase pilot projects to start at the end of the summer for ~1 month
    • LUMI-G available for users after the pilot phase ends
  • Training
    • LUMI-C
      • CoE+LUST training for LUMI-C users 27-28.04. Additional training planned before the end of the year
    • LUMI-G
      • CoE training for pilot users + LUST
      • CoE+LUST training for LUMI users
  • Software environment
    • CSC to start software installation e.g. Amber will be made available to all academic users
    • Country-local HPC support organizations have the possibility to manage their own software installations in a separate directory in /appl/local.

Questions

  • Are the materials from the CoE+LUST training for LUMI-C users 27-28.04 freely accessible someplace? Where?

    • Not yet and probably not all recordings. But we are working on uploading the slides nad some of the recordings.

  • How are the resources billed (core hours)? Is there a difference between normal and largemem

    • Currently only billed on core hours. So there is no difference between normal and large memory nodes, but it might change in the future. As a resource allocator, you have to be careful how much you allocate in terms of core hours, because we have so few large memory nodes.
    • Plan: 1 coreh per 2GiB but not enforced at the moment.
      • Would that be additionally to the usual coreh or just if you ask for too much memory with respect the number of cores?
      • No, only if you request more memory. corehours = max(ncore, ceil(mem/2GB)) x time

  • How can you check spend core hours/storage quota?

  • How to get access to the early access platform to test GPUs?

    • Every project has access. No additional rights or allocations required.
    • You just need --partition=eap and something like --gpus=1
    • Also see the docs

2022-06-29 13:00 – 13:45 CEST

Updates:

  • LUMI news
    • Users are likely to be able to log in after 12.07.2022
    • Shortly thereafter there will be another update that should be mostly a rolling update (meaning it is done while the system is live) but at some point a reboot and possible some more off-line work will be needed. It is currently scheduled that LUMI will be available to users for at least 2 weeks.
    • However, LUMI will not always run at full capacity during those two weeks. The coolant needs to be flushed which is done by half system for LUMI-C.
      • LUST will ask CSC sysadmins what the consequences are for the availability of the small and standard partition, as half the system down may mean that the small partition becomes totally unavailable.
    • Once LUMI is available again, hopefully one of the following pages will be updated : https://www.lumi-supercomputer.eu/service-breaks/ , https://www.lumi-supercomputer.eu/service-breaks/lumi-maintenance-break-starting-on-monday-the-6th-of-june/

Questions

  • I don't see TensorFlow nor PyTorch under the contrib EasyConfig repository, will these be created and supported by LUST?

    • It is not clear how this will be approached, but these are high in prioroty to be made available. No ETA, though.
    • AMD containers should work within a node but it is not clear yet how to deal with distributed ML.

  • What will be the preferred way to run distributed ML? E.g. will LUST support Horovod or recommend distributed PyTorch?

    • AMD distributed sw will not work at all or work very inefficiently (TCP fallback in the communication) as they build for a different network architecture with different network libraries that do not exist yet for LUMI.

  • Will OpenCL work?

    • It should work, in principle. But AMD seems to be emphasizing it less these days, and we hear from some sources that the implementation is not that good.

  • Where does LUST deliniate between local country support and LUST when it comes to creating/supporting software packages that are needed to compile and build other software? E.g. for hipSYCL should local support be expected to create an EasyBuild/install or is that the "responsibility" of LUST?

    • hipSYCL is on the wish list, but no specific ETA is expected.
    • SYCL will also be needed for future versions of GROMACS. That alone nakes it important to get it on LUMI.

  • Do you have a priority list of work for LUST?

    • Not one that we can publish, and it would not tell much about ETA of project results as the problem is that at this moment any scheduling of time is impossible as the system is simply not in a stable state. For some things that we consider a high priority we also depend on other parties delivering something first, so even if it may have a high priority we may still be unable to work on it. And there is always the unpredictable load of user tickets. But of course basically the more users (in terms of number of projects etc.) are impacted by something, the more likely it will get a higher priority. Also, the more system resources get lost because of something not being available, the more likely it will get a higher priority. So a very scalable GPU-based software package will get a higher priority than, e.g., a software package that can only use a single core of a regular CPU.
      • For local support sites, will using the contrib EasyConfig repository to collaborate be an acceptable solution in the short run? Both to try to contribute new EasyConfigs, but also to bring software related issues?https://tinyurl.com/lumicoffee

2022-07-27 13:00 – 13:45 CEST

No notes or question taken

2022-08-31 13:00 – 13:45 CEST

  1. When will LUMI-G start scheduling jobs again?

    • We don't know yet. LUMI-G is officaily still owned by HPE and they will prepare for the acceptance tests of the lumi phase-2 (gpu partition). Because the pilot period has not officially started and the GPU partition is HPEs property currently, there's no commitment for LUMI users to have access to it. The access was given just for letting the pilot projects, lust, etc. to prepare for the actual pilot period and also for avoiding the nodes sitting idle during this in-between period.

  2. What about just the eap?

    • It is part of LUMI-G

  3. I want to install a code (Elmer) on LUMI-C which has either a working singularity image or a Spack recipe (and a lot of dependencies; NetCDF, Mumps, Hypre, ParMMG, …) - which way to go? We tried to compile it with Cray compilers - no way.

    • Did you try to use Spack?
      • There is a spack recipe for Mahti/Puhti
      • Spack support?

  4. I know you (or HPE) have/has challenges getting the s/w environment up and running. Complilers etc. But isn't the speciality only the slingshot? Should a single-gpu environment be fully operational already? More practically, would ready binary distributions for rocm just work for one gpu?

    • Make sure you use the same gpu driver version 5.0.2.
    • E.g. pytorch p

  5. I'd like to install software Flexpart. Does it already have a Easybuild recipe for LUMI? How do I find which Easybuild recipies are available for LUMI? Flexpart depends on jasper, netcdf and eccodes. Jasper and netcdf are already available as modules. Can I request eccodes to be installed, or how should I install it myself (using Easybuild?)?

  6. When will LUMI start enforcing REFEDS Assurance Framework? I believe we were told that this would be in effect from April, but users told me very recently that they could still log in without being flagged with the right properties from our side.

    • GEANT that is providing identity management is planning to implement REFEDS starting next year March

  7. (LUMI-C) We see often crashes reported as bus errors, in various places in our code but seemingly always in threaded code doing vector ops. It also occurs on Mahti but it jumps around in terms of where in the code we crash so we're not sure where to start debugging, or whether this is possibly an issue with the system/libraries/compilers. Has anyone else seen similar issues? (Yann/Vlasiator/U Helsinki)

    • in mahti bus errors were caused by storage system, if I understood correctly. We have not seen them after the last update ( Juha @ CSC)
    • OK thanks, I was annoyed by too many MPI/network crashes on Mahti, we'll see after the upcoming maintenance there.
    • there has been some dropped links in the interconnect after the last update. We hope those will be fixed in the next service break ( Juha )

  8. I'm working on porting Fortran code which calls some CUDA via c-iso-binding to HIP. Porting simple mini-apps to HIP works well. My issue with the full code is porting the cmake files. In the CUDA version we have check_language(CUDA) and enable_language(CUDA). When I try to do someting like this for HIP I get lots of cmake configuration errors around "Check for working HIP compiler: /opt/rocm/llvm/bin/clang++ - broken" no matter whether I set CXX to hipcc or CC . Should I remove these completely? (Andreas, ECMWF)

module use /pfs/lustrep2/projappl/project_462000125/samantao/mymodules
module load rocm/5.2.3

  1. What is the recommented way to link agains libsci as a drop-in replacement to lapack?

    • by default -lsci, -lsci_gnu or -lsci_amd (for AMD compilers) link flag is sufficient. The libraries are installed in e.g. /opt/cray/pe/libsci/22.08.1.1/gnu/91/x86_64/lib/
    • Note, there are different versions of the library: mp == OpenMP, mpi == only MPI, mpi_mp == MPI+OpenMP

  2. I am quite new to LUMI. Could we log in to LUMI via PuTTY and WinSCP? I am on Windows.

  1. Can GPU binding (as introduced in the recent online workshop) be made to work with pytorch?
    - You can invoke pytorch without the torch.distributed that makes some assumptions that are not quite right. You can use the environment:
export MASTER_ADDR=\$(scontrol show hostname "\$SLURM_NODELIST" | head -n1)
export MASTER_PORT=34567
export OMP_NUM_THREADS=2
export WORLD_SIZE=\$SLURM_NTASKS
export RANK=\$SLURM_PROCID
export LOCAL_RANK=\$SLURM_LOCALID
# invoke python with --local_rank \$LOCAL_RANK"

and then start your job with:

MASKS="ff000000000000,ff00000000000000,ff0000,ff000000,ff,ff00,ff00000000,ff0000000000"
  srun --jobid=$MYSLURMID -N $NNODES -n $((NNODES*NPROC_PER_NODE)) \
  --gpus=$((8*$NNODES)) \
  --cpus-per-task=8 --cpu-bind=mask_cpu:$MASKS

  1. Please finto use the host mpich

  2. I need different Python libraries/packages for running different ML models. What's the best approach to install the necessary Python libraries? Do we need to create a separet conda/python environment for each task, would there be any compatibility issues if different libraries/packages are installed to the same directory?

    • or small environments (small number of files) you can do pip install but with installations that need tens of thousend of files maybe you can have a looks at this solution https://docs.lumi-supercomputer.eu/software/containers/wrapper/. It will put the installations in a squashfs file. It takes care of mounting it at run time.

  3. ppi3 instldlled eaybuil robot search path is empty untill I load the EasyBuild moudule. But installing end with can not copy src
    Easy16. Is it possible that LUMI-usage is limited during winter months due to possible energy crisis in Finland?

    • They announced power cuts for the capital area/larger cities so I hope/guess that Kajaani might not be affected?

2022-09-28 13:00 – 13:45 CEST

Updates

Official 2nd phase pilot project postponed as preparatory work on GPUs (including benchmark tests) is still ongoing.

Questions

    1. We have been unable to run anything with the Vlasiator code for four weeks now as we run into bus errors almost 100% of the time. The location in the code jumps about pretty randomly but it only occurs at large-enough/production-scale node counts, i.e. above 64-100 nodes. Last Thursday I was able to run a longer test on 128 nodes and thought I had found a magic node count but that is equally crashing this week. I tried loads of modules/compilers/code versions/C++ standards 14, 17, 20/removing jemalloc/removing threading/removing vectorisation/etc. to no avail. I also noticed that I get a spurious load-balance triggered in some of my runs which I don't trigger manually nor should it be triggered by the code, which points at something being broken somewhere, not sure if it's memory or MPI or something else, but it doesn't always result in a bus error it seems. How/when will this be fixed? Are we the only users seeing this/running at 100+ node scale at the moment? Any "magic flags" to try?
    • We do see from other tickets that the communication network is not as stable as it should be though usually this results in clear error messages from MPI and OFI and not just bus errors.
    • There was a file system issue that caused bus errors but that was supposedly fixed with the last upgrade, but so it looks like that may not be fully the case.
      • I now started a test where I removed all IO except for the restart file reading and logfile output which crashed too. Should I try to also disable logfile writing to run without any writing?
    • I wonder if fluctuating performance may trigger a load-balance in your code? We do have reports about performance inconsistencies and we know of slow-downs of the file system that may cause jobs that do I/O to hang temporarily.
    • We haven't checked again after the last upgrade (as it requires a reservation) but we do know that some GROMACS simulations trigger a hardware instability in some of the processors causing crashes. I hope Vassiator doesn't do the same… But then in any case a more precise error message would be helpful.
    • Yann, do you have many MPI_Allreduce?
      • Yes we have many different things, global communication, loads of MPI data types created and committed every step, and also MPI_Allreduces. Some behaviour (the spurious load balance in particular) reminds me of the phase on Hawk where collectives silently dropped part of the nodes and we obtained inconsistent results e.g. of Allreduces across nodes/tasks.
    • According to HPE some cabinets have delay in MPI_Allreduce every some iterations, and they investigate the network. Could you repeat the sucessfull experiments on the same nodes? The network is not considered perfect yet, HPE engineers are investigating and cleaning it.
      • Turnover might be slow then, but how do I go about requesting a specific, say, 128 nodes?
      • In that respect I tried excluding nodes progressively and ended up with ~200 nodes on my list, and that list didn't seem to show patterns, but I can post it to ticket #820.
      • On, I think, Sisu, we had a flag to request running on a single cabinet or pair of cabinets, which made our code fatser by reducing network slowness – is there a similar slurm flag to try to see if it gets better if I run more locally/in one cabinet?
        • is so easy to choose, I remember the continouee does not mean they ar eon the same cabinet necessary.

    1. OpenMP bottleneck. I am trying to identify a bottleneck in an OpenMP loop that I observed on Lumi and not on our local cluster based on Intel processors. I am currently trying to figure out how to use the PAT tools, and the apprentice2 GUI but I am not really able to identify which line in the OpenMP loop which is causing this bottleneck. I could already instrument the code with build_pat -w exec, run pat_report after running a representative case and load the resulting directory in app2. Are there additional options which could give me more clues? Any help with this would be greatly appreciate. Thank you.

    1. Cray-MPICH: NULL C++ data types problem

    We have two c++ codes that uses MPI_CXX_BOOL and MPI_CXX_DOUBLE_COMPLEX but these are defined as MPI_DATATYPE_NULL in the mpich mpi.h header file:

    ​​​​> egrep 0x0c0 /opt/cray/pe/mpich/8.1.18/ofi/gnu/9.1/include/mpi.h 
​​​​#define MPI_DATATYPE_NULL  ((MPI_Datatype)0x0c000000)
​​​​#define MPI_CXX_BOOL                ((MPI_Datatype)0x0c000000)
​​​​#define MPI_CXX_FLOAT_COMPLEX       ((MPI_Datatype)0x0c000000)
​​​​#define MPI_CXX_DOUBLE_COMPLEX      ((MPI_Datatype)0x0c000000)
​​​​#define MPI_CXX_LONG_DOUBLE_COMPLEX ((MPI_Datatype)0x0c000000)

    Looking at the MPICH 4.0.2 source these types seems to be configured "compile time"

    ​​​​% wget https://www.mpich.org/static/downloads/4.0.2/mpich-4.0.2.tar.gz
​​​​% tar xzvf mpich-4.0.2.tar.gz    
​​​​% cd mpich-4.0.2
​​​​mpich-4.0.2 % egrep MPI_CXX src/include/mpi.h.in
​​​​#define MPI_CXX_BOOL                ((MPI_Datatype)@MPIR_CXX_BOOL@)
​​​​#define MPI_CXX_FLOAT_COMPLEX       ((MPI_Datatype)@MPIR_CXX_FLOAT_COMPLEX@)
​​​​#define MPI_CXX_DOUBLE_COMPLEX      ((MPI_Datatype)@MPIR_CXX_DOUBLE_COMPLEX@)
​​​​#define MPI_CXX_LONG_DOUBLE_COMPLEX ((MPI_Datatype)@MPIR_CXX_LONG_DOUBLE_COMPLEX@)

    Is it possible to get cray-mpich with C++ type support?

    • We definitely cannot build it as it is installed from binaries. The sources are proprietary.

    • This is a question to ask HPE, so please submit a ticket.

    • Thank you, I have submitted a ticket.

    1. Hi, Being a member of a pilot project, I was supposed to get access to the pilot partition but when I tried to get the allocation using srun, I recieve "Invalid account of account/partition combination specified". Any idea/suggestion to get access to the gpu/pilot partition. Thanks !
    • As announced yesterday by email (did you receive it?), the pilot phase can't start this Friday as initially announced, we are discussing with our vendor to be able to use GPUs for the pilots. We will inform you as soon as it is possible. Sorry for the inconvenience.
    1. How do we get CCE 14.0.3 installed (does contain substantial bug fixes for us)
    • Not an easy thing to do as any installation has influence on the other PEs already installed and hence on the libraries that other users will unknowingly use. So whenever we make a new PE the default, the behaviour of other programs may change which of course users who want stability cannot appreciate.
    • Should check what CSCS is doing, but it may not work on LUMI as I suspect Piz Daint uses an older version of the management environment. I do know that they also experimented with the Cray PE in a container which would solve the above problem, but when we asked them about it they were not too enthousiastic about it.
    1. Are there any upcoming calls that offer support in porting CUDA software to AMD for LUMI-G?
    • Probably at some point (next year) but none is currently planned
    • It is also pretty much meant to be a task for L3 support which should be offered by the local organisations in countries, and also by other EuroHPC programs such as the National Competence Centres and Centres of Excellence.
    1. Is there any effort of installing Alphafold globaly, as running it on singularity maight be painful for users due lot of dependencies.
    • Not in the short term
    • What is the problem with running it in a container?
    1. What status of the OpenACC support vs OpenMP-GPU? https://www.lumi-supercomputer.eu/may-we-introduce-lumi/ seems to suggest not to use OpnACC but OpenMP.
    • OpenACC for AMD GPU is only available in the Cray Fortran (but not for C/C++). It is supposed to work, but has not been tested much yet. Hopefully, we will know more after the pilot period. The official word of AMD and HPE Cray is that OpenMP should be used. AMD has no plans to support OpenACC, HPE says they have no plans to support it in C/C++.

    1. Can we install an earlier version (4.x) of ROCm? There is a QCD library that doesn't compile with 5.x at the moment.

      • What code you trying to build? Please report the problems you see with ROCM 5.
      • It's installed:
    ​​​​module load CrayEnv # or LUMI/21.12
​​​​module load rocm/4.5.2
    1. Does Cray compiler supports OpenMP offloading async e.g fortran)?
    1. Will the 1st LUMI Extreme Scale Access call projects in Finland be extded to run for one year on LUMI-G or should we apply to 2nd?
    • 1st call is meant for 1-year projects
    1. Does srun have an option to pin a thread to a core
    • For OpenMP threads this is the task of the OpenMP run time, not srun
    • It does pin MPI processes to those cores assigned to the task
    • Assuming you discuss about CPUs, SLURM has many options: https://slurm.schedmd.com/mc_support.html. For GPUs, we use like masking to map proceicate to GPUs.
    1. Is there any plan to install Tau on Lumi?
    • You could try installing it with Spack in your own home directory. "module load spack; spack install tau …"
    1. How can I monitor GPU load of my python script that I run in singularity container?

2022-10-26 13:00 – 13:45 CEST

Updates

  • GPU nodes handed over to HPE for benchmark performance tests. We will communicate by the end of the week if HPE needs them for a longer time.
    • Training: Detailed introduction to the LUMI-C environment and architecture - Hybrid training (Brussels & online) - 23-24.11.2022.

Questions

  1. Will the Nextflow module going to be installed in the Lumi cluster? There is no Nextflow module when I last checked with module spider / module avail command.

    Answer: Can you send a ticket to the support requesting it? (https://lumi-supercomputer.eu/user-support/need-help/software/)

  2. Having EAP access (via CSCS), will there be a possibility to get access to more then 16 GPUs for scaling tests and preliminary results for upcoming Euro HPC call (https://prace-ri.eu/hpc-access/eurohpc-access/eurohpc-extreme-scale-access/) with deadline Nov. 10, 2022?

    Answer: The system is currently still owned by HPE so everything depends on then. Also, passing the benchmarks so that the system can be accepted has the highest priority and some of those benchmarks need the whole system.

  3. When might LUMI-G pilots be able to start with access to the full partition?

    Answer: After all the experiences with promises that were not made true there is nothing we want to promise at the moment. The system needs to pass the benchmarks first, then they need to show that the system can be rebooted without loosing configuration information and then we can consider starting pilot.

  4. When do you plan to make large memory nodes available (4TB)? And can existing projects ask for access to Flash storage - LUMI-F scratch?

    Answer: Flash storage is currently still unavailable. There is nothing special needed to get access when it is available again. Remember to ask your RAs to get enough billing units, as the rate is 10 times higher for LUMI-F storage. It should become available again shortly after the planned reboot, assuming the storage acceptance tests that will be run after the reboot succeed.

    The 4TB-nodes have a different hardware design from the compute nodes so that is probably why they are out of use since the update of July. I assume there are problems with the OS image on those nodes, and getting the GPU nodes up-and-running has obviously a higher priority.

  5. Is the LUMI-EasyBuild-contrib repository "directly" accessible on LUMI without having to manually check it out. E.g. is there an official way to have the LUMI-EasyBuild-contrib EasyBuilds show up in eb -S (thanks for the answer, couldn't find this information in the documentation and it didn't occur to me to try)

    Answer: They have always been searchable and directly installable in the EasyBuild-user module. It is synchronised with the online GitHub whenever we do an update to the software stack which is rougly once every two weeks. If people have special requests we may sent them a link to files that are still in a separate branch though until everything is properly tested.

  6. Any comments about Julia ("for HPC") usage/users on the LUMI-G (I am the person testing things from CSCS ;-) )?

    Answer: Someone from CSCS is testing and playing with it. The downloads seem to work. There is currently nothing being done in LUST. We have (unfortunately?) higher priorities.

    One can try standard docker-hub container as a starting point. It may require some adjustments to enable multi-process Julia (-p on multiple nodes).

    Follow-up: Thanks! Actually, things work pretty smooth up to now. Julia binaries work out of the box and AMDGPU.jl + MPI.jl allow for standard HPC configs and codes to run. So no need for urgent measures others than up to date soft stack and good support in ROCm-aware MPI. Here are preliminary results if interested https://github.com/luraess/ROCm-MPI using LUMI-G EAP.

  7. LUMI-D: Is it possible to use LUMI-D GPUs for Ray-tracing feature? One of our projects is using Ray-tracing for some smart calculations and optimizations. The code is well tested on a local smaller cluster with NVIDIA-GPUs, but look forward to utilising bigger better platform like LUMI for better results.

    Answer: The LUMI-D GPU nodes are not available and there is no message yet when they will be available. As long as it is visualisation, I guess it will be OK. Currently there is also no CUDA yet which may be a gamebreaker for you I guess. It is also not clear if you will be able to get more than one node for a single job as they should not be monopolised by a single user and will be essential for Open OnDemand also.

2022-11-30 13:00 – 13:45 CET

Updates

  • 2nd pilot phase has started last 14th of November on the pilot partition (GPUs). Slightly more than half of the pilot projects have started to compute on the pilot partition. PIs of pilot projects which have not yet started have been contacted.
  • Tentative training course schedule for 2023
    • Jan: LUMI-G short intro for pilots users (1 day) online
    • Feb: LUMI general (4 days) online, end of Feb/beginning of March
    • Apr: Hackathon full week (5 days) on-site
    • Jun: LUMI general (4 days) on-site
    • Oct: LUMI general (4 days) on-site
  • EuroHPC JU has again extended the Extreme Scale mode deadline, now to 15 Dec 2022 at 10.00: https://eurohpc-ju.europa.eu/eurohpc-ju-call-proposals-extreme-scale-access-mode_en

Questions

  1. Is it possible to get a summary of GPU usage in the Slurm output (as it is done on Piz Daint, for example)?
    +1
    Answer:

  2. I noticed that Rocm 5.1.4 is the most up-to-date module available in LUMI SW. Is there a reason why there isn't a more up-to-date version on LUMI? Such as 5.2.0, 5.3.0. I can install this myself of course, I just want to know if there's a good reason for this.
    Answer:

    • The system is still in the ownership of HPE and they decide for now what they need for their benchmarks.
    • There are problems with multiple versions of ROCm on one system due to some hard-coded paths that do not contain the version.
    • There are further restrictions because not every version of the MPI libraries works with every version of ROCm. 5.2 should in principle work with the most recent MPI library on the system.
      Additional Follow-up Qs:
      • Do you know if there any potential problems with 5.3.0?
      • It is certainly not officially supported by HPE at the moment though from what I heard it should still be compatible with the current drivers on the system. But no word about compatibility with the MPI libraries for instance.

  3. I have a question regarding software installation. In our EUROHPC application form we wrote 4 different softwares we would like to test. Are we alowed to install software that wasn't on the list?
    Answer:

    • That's something that is between EuroHPC and you. We do not check if the software you run is part of your project application.

  4. Only problem in running jobs on eap was random “hipGetDeviceCount(&m_hipDevCount) error( hipErrorInvalidDevice)” errors. Re ran the job and worked fine so no idea
    Answer:

    • Please provide JOB ID that we can communicate to sysadmins to check nodes state, and use –exclude NODE ID to prevent the job running on particular nodes.
    • job=2088089 node=nid005000 (as one example). For more information it was using Kokkos installed via spack as in lumi-docs.
    • I will let sysadmins know in the chat.
    • RuntimeError: hipGetDeviceCount(&m_hipDevCount) error( hipErrorInvalidDevice): hipErrorInvalidDevice /pfs/lustrep1/scratch/project_4/.../...kokkos/build/temp.linux-x86_64-3.9/_deps/kokkos-src/core/src/HIP/Kokkos_HIP_Instance.cpp:103

End of archive

Last changed by 
 
HPC training
Training in high-performance computing and storage.
0
201

Read more

Archive of LUMI Coffee Break 2023

This is the archive document of the past sessions of the public LUMI coffee break.The archived questions and answers of 2022 sessions can be found here: https://hackmd.io/@lust/coffeearchive2022

Mar 13, 2024
Q & A sessions

:::info Link to this document: https://hackmd.io/@hpc/q-a ::: Please add here below questions or comments or share links (edit button top left or top right): Upcoming session 2024-01-25 :::info

Jan 25, 2024
NRIS HPC On-boarding 18-20 October 2022

:::info Since yesterday HackMD editing was a bit problematic (delayed response, overwritten answers), we will try this document instead and hope it behaves better. Please go here: https://md.sigma2.no/rDslp0uKRw6kvQ3Y6aZZ4Q :::

Oct 19, 2022
Best Practices on NRIS Clusters 23-24 May 2022

General information Link to this document Course page Schedule We follow the Carpentries Code of Conduct Course account and reservation:#SBATCH --account=nn9989k #SBATCH --reservation=nn9989k GPU tutorial

May 24, 2022
Read more from HPC training
Published on HackMD