Try   HackMD

Spack at HPCFS

Spack provides prebuild modules for all users on HPCFS. Specific module builds are listed below. We restric our builds to RHEL8.4 provided compiler GCC@8.5.0 in order to provide haswell and rome CPU compatibility (arch=zen) for system provided layer (SLURM, UCX, knem).

Higher version GCC compilers are used on ROME with zen2 (avx2) compatibility across rome and haswell partitions, while AOCC compilers are intended just for AMD rome partition.

User Spack development

It is also possible to use modules locally for your own use by setting up the following configurations for local build and deployment.

rm -rf ~/.spack
mkdir ~/.spack
sed -e '/root:/s,$spack,/work/$USER,' /opt/spack/etc/spack/defaults/config.yaml  > ~/.spack/config.yaml
sed -i -e '/source_cache:/s,:.*,: ~/.spack/cache,' ~/.spack/config.yaml
sed -i -e 's/# build_jobs: 16/build_jobs: 32/' ~/.spack/config.yaml
sed -e 's/# roots:/roots:/' -e 's,#  lmod:.*,  lmod: /work/$USER/opt/lmod,' -e 's,#  tcl:.*,  tcl: /work/$USER/opt/modules,' -e '/- tcl/a\      - lmod' /opt/spack/etc/spack/defaults/modules.yaml > ~/.spack/modules.yaml

Run source in active bash or add source line in your .bashrc profile file

​​​​source /opt/spack/share/spack/setup-env.sh

hwloc

Hardware locality detects CUDA with the latest version since it is not used for computing but rather detection. Cuda provides OpenCL devices. NVML is provided by NVIDIA drivers and can't be part of hwloc since only few gpu nodes provides drivers and corresponding libnvidia-ml.so.1 library.
The rest of locality info are provided through ubus and pci.

[spack@gpu02 ~]$ spack spec  --install-status --long  hwloc
Input spec
--------------------------------
 -   hwloc

Concretized
--------------------------------
 -   gg2gaiz  hwloc@2.6.0%gcc@8.5.0~cairo+cuda~gl+libudev+libxml2~netloc~nvml+opencl+pci~rocm+shared arch=linux-almalinux8-zen
[+]  5lbelaa      ^cuda@11.5.1%gcc@8.5.0~dev arch=linux-almalinux8-zen
...

PMIX

Process managment interface is used by SLURM and OpenMPI.
To provide a chained compilation with input specification propagated by hash of the build (e.g. /gg2gaiz)

[spack@gpu02 ~]$ spack spec -lI pmix@3.2.1 ^hwloc/gg2gaiz
Input spec
--------------------------------
 -   pmix@3.2.1
[+]      ^hwloc@2.6.0%gcc@8.5.0~cairo+cuda~gl+libudev+libxml2~netloc~nvml+opencl+pci~rocm+shared arch=linux-almalinux8-zen
[+]          ^cuda@11.5.1%gcc@8.5.0~dev arch=linux-almalinux8-zen
[+]              ^libxml2@2.9.12%gcc@8.5.0~python arch=linux-almalinux8-zen
[+]                  ^libiconv@1.16%gcc@8.5.0 libs=shared,static arch=linux-almalinux8-zen
[+]                  ^xz@5.2.4%gcc@8.5.0~pic libs=shared,static arch=linux-almalinux8-zen
[+]                  ^zlib@1.2.11%gcc@8.5.0+optimize+pic+shared arch=linux-almalinux8-zen
[+]          ^libpciaccess@0.16%gcc@8.5.0 arch=linux-almalinux8-zen
[+]          ^ncurses@6.1.20180224%gcc@8.5.0~symlinks+termlib abi=6 arch=linux-almalinux8-zen

Concretized
--------------------------------
 -   atirvkd  pmix@3.2.1%gcc@8.5.0~docs+pmi_backwards_compatibility~restful arch=linux-almalinux8-zen
[+]  gg2gaiz      ^hwloc@2.6.0%gcc@8.5.0~cairo+cuda~gl+libudev+libxml2~netloc~nvml+opencl+pci~rocm+shared arch=linux-almalinux8-zen
[+]  5lbelaa          ^cuda@11.5.1%gcc@8.5.0~dev arch=linux-almalinux8-zen
...

SLURM

Slurm includes PMIX module with hwloc and PMI2 provided internally.

[spack@gpu02 ~]$ spack spec -Il slurm@21-08-1-1%gcc@8.5.0+hwloc+pmix ^pmix/atirvkd
Input spec
--------------------------------
 -   slurm@21-08-1-1%gcc@8.5.0+hwloc+pmix
[+]      ^pmix@3.2.1%gcc@8.5.0~docs+pmi_backwards_compatibility~restful arch=linux-almalinux8-zen
[+]          ^hwloc@2.6.0%gcc@8.5.0~cairo+cuda~gl+libudev+libxml2~netloc~nvml+opencl+pci~rocm+shared arch=linux-almalinux8-zen
[+]              ^cuda@11.5.1%gcc@8.5.0~dev arch=linux-almalinux8-zen
[+]                  ^libxml2@2.9.12%gcc@8.5.0~python arch=linux-almalinux8-zen
[+]                      ^libiconv@1.16%gcc@8.5.0 libs=shared,static arch=linux-almalinux8-zen
[+]                      ^xz@5.2.4%gcc@8.5.0~pic libs=shared,static arch=linux-almalinux8-zen
[+]                      ^zlib@1.2.11%gcc@8.5.0+optimize+pic+shared arch=linux-almalinux8-zen
[+]              ^libpciaccess@0.16%gcc@8.5.0 arch=linux-almalinux8-zen
[+]              ^ncurses@6.1.20180224%gcc@8.5.0~symlinks+termlib abi=6 arch=linux-almalinux8-zen
[+]          ^libevent@2.1.8%gcc@8.5.0+openssl arch=linux-almalinux8-zen
[+]              ^openssl@1.1.1l%gcc@8.5.0~docs certs=system arch=linux-almalinux8-zen

Concretized
--------------------------------
 -   bpmt4tx  slurm@21-08-1-1%gcc@8.5.0~gtk~hdf5+hwloc~mariadb+pmix+readline~restd sysconfdir=/etc/slurm arch=linux-almalinux8-zen
...

OpenMPI

OpenMPI is build on top of SLURM, PMIX.
UCX fabrics with knem is provided externally as part of Mellanox OFED HPC build. To sumarize these .spack/packages.yml

packages:
  lustre:
    externals:
    - spec: lustre@2.14.55
      prefix: /usr
    buildable: false
#...
#...
  hwloc:
    compiler: [gcc@8.5.0]
    variants: +cuda+opencl+libudev^cuda@11.5.1
  munge:
    compiler: [gcc@8.5.0]
    variants: localstatedir=/var
  slurm:
    compiler: [gcc@8.5.0]
    variants: +hwloc+pmix sysconfdir=/etc/slurm
  knem:
    externals:
    - spec: knem@1.1.4
      prefix: /opt/knem-1.1.4.90mlnx1
  ucx:
    externals:
    - spec: ucx@1.11.1+thread_multiple +cma +rc +ud +dc +mlx5-dv +ib-hw-tm +dm +cm +knem
      prefix: /usr
  openmpi:
    variants: +pmi+pmix+lustre fabrics=ucx,knem schedulers=slurm
  mpich:
    variants: +slurm pmi=pmix

For various compilers slurm concretization is applied with the above defaults.

[spack@gpu02 apps]$ spack spec -lI openmpi%gcc@8.5.0 ^slurm/bpmt4tx                                                                                           
Input spec
--------------------------------
 -   openmpi%gcc@8.5.0
[+]      ^slurm@21-08-1-1%gcc@8.5.0~gtk~hdf5+hwloc~mariadb+pmix+readline~restd sysconfdir=/etc/slurm arch=linux-almalinux8-zen
           
...

Concretized
--------------------------------
 -   nezrdtx  openmpi@4.1.2%gcc@8.5.0~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers+lustre~memchecker+pmi+pmix~singularity~sqlite3+
static~thread_multiple+vt+wrapper-rpath fabrics=knem,ucx schedulers=slurm arch=linux-almalinux8-zen
[+]  gg2gaiz      ^hwloc@2.6.0%gcc@8.5.0~cairo+cuda~gl+libudev+libxml2~netloc~nvml+opencl+pci~rocm+shared arch=linux-almalinux8-zen
[+]  5lbelaa          ^cuda@11.5.1%gcc@8.5.0~dev arch=linux-almalinux8-zen
...

[spack@gpu02 apps]$ cat >> $(spack location -i openmpi%gcc@8.5.0)/etc/openmpi-mca-params.conf << EOF
btl=^openib
opal_common_ucx_opal_mem_hooks=1
EOF

[spack@gpu02 ~]$ spack install openmpi%aocc@3.1.0 ^slurm/bpmt4tx ^knem%gcc@8.5.0 ^lustre%gcc@8.5.0
[spack@gpu02 ~]$ cat >>  $(spack location -i openmpi%aocc@3.1.0)/etc/openmpi-mca-params.conf << EOF
btl=^openib
opal_common_ucx_opal_mem_hooks=1
EOF


spack spec -lI openmpi%intel@2021.4.0 ^slurm/bpmt4tx 
spack install openmpi%intel@2021.4.0 ^slurm/bpmt4tx
==> Warning: Intel compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors
cat >>  $(spack location -i openmpi%intel@2021.4.0)/etc/openmpi-mca-params.conf << EOF
btl=^openib
opal_common_ucx_opal_mem_hooks=1
EOF

[spack@gpu02 ~]$ spack install openmpi%gcc@11.2.0 ^slurm/bpmt4tx  ^ucx/i4eomv4
cat >>  $(spack location -i openmpi%gcc@11.2.0)/etc/openmpi-mca-params.conf << EOF
btl=^openib
opal_common_ucx_opal_mem_hooks=1
EOF

Compiling and running OpenMPI with srun requires PMIX

cat > hello.f90 <<EOF
program hello
   use mpi
   integer rank, size, ierror, strlen, status(MPI_STATUS_SIZE)
   character(len=MPI_MAX_PROCESSOR_NAME) :: hostname

   call MPI_INIT(ierror)
   call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror)
   call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
   call MPI_GET_PROCESSOR_NAME( hostname, strlen, ierror )
   print*, trim(hostname), rank, size
   call MPI_FINALIZE(ierror)
end
EOF
ml openmpi-4.1.2-gcc-8.5.0-nezrdtx 
mpif90 hello.f90
srun  --mpi=pmix -p rome -N5 -n40 --mem=0 a.out

ml purge
ml openmpi-4.1.2-aocc-3.1.0-43vdsd3
ml aocc-3.1.0-gcc-11.2.0-plf5zph
mpif90 hello.f90 
srun  --mpi=pmix -p rome -N5 -n40 --mem=0 a.out

Wrapper mpirun is not built by default anymore with OpenMPI.
You should use srun --mpi=pmix instead. See http://hpc.fs.uni-lj.si/slurm examples.

Linpack

[spack@gpu02 ~]$ spack spec -lI hpl+openmp%aocc@3.1.0 ^amdblis%aocc@3.1.0 threads=openmp ^openmpi/43vdsd3         
Input spec
--------------------------------
 -   hpl%aocc@3.1.0+openmp
 -       ^amdblis%aocc@3.1.0 threads=openmp
 ...

OpenFOAM

spack spec -Il openfoam%gcc@11.2.0+metis+zoltan+mgridgen+paraview  ^openmpi/ip5xqcx ^paraview+qt ^mesa@21.3.8

module load openfoam-2112-gcc-11.2.0-qi6xbnd

There are also AOCC compiled OpenFOAM modules but it seems that not all utilities are compiled due to incompatibility or compiler detection.

Sample OpenFOAM batch script can be

#SBATCH -p rome
#SBATCH -n 96
#SBATCH --mem=0
#SBATCH --ntasks-per-node=48

module purge
module load openfoam-2112-gcc-11.2.0-qi6xbnd

# Decompose solution (serial)
decomposePar -force > log.decomposeParDict 2>&1

# Run the solution (parallel)
srun --mpi=pmix compressibleInterFoam -parallel > log.CIF 2>&1

or

TAU profiler and analysis utility

Built for Intel and gcc compiler

$ spack install tau%intel+mpi+ompt+openmp ^openmpi/hovu6pi
$ spack install tau%gcc@11.2.0+mpi+ompt+openmp ^openmpi/ip5xqcx

AMDScalaPack, MUMPS, PETSc

​​​​spack spec -I amdscalapack%aocc ^openmpi/43vdsd3
​​​​spack spec -I mumps%aocc ^amdscalapack/g4eckyd
​​​​spack spec -I petsc%intel ^amdscalapack/g4eckyd
​​​​spack spec -I petsc%intel +mkl-pardiso+scalapack+valgrind  ^openmpi/hovu6pi

stream benchmark

According to AMD developer instructions the benchmark should be:

​​​​spack spec -I stream%aocc+openmp cflags="-mcmodel=large -DSTREAM_TYPE=double -mavx2 -DSTREAM_ARRAY_SIZE=260000000 -DNTIMES=10 -ffp-contract=fast -fnt-store"

ParaView/tajjatz

​​​​spack install paraview+mpi+qt ^openmpi/nezrdtx ^mesa@21.3.8

ha7jdza

​​​​spack install paraview+mpi+qt+python3 ^openmpi/nezrdtx ^mesa@21.3.8

Fenics

vtzd4lz

​​​​spack spec fenics-dolfinx ^openmpi/ip5xqcx

gmsh@4.8.4/p2z7wue

​​​​spack install gmsh%gcc ^openmpi/nezrdtx
tags: HPCFS spack
Last changed by 
 
HPCFS
documentation
0
1860

Read more

SLURM at HPCFS

20 compute nodes with Rocky Linux 8.5 have the folowing characteristics:

May 6, 2025
Welcome to HPCFS

===

Mar 26, 2025
Mathematica

License server flex.hpc When starting ​​​​mathematica or ​​​​wolframnb for the first time ab actrivation window pops out suggesting activation through web. Please find a button Other ways to activate in the bottom of the dialog and then

Mar 26, 2025
News

Upgrade of Lustre file servers to 2.15.1

Oct 28, 2024
Read more from HPCFS
Published on HackMD