# ExCALIBUR H&ES Benchmarking demo ## June 23 Cambridge RSE seminar notes ### Local Demo 1) Clean spack chache ``` rm -rf ~/.spack/cache ``` 3) Create virtual environment ``` python3 -m venv demo_python_env source demo_python_env/bin/activate ``` 3) Install spack ``` git clone -c feature.manyFiles=true https://github.com/spack/spack.git source spack/share/spack/setup-env.sh ``` 4) Install framework ``` git clone https://github.com/ukri-excalibur/excalibur-tests.git pip install -e ./excalibur-tests export RFM_CONFIG_FILES=$(pwd)/excalibur-tests/behcmarks/reframe_config.py ``` 5) Setup spack environment (Done automatically on supported systems) ``` spack env create -d ./demo_spack_env spacktivate ./demo_spack_env spack compiler find spack external find openmpi spack external find python spack config blame [packages, compilers] export EXCALIBUR_SPACK_ENV=$(pwd)/demo_spack_env ``` 6) Run examples ``` reframe -c excalibur-tests/benchmarks/examples/sombrero -r --performance-report ``` ``` reframe -c excalibur-tests/benchmarks/apps/imb -r --performance-report --tag pingpong ``` 7) Look at output files ``` less output/default/default/default/SombreroBenchmark/rfm_build.out less perflogs/default/default/SombreroBenchmark.log ``` ### Demo on CSD3 1) SKIP Clean spack chache ``` rm -rf ~/.spack/cache ``` 2) Create virtual environment ``` module load python/3.8 python3 -m venv demo_python_env source demo_python_env/bin/activate pip install --upgrade pip ``` 3) SKIP - Install spack ``` git clone -c feature.manyFiles=true https://github.com/spack/spack.git source spack/share/spack/setup-env.sh ``` 4) Install framework ``` git clone https://github.com/ukri-excalibur/excalibur-tests.git pip install -e ./excalibur-tests export RFM_CONFIG_FILES=$(pwd)/excalibur-tests/behcmarks/reframe_config.py ``` 5) SKIP - Setup spack environment (Done automatically on supported systems) ``` spack env create -d ./demo_spack_env spacktivate ./demo_spack_env spack compiler find spack external find openmpi spack external find python spack config blame [packages, compilers] export EXCALIBUR_SPACK_ENV=$(pwd)/demo_spack_env ``` 6) Run example ``` reframe -c excalibur-tests/benchmarks/examples/sombrero -r --performance-report ``` NOTE: Building was very slow on skylake -- should probably run on icelake or cascade lake nodes. 7) Look at output files ``` less output/default/default/default/SombreroBenchmark/rfm_build.out less perflogs/default/default/SombreroBenchmark.log ``` ## July 22 Met Office meeting -- Presenter notes 1) Clean up - clear spack cache if it exists `rm -rf ~/.spack/cache` - unload python module(s) - deactivate conda 2) Install reframe - git clone [reframe](https://reframe-hpc.readthedocs.io/en/stable/started.html#github) ``` git clone -q --depth 1 --branch v3.10.1 https://github.com/reframe-hpc/reframe.git cd reframe ``` - run reframe bootstrap script `./bootstrap.sh` - optionally, add reframe to path `export PATH=$(pwd)/bin:$PATH` 3) install spack - git clone [spack](https://spack.readthedocs.io/en/latest/getting_started.html#installation) ``` git clone -c feature.manyFiles=true https://github.com/spack/spack.git ``` - run spack setup script `source ./spack/share/spack/setup-env.sh` - check spack is in `$PATH`, for example `spack --version` 4) install excalibur-tests - git clone excalibur-tests ``` git clone https://github.com/ukri-excalibur/excalibur-tests.git ``` - set reframe config file `export RFM_CONFIG_FILE="$(pwd)/excalibur-tests/reframe_config.py" ` 5) run benchmarks :tada: - run sombrero example ``` reframe/bin/reframe -c excalibur-tests/examples/sombrero/ -r --performance-report --job-option='--account=do006' ``` - run Intel MPI benchmarks ``` reframe/bin/reframe -c excalibur-tests/apps/imb/ -r --performance-report --job-option='--account=DIRAC-DO006-CPU' --system csd3:icelake -t intel-mpi -t biband ``` ### Hacks - On csd3, we had to manually request sombrero to be built with openmpi (instead of impi) using `-S spack_spec=sombrero@2021-08-16^openmpi` to reframe - If a benchmark has defined tags, pass `-t` to include a tag, `-T` to exclude a tag ### Project codes - cosma8: `do006` - csd3: `DIRAC-DO006-CPU`