SCC Training: Benchmarking GPAW

# SCC Training: Benchmarking GPAW Training reservation in Mahti: `SCC22` (partition `medium`) CSC user documentation: https://docs.csc.fi GPAW documentation: https://wiki.fysik.dtu.dk/gpaw GPAW presentation (50 min, no need to watch for this training): https://www.youtube.com/watch?v=-z2M3g-o_sA ## Groups - Ilmari, Leopekka, Roope: https://meet.csc.fi/SCC1 - Matias, Niklas, Huy: https://meet.csc.fi/SCC2 ## Building GPAW GPAW is a Python based software with a few Python dependencies, together with some other library dependencies. CSC's standard Python environments are container (Apptainer) based, which can sometimes complicate custom software installations, thus in this training we use a "special" Python environment in Mahti. 1. Load the Python module in Mahti: ``` module use /projappl/project_2005493/modules module load python-gpaw ``` 2. Get the source code Download the version 22.1.0 along the instructions at https://wiki.fysik.dtu.dk/gpaw/install.html#getting-the-source-code 3. Install GPAW with `pip` The downloaded code can be installed by issuing `pip install .` in the upmost directory (`gpaw`). As the default installation directory Python packages does not have write permissions, you need to provide install location either with - by setting `PYTHONUSERBASE` environment variable and using `--user` option to `pip` - `--prefix` option to `pip` :::info Note! It is not recommended to use `--user` without `PYTHONUSERBASE` as it is very easy to get conflicting versions of packages under the default "user" directory in $HOME ::: Missing Python requirements will be installed automatically, however, external library dependencies might require loading modules and customizing the installation. Look at https://wiki.fysik.dtu.dk/gpaw/install.html#customizing-installation for some more details. Note: in Mahti **libxc** (required) and **ELPA** (optional) are not available as modules but can be found at ``` /appl/spack/v017/install-tree/gcc-11.2.0/libxc-5.1.5-oa6ihp/ ``` and ``` /appl/spack/v017/install-tree/gcc-11.2.0/elpa-2021.05.001-a3dh2f/ ``` 4. Testing the build In order to run GPAW `bin` directory in the installation tree needs to be added to `PATH`. Also, `PYTHONUSERBASE` needs to be set (or the `installation_base/.../site-packages` added to `PYTHONPATH`). Furthermore, the PAW datasets need to be installed as described at https://wiki.fysik.dtu.dk/gpaw/install.html#install-paw-datasets Once everything is set, a simple serial test calculation (can be done in the login node) can be run as ``` gpaw test ``` and the same thing in parallel (in a batch job) with ``` srun gpaw test ``` ## Running the benchmark The input file `si-divacancy.py` is available at `/scratch/project_2005493/gpaw-input` in Mahti, copy the file to directory where you carry out the benchmark runs. The GPAW input files are Python scripts themselves, and are run as `gpaw python input.py`, *i.e* here with batch job as ``` srun gpaw python si-divacancy.py ``` The main output file `output_si_divacancy.txt` which contains timing information at the end, look for total time, *i.e* `Total:` :::warning If you perform multiple runs in the same directory, the output file will be overwritten. Thus, make each run in separate directory or copy the output file between the runs. ::: Running time with one node should be less than three minutes. ### Enabling ELPA Even if GPAW was built with ELPA support, it is not automatically used, but needs to be enabled in the input file. See how to enable ELPA at https://wiki.fysik.dtu.dk/gpaw/documentation/parallel_runs/parallel_runs.html#parallelization-options and investigate the effect of ELPA options on performance.