# Demo of testing containerized code

## Setup computer

The `computer-setup.yaml` and it is important to configure `use_double_quotes` to `True`.

---
label: "localhost"
hostname: "localhost"
description: EPFL theospc38 computer for test purpose
transport: "core.local"
scheduler: "core.direct"
work_dir: "/scratch/{username}/aiida_run/"
shebang: "#!/bin/bash"
mpirun_command: "mpirun -np {tot_num_mpiprocs}"
use_double_quotes: True
mpiprocs_per_machine: 2
prepend_text: " "
append_text: " "

$ verdi computer setup --config computer-setup.yaml
$ verdi computer configure core.local localhost

## Creat (setup) the containerized code

### Singularity

The `singularity-pw-7.0.yaml` is the config for code setup of `pw.x` from singularity container I prepared for test (https://hub.docker.com/r/containers4hpc/qe-mpich314).
As for singularity, the image (`.sif` file) need to be
pulled first (also the case for Sarus and docker, but they can pull and have set the default directory to store and load the image by name, while for sigularity the image should be the absolute path of the `.sif` file).

I have singularity installed on my WS and the image is download by:

$ singularity pull /home/jyu/Projects/WP-aiida/containerized_code_demo/qe-mpich314_0.1.0.sif docker://containers4hpc/qe-mpich314:0.1.0

Or simply put the docker image name which will then downloaded and stored in cache.

---
label: "singularity-pw-7.0"
description: "pw.x in singularity container"
default_calc_job_plugin: "quantumespresso.pw"
image: "docker://containers4hpc/qe-mpich314:0.1.0"
engine_command: "singularity exec --bind $PWD:$PWD {image}"
filepath_executable: "/usr/local/bin/pw.x"
computer: "localhost"
use_double_quotes: true
prepend_text: " "
append_text: " "

Create the code from configure file by following command.
**(Missing action) It is also fine for config the code from interactive mode but the prompt messages are in wired order (label and description popping up after engine name/image etc.) which need to be investigate more.**

$ verdi code create core.code.installed.containerized --config singularity-pw-7.0.yaml

The executable line of the script is:

"mpirun" "-np" "2" "singularity" "exec" "--bind" "$PWD:$PWD" "docker://containers4hpc/qe-mpich314:0.1.0" "/usr/local/bin/pw.x" "-in" "aiida.in" > "aiida.out"

### Sigularity with portable script code

Useful for running python scripts with certain libraries required. With this feature, there is no need to go to remote computer to build a enviroument install required dependencies for running the script, but just build a docker container an use it.

I use the example where the `CalcJob` run the python script that use `numpy` to calculate. The code config yaml file is:

---
label: numpy-double-portable
description: "containerized portable code of numpy multiply by 2"
default_calc_job_plugin: core.templatereplacer
filepath_executable: doubler.py
filepath_files: PLACEHOLDER_PORTABLE_PATH_DOUBLER
image: docker://clearlinux/numpy-mp
engine_command: singularity exec --bind $PWD:$PWD --env APPEND_PATH=$PWD {image}
prepend_text: ' '
append_text: ' '

`APPEND_PATH=$PWD` is needed otherwise the script not in the PATH. The `doubler.py` can be any script with the corresponding `CalcJob` and `Parser`.

To run the code, it is all the same as other code. But notice that for the portable code, there is no need to bind the computer with the code, the `computer` is provided in the meta-info of calc job and can be changed on the fly.

### Conda (Not going to support)

The configure is as following, need to aware that the `escap_exec_line` need to be set to `True` to parse the container command in quotes and call by `sh -c` and `inner_mpi` need to be set to `True` to use the MPI command from inside the container rather than from the host system.
These are required by conda and docker because
1. The stdout/stderr are not mapped to host.
2. The MPI libraries are not mapped to host.

The executable line of script will be like:

"mpirun" "-np" "2" "conda" "run" "--name" "pw7-env" "bash" "-c" "'/home/jyu/miniconda3/envs/pw7-env/bin/pw.x' '-in' 'aiida.in'  > "'"'"aiida.out"'"'" "

Moreover, the conda environment need to be create and the simulation code need to be installed prehead just more or less as the regular Installed code. The `filepath_executable` has to be the absolute path and has the conda env name in the path which is inconformed with the portable goal of the containerized code.

Create the conda environment and install qe.

$ conda create -n pw7-env -y -c conda-forge qe

---
label: "conda-pw-7.0"
description: "pw.x in canda container"
default_calc_job_plugin: "quantumespresso.pw"
image: "pw7-env"
engine_command: "conda run --name {image} bash -c"
# override_mpirun_command: "mpirun -np {tot_num_mpiprocs}" <- not inplemented in command but has the attribute to set https://github.com/aiidateam/aiida-core/blob/4e422a4f26920b21fce629b27d20e99910c811bf/aiida/orm/nodes/data/code/containerized.py#L142-L146.
filepath_executable: "/home/jyu/miniconda3/envs/container-run/bin/pw.x"
computer: "localhost"
use_double_quotes: false
prepend_text: " "
append_text: " "

## Running calculation with containerized code

All the same as regular code

from aiida import orm, plugins
from aiida.engine import submit

#
# ╰─○ verdi code list
# List of configured codes:
# (use 'verdi code show CODEID' to see the details)
# * pk 1 - singularity-pw-7.0@localhost
# * pk 2 - singularity-pw-7.0-interactive@localhost
# * pk 3 - conda-pw-7.0@localhost
code = orm.load_code('singularity-pw-7.0@localhost')
# code = orm.load_code('conda-pw-7.0@localhost')
builder = code.get_builder()

# run 
# wget https://aiida-tutorials.readthedocs.io/en/tutorial-2021-intro/_downloads/92e2828a59fc133b391bbf62f0fd1b59/Si.cif
# verdi data core.structure import ase Si.cif
structure = orm.load_node(89)
builder.structure = structure
pseudo_family = orm.load_group('SSSP/1.1/PBE/efficiency')
pseudos = pseudo_family.get_pseudos(structure=structure)
builder.pseudos = pseudos
parameters = {
  'CONTROL': {
    'calculation': 'scf',  # self-consistent field
  },
  'SYSTEM': {
    'ecutwfc': 30.,  # wave function cutoff in Ry
    'ecutrho': 240.,  # density cutoff in Ry
  },
}
builder.parameters = orm.Dict(dict=parameters)

KpointsData = plugins.DataFactory('array.kpoints')
kpoints = KpointsData()
kpoints.set_kpoints_mesh([4,4,4])
builder.kpoints = kpoints

builder.metadata.options.resources = {'num_machines': 1, 'num_mpiprocs_per_machine': 2}

calcjob_node = submit(builder)

## The extra feature to run portable code with container

Using container as environment system and run the local potable script.
Useful but only tested with docker.

## Know issues

- Running with docker, the files created inside docker is owned by `root` (USERID=1000) which means the remote folder can not be cleared without `sudo`.
- override_mpirun_command should be set for when `inner_mpi` is true to overrid the computer mpirun command which is used as default for the inner_mpi. Not sure the new `_get_cli_options` support `prompt_fn`? The attribute for containerized code class [is implemented](https://github.com/aiidateam/aiida-core/blob/4e422a4f26920b21fce629b27d20e99910c811bf/aiida/orm/nodes/data/code/containerized.py#L142-L146) and can set by instant the code object.

## Follow up after the discussion

- Get rid of the specification implementation of Calcjob the part of containerized code.
- "filepath_executable" should be able to be overrided by subclass (No need to be implemented at the moment.)
- Remove the conda/docker the "inner_mpi" from this PR and using other PR with another subclass for these.
- Remove the `on_computer/on_container` prompt out option from CLI which is not used anymore.