# AiiDA Team Meeting 2021-09-08

###### tags: `team meetings`
###### time: 11am CET

[TOC]

### Present

* Francisco
* Ninad
* Leopold
* Sebastiaan
* Marnik
* Louis
* Martin
* Giovanni
* Chris

### Catch-up round

*Max. 3 minutes each*

Sebastiaan
* **Merged PRs**:
* [CI: do not run the benchmark on forks](https://github.com/aiidateam/aiida-core/pull/5108)
* [Entry points: add the core. prefix to all entry points](https://github.com/aiidateam/aiida-core/pull/5073)
* [CLI: fix bug in the CliFormatter log utility
](https://github.com/aiidateam/aiida-core/pull/5107)
* [Docs: add topics section on repository usage](https://github.com/aiidateam/aiida-core/pull/5113):
* **Open PRs**:
* [Engine: implement functionality to immigrate completed `CalcJobs`](https://github.com/aiidateam/aiida-core/pull/5086): The [AEP](https://github.com/aiidateam/AEP/pull/12) has been updated and finished. It is ready for final review. I have chosen the name "importer" instead of "immigrator". Justification for this naming is included in the AEP.
* [Dependencies: update to `click==8.0` and remove `click-completion` ](https://github.com/aiidateam/aiida-core/pull/5111): Typical case of underestimating the amount of work required. Quite some changes were necessary, but it does allow us to drop a dependency.
* [`Node`: add the `copy_tree` method](https://github.com/aiidateam/aiida-core/pull/5114): Basic operation that was possible but required a lot of manual code and was requested a lot. Should be a straightforward review.
* [`CalcJob`: allow directories in `local_copy_list`](https://github.com/aiidateam/aiida-core/pull/5115): Builds on the previous PR as it leverages the `Node.copy_tree` method. This makes writing `CalcJob` plugins that need to copy files from a `FolderData` a lot easier. For example, `aiida-quantumespresso` needs this for various calcjob plugins.
* [CLI: rework the logging verbosity implementation](https://github.com/aiidateam/aiida-core/pull/5119): Change in behavior of the recently implemented `--verbosity` option for the command line requested by @nnmOmSLLTKymjzpfuwixfA

Chris:
* https://github.com/chrisjsewell/aiida-task-controller
* RabbitMQ >= 3.7 will start breaking without manual configuration: https://github.com/rabbitmq/rabbitmq-server/pull/2990
* A working prototype for a non-RabbitMQ Process execution coordinator (README provides full description)
* Helpful feedback from Martin and Gio
* disk-objectstore
* 🚀 RELEASE: v0.6.0 (#118)
* ✨ NEW: Add basic CLI (#116)
* ⬆️ UPGRADE: sqlalchemy v1.4 (v2 API) (#114)
* 🔧 MAINTAIN: Add typing (#113)
* aiida
* ♻️ REFACTOR: Remove QueryManager (#5101)
* 🔧 MAINTAIN: Remove _DeprecateConfigCommandsGroup (#5110)
* awaiting review
* ⬆️ UPGRADE: sqlalchemy v1.4 (v1 API) #5103
* ✨ NEW: Add orm.Entity.fields interface for QueryBuilder
* In progress
* 👌 IMPROVE: Add SQLA migration for parity with Django schema #5097

Simon:
* _Attending the all-day MarketPlace consortium meeting._
* Reviewed [PR #5104](https://github.com/aiidateam/aiida-core/pull/5104). (allow upgrade of psycopg2)
* Assigned and working on [issue #5117](https://github.com/aiidateam/aiida-core/issues/5117).

Francisco:
* Currently working on the maintain CLI ([#4965](https://github.com/aiidateam/aiida-core/pull/4965))
* Support to Samuel on querying for sub-called processes of some workchains. **Lessons:** the `verdi process status` is a useful command to consider for the querying section of the docs. Even better if we can make something that is more comfortable to use in the verdi shell instead of just text that needs to be parsed (idea: a process/workflow method that returns a nested dict of called processes)
* Attended CSCS User Lab day (Last Friday Sep 03)
* They gave an overall view of the services they have (Sarus for containers, Jenkins for CI, FirecREST for workflows, etc) and of the submission process to request running time via projects.
* Discussed about the "HTC scheduler problem"
* **PROBLEM:** schedulers in computing clusters (at least SLURM) seem to, by default, have the nodes as the minimal resource unit (for example, this is how CSCS works). As nodes become more powerful ([ALPS](https://www.cscs.ch/computers/alps/) will have 2 sockets with 64 cores each = 128 cores) most HTC calculations don't need to use a whole node.
* The reason for this seems to be that supporting a lot of small runs requires the SLURM installation to occupy a lot of the resources for management that would otherwise be available for running. It is **not** related to inefficient/unfair resource distribution (SLURM can handle that just fine).
* **CSCS SOLUTION:** they use [GREASY](https://user.cscs.ch/tools/high_throughput/) as a meta scheduler. Typically you create a text file with all HTC calls (tasks), and you submit a script that starts an instance of GREASY in the nodes which reads that text file and executes the runs.
* **INTERACTION WITH AiiDA:** In principle is complicated, since AiiDA wants to have a separate script submission to a queue for each run and GREASY launches 1 SLURM script for many calculations.
* New SLURM-GREASY AiiDA scheduler? Is it worth it? (consider: specific purposes like LUMI, and general direction of computer clusters).
* Should be coordinated with the [Fireworks scheduler](https://aiida-fireworks-scheduler.readthedocs.io/en/latest/) and task farming (see Leopold's comment).
* GREASY reads updates on the textfile with the tasks, so the "submission script" could append to that file.
* There should be a mechanism that checks if there is already a GREASY instance running in the SLURM queue, and if not it needs to be generated (with an initial 5 min sleep task?)
* GREASY keeps a log file that could be parsed to figure out when jobs have finished.
* It also currently have some limitations but I was told these can be fixed if requested (for example: all tasks in the same GREASY instance currently have to use the same ammount of cores).

Leopold:
* opened [thread on task farming](https://github.com/aiidateam/aiida-core/discussions/5112) - @francisco, perhaps you want to add this info there?

Marnik:

* Nothing to report on the `aiida-core` front.
* Attended LUMI briefing today with Francisco. Mainly information on when the pilot phase will finally start after some delays (11th of October). I don't see any changes required for `aiida-core`, however Francisco did mention we might be able to use Bonan's Fireworks scheduler for some of the post-processing.

Giovanni:

* Any feedback on the implementation suggestion [in this issue comment](https://github.com/aiidateam/aiida-core/issues/5084#issuecomment-900278316)? This would be preliminary to any meta-scheduler work and simplify the life also for the aiida-fireworks-scheduler.
* Nothing new to report.

Jason:
* _Attending the all-day MarketPlace consortium meeting_ with Simon.
* I am sorry guys. I really want to say hi to you all! Anyway, I am stay on the AiiDA launcher now and prepared to start in this week. :rocket:

### Specific Agenda Items

* Presentation of `aiida-restapi` plugin by Ninad Bhat (GSOC 2021) [slides](https://docs.google.com/presentation/d/1UtADuobi0BX9MhUi3phAe9ESR801g1wUGzQB8HAz9sM/edit#slide=id.p)
* You can comment directly on the slides
* Feedback: Sebastiaan - use just the node type (not the "full type"); use the entry points instead