AiiDA Team Meeting 2021-12-15

# AiiDA Team Meeting 2021-12-15 ###### tags: `team meetings` ###### time: 10am CET [TOC] ### Present * Chris * Giovanni * Francisco * Marnik * Sebastiaan * Jusong * Simon ### Catch-up round *Max. 3 minutes each* Chris: * Ready for review: * 👌 IMPROVE: Add SQLA migration for parity with Django schema [#5097](https://github.com/aiidateam/aiida-core/pull/5097) * 👌 IMPROVE: Legacy tar file archive migration performance [#5275](https://github.com/aiidateam/aiida-core/pull/5275) * Updated: * ✨ NEW: Add orm.Entity.fields interface for QueryBuilder [#5088](https://github.com/aiidateam/aiida-core/pull/5088) * Meeting with Jonathon Chico on revamped aiida-lammps Sebastiaan: * Misc * AEP 006: fixed the formatting and am now reviewing it. When done, will also finalize my [AEP on the implementation of the disk object store in `aiida-core`](https://github.com/aiidateam/AEP/pull/7). * Merged PRs: * [`DirectScheduler`: remove the `-e` option for bash invocation](https://github.com/aiidateam/aiida-core/pull/5264): This ensures that the submit script is not exited as soon as any command return non-zero exit status. This is the behavior we want and all other scheduler plugins work like this. * Open PRs: * [`Process`: raise when `exposed_outputs` gets non-existing `namespace`](https://github.com/aiidateam/aiida-core/pull/5265): Raise useful message when users misspell the namespace when attaching exposed outputs as outputs. * Small Demo: * Recently worked on a new plugin [`aiida-fenics`](https://github.com/sphuber/aiida-fenics) for the [FEniCS](https://fenicsproject.org/) project which is a tool for solving partial differential equation. Was brought to my attention by German researcher who are comparing workflow systems to implement their workflows. Part of their workflow involves submitting a heavy calculation to a cluster, for which AiiDA is ideally suited, but many other parts are the simple invocation of shell scripts or executables that operate on files and produce new files. These simple operations have been notoriously difficult to include into an AiiDA workflow, yet outside of computational materials science, they are very common operations. I was wondering if it would be possible to simplify this and came up with the `@shellfunction` decorator. Simon: * Started working on identifying outdated dependencies, some of them are quite old. * As part of this: Exploring to refactor the `utils/dependency_management.py` script into a stand-alone package (dep-utils or so) that could also be used in another projects. * Can we drop the use of `setup.json` and use the _state-of-the-art_ `setup.cfg` approach to define our build and install process? * Recommend to drop support Python 3.7 and NumPy 1.17 at the next minor or major release. Giovanni: * Shall we drop Jenkins? It's a lot of burden and gives not enough advantages to merit keeping it online [SPH: it is true that we rarely benefit from it, but I do want to mention that recently (few weeks ago) it caught a bug that wasn't captured on GHA.] * Task farming: tested hyperqueue, really nice. Discussed with developers who are very active. Marnik implemented aiida-hyperqueue, I will let him discuss the details (might benefit from changes by Jason about optionally not escaping with single quotes). * Added docs to disk-objectstore. I realize now it's not displaying on RTD... Chris can you help me check? Is the RTD docs under your account or somewhere I can access? * AEP: https://github.com/aiidateam/AEP/pull/11 updated with current state and ready to be merged - great if someone (Seb? Chris?) could approve and merge. Maybe then update and merge also https://github.com/aiidateam/AEP/pull/7 by Seb? * Other minor PRs with fixes * Empa: working (Sasha et al., me helping for design) on an aiida-flexpart plugin for atmospheric simulations. * Working with Empa on using AiiDA to submit experimental jobs (create a battery, test a battery, ...), with Loris and Francisco. Main discussion: how to represent in AiiDA experimental samples that, once processed, they "change" so cannot be reused anymore for new experiments ("calculations") in AiiDA. Francisco: * CLI repository maintenance tool merged ([PR 4965](https://github.com/aiidateam/aiida-core/pull/4965)) * **To-Do:** [add documentation](https://github.com/aiidateam/aiida-core/issues/5271), get some performance stats. * **Extras:** some outstanding non-essential (for 2.0) features (gathered [here](https://github.com/aiidateam/aiida-core/issues/5272)). * Proposal for the profile locking mechanism ([PR 5270](https://github.com/aiidateam/aiida-core/pull/5270)) * **Waiting for feedback** Marnik: * **Task Farming**: first version `aiida-hyperqueue` plugin is on GitHub: https://github.com/aiidateam/aiida-hyperqueue * Seems to work quite well, I've tried submitting ~1000 one core calculation jobs through the scheduler, and HQ does a good job at automatically starting Slurm submissions. Using the `--cpu-bind` tag in combination with the `$HQ_CPUS` environment variable also makes sure no processes are run on the same CPUs, also on Eiger. * Wrote a small "Getting Started" page with the setup steps in [the documentation](https://aiida-hyperqueue.readthedocs.io/en/latest/get_started.html#getting-started). * HyperQueue team is very responsive and has already fixed several of our issues (e.g. disabling hyperthreading). Jason: * More issue about `PhCalculation` caching. * More about `aiormq.exceptions.ChannelInvalidStateError: writer is None`. The issue always appears along with [ChannelInvalidStateError: avoid excepting the job when this happens](https://github.com/aiidateam/aiida-core/issues/5031) in my case, I guess they are related? This block me in running SSSP workflow. Can anyone look into it with me together. ### Specific Agenda Items