AiiDA Team Meeting 2021-09-22

# AiiDA Team Meeting 2021-09-22 ###### tags: `team meetings` ###### time: 10am CEST [TOC] ### Present * Chris * Sebastiaan * Jason * Francisco * Simon * Lorenzo * Marnik * Louis ### Catch-up round *Max. 3 minutes each* Giovanni: * (Note: I might not be present as I'm on paternity leave today - maybe I join only briefly to introduce Lorenzo) * Lorenzo is joining the group for 3 months, working on AiiDA-VASP related workflows - let's welcome him and help him should he have any AiiDA question! * Related: does the AiiDA tutorial page now says how to use via Quantum Mobile, rather than the AWS machine now offline? It's urgent to fix this ASAP to avoid users go to the tutorials page, get lost and go away. If Marnik didn't manage to fix it already, let's share the work (Lorenzo and Jason were willing to help, but also Francisco and Chris can help - let's make sure it's done by the end of this week) **[mbx] This has been fixed** 😁 * If you are interested in a participating in a meeting about task farming? Discussion here: https://github.com/aiidateam/aiida-core/discussions/5112 and J. Weston wrote in the AiiDA ML and then to me that he was interested in brainstorming. (The priority is still 2.0 before this, but having a discussion to collect usecases is good already now to see what could be beneficial to multiple users) * If you are interested in participating in a follow-up discussion with an external team that tried to used a multi-profile REST API and has some feedback please contact Sebastiaan, Chris and me (ideally all three so we don't miss the messages). * Will start a discussion at Empa with a group interested in using AiiDA in a non-materials-science context (atmospheric simulations). This is part of a potential future project that could get funding in collaboration with Empa (Carlo). Anybody interested, please let me know (I will have a first meeting in a few weeks, at least to hear about the usecase and understand if it's easy or complex to use AiiDA for their codes). Chris: * SQLAlchemy -> v1.4 and v2 API * New repository draft! https://github.com/aiidateam/aiida-core/pull/5145 * Twice the export speed, ~5 times less memory usage! * ✨ NEW: Add orm.Entity.fields interface for QueryBuilde #5088 * Seb has reviewed thanks, circle back soon Sebastiaan: * **Merged PRs**: * [`DirectScheduler`: use `num_cores_per_mpiproc` if defined in resources](https://github.com/aiidateam/aiida-core/pull/5126): Small fix that defines `OMP_NUM_THREADS` environment variable. * [Engine: implement functionality to immigrate completed `CalcJobs`](https://github.com/aiidateam/aiida-core/pull/5086): The [AEP](https://github.com/aiidateam/AEP/pull/12) has been merged and the implementation is also merged. I have written a [wiki-page with instructions on how to write a `CalcJobImporter`](https://github.com/aiidateam/aiida-core/wiki/Writing-a-CalcJobImporter-plugin). Asked during a Common Workflow meeting whether the various plugin developers could have a stab at it. * [`Node`: add the `copy_tree` method](https://github.com/aiidateam/aiida-core/pull/5114): Basic operation that was possible but required a lot of manual code and was requested a lot. * [`CalcJob`: allow directories in `local_copy_list`](https://github.com/aiidateam/aiida-core/pull/5115): Builds on the previous PR as it leverages the `Node.copy_tree` method. This makes writing `CalcJob` plugins that need to copy files from a `FolderData` a lot easier. For example, `aiida-quantumespresso` needs this for various calcjob plugins. * [CLI: rework the logging verbosity implementation](https://github.com/aiidateam/aiida-core/pull/5119): Change in behavior of the recently implemented `--verbosity` option for the command line. * **Open PRs**: * [Dependencies: update to `click==8.0` and remove `click-completion` ](https://github.com/aiidateam/aiida-core/pull/5111): Typical case of underestimating the amount of work required. Quite some changes were necessary, but it does allow us to drop a dependency. * [`List`: register the class with the `to_aiida_type` serializer ](https://github.com/aiidateam/aiida-core/pull/5142): Makes it behave more like other base types, e.g., `Int`, `Str` etc. * [REST API: make the profile configurable as request parameter ](https://github.com/aiidateam/aiida-core/pull/5054): Has been open for a while. Would be good if it could get some performance testing from the REST API users. The external team mentioned by Giovanni would require this feature, among other things. Marnik: * Wrote executive summary on node equality comparison discussion, see [here](https://docs.google.com/document/d/1JsJg4r-YT8-Pc9IOdjoXcUuinEzFP5IQWDz56MVVEds/edit#). Suggest to organise a meeting to make a decision on this. Before we do though, Gio suggested to set up a poll on expected behavior and send it to the mailing list. [Please have a look at my suggestion and comment](https://docs.google.com/document/d/1JsJg4r-YT8-Pc9IOdjoXcUuinEzFP5IQWDz56MVVEds/edit#heading=h.syfkzpxa8or2). * Opened issue on adding a module on input serialization to the tutorials repo, see [[#404](https://github.com/aiidateam/aiida-tutorials/issues/404)]. Jason: - Merged PR: [fix for Exit codes spec message of process show by verdi plugin list.](https://github.com/aiidateam/aiida-core/pull/5039) - Open PR: [allow to not set input plugin for code when setting code through verdi.](https://github.com/aiidateam/aiida-core/pull/5140) The issue from click side is that click will keep prompting for the value each time when user don't give any input but just hitting an ENTER. There are two ways to deal with it: - use an extra flag for example `--bind-input-plugin` to control whether to set the input plugin for the code. - change the default behaviour of click to read the ENTER as a `None` input for the parameter (accept empty value for an optional prompted value). - ~~[aiida-testing](https://github.com/aiidateam/aiida-testing): who continues to contribute to it? Any plan to release it to pypi? (@Leopold, I still have problem in running it, can we have a meeting so you can help me to address it?)~~ Simon: - [80% done] Revised AiiDAlab docs in collaboration with Leopold and Sasha. - [ongoing] Better support locally installed (non-registered) AiiDAlab apps. - [ongoing] Support direct installation of AiiDAlab apps from PEP 508 compliant URLs. - [ongoing] Make it easier to inject private SSH key into AiiDAlab instance. - [ongoing] Improve logic for the code #CPU detection on aiida-prerequisites [#5117](https://github.com/aiidateam/aiida-core/issues/5117) - [chris] Do something similar in Quantum Mobile, may be worth a look - [simon] do you have a link? [chris] looking... (maybe slack ) Francisco - Proposal for the repository maintainance CLI should be ready for review ([#4965](https://github.com/aiidateam/aiida-core/pull/4965)) - Main command `verdi repository maintain` - [chris] I feel this should be `verdi repository optimize`, so that user actually want to use it! At least this whole functionality should be made clear enough to users, so that it is not wasted - Does deep maintainance, requires the user to stop using the profile, `--live` flag available to do only safe operations. - Option to pass further customized options to the backend (testing, power using). - Also trying to figure out the ~~deamon~~ profile locking mechanism that would be quite important to make the maintain CLI safer ([#5135](https://github.com/aiidateam/aiida-core/pull/5135)). - [simon] The most reliable locking mechanism remains to create a file in a known location. - [FFR] The problem is not so much locking for future access, but making sure there is no previous access still open. - [simon] the daemon creates the lock file when it starts and deletes it when it stops. you turn the issue on its head. a daemon that crashes can be detected and then the user can opt to delete those manually after checking that all daemons are actually stopped. similar to the vim lock files. - [chris] this is basically what I already did https://github.com/aiidateam/aiida-core/pull/4924 - [FFR] @simon yes but the main problem is that it is not just the daemon that can access the DB and cause problems (sorry for the misleading description, I now changed "daemon" to "profile" to be more precise on what we are trying to do). You can also have `verdi` commands running, `verdi` shells open, restAPI accesses, and even scripts that just load the profile themselves. These other ways of accessing the DB ("official AiiDA ways", we can't prevent the user from just psql-ing into his DB if so he chooses) have a single point of entry (loading the profile) so it is easy to record when they start accessing, but there is no access termination process. Last idea on this direction was keeping track of which processes are requesting access and checking if they are still running during some automatic cleanup stage, but there are some considerations on performance and "syncronicity" to be discussed (see the PR where all of these ideas are explained a bit more concretely). # General points * v2.0 milestone: we set deadline for September 17th which we didn't make. What does currently still really need to go in and what is the estimated required time for this. * [chris] Main elements * Repository CLI (PR in draft) * New archive format (PR in draft) * Migration to sync SQLAlchemy backend with Django (PR in draft) * Drop Django 🤞(no PR yet, but I have a general idea of how this would work) * Probably also node equality change * [marnik] Minor elements: * Small changes to `List` and `Dict` data types [[#4495](https://github.com/aiidateam/aiida-core/issues/4495)]: The UX changes should require little time, whether to make `Dict` a proper `BaseType` will depend on the decision we make regarding content comparison re equality. * ProcessBuilder: Add pretty representation 😍 [[#4970](https://github.com/aiidateam/aiida-core/pull/4970)]: Just some pre-commit issues, should manage to fix these next week. General consensus seems to be that the critical issues listed above will take quite a few more weeks to fix. A more realistic date seems to be early November, putting an actual release closer towards the end of 2021. The v2.0 milestone will be cleaned up and only the critical issues will be kept. Everything else is relegated to the post v2.0 milestone.