# AiiDA Team Meeting 2020-12-01

###### tags: `team meetings`
###### time: 4pm CET

[TOC]

### Present

* Sebastiaan
* Chris
* Giovanni
* Leopold
* Marnik
* Casper
* Francisco
* Simon
* Sasha

### Catch-up round

*Max. 3 minutes each*

Sebastiaan:

Updated the [v1.6](https://github.com/aiidateam/aiida-core/milestone/38) and [v2.0](https://github.com/aiidateam/aiida-core/milestone/18) milestones following the roadmap discussion. Two more critical bugs that should probably be released as [v1.5](https://github.com/aiidateam/aiida-core/milestone/37)
* Last weeks todo's
* The `asyncio` migration work has been merged into develop. Giovanni has identified a potential issue that will need to be investigated [[#4595]](https://github.com/aiidateam/aiida-core/issues/4595)
* Found the bug in the SqlAlchemy migrations of the [new repository implementation branch](https://github.com/sphuber/aiida-core/tree/fix/3445/disk-object-store-repository-rebased). Will add a test case and update the branch. The branch is then ready to be tested against databases, first to check that the migrations work, and second that normal operation works as expected.
* Releases
* `aiida-core==1.5.0`: released by and together with Francisco and Chris
* Pull requests
* Dependencies: update minimum version for `notebook>=6.1.5` [[#4593]](https://github.com/aiidateam/aiida-core/pull/4593)
* CI: Add workflow to run tests against various RabbitMQ versions [[#4579]](https://github.com/aiidateam/aiida-core/pull/4579)
* `CalcJob`: add `additional_retrieve_list` [[#4437]](https://github.com/aiidateam/aiida-core/pull/4437)
* Replace `tornado` dependency with built-in module `asyncio` [[#4317]](https://github.com/aiidateam/aiida-core/pull/4317)
* **[OPEN]** Fix bug in SqlAlchemy migrations [[#4602]](https://github.com/aiidateam/aiida-core/pull/4602)
* **[OPEN]** `CalcJob`: add the option to stash files after job completion [[#4602]](https://github.com/aiidateam/aiida-core/pull/4424)
* To do
* Discuss and merge the calculation job stash PR
* Finalize the new repository implementation and organize the extensive testing by the team

Chris:

* Quantum Mobile: Released v20.11.0a
* See https://quantum-mobile.readthedocs.io/en/latest/releases/both/20.11.0a.html
* The key additions for this release are:

- aiida-core v1.5.0 (in a Python 3.7 environment)
- Jupyter Lab (and NodeJS) installation
- Big-DFT executable and AiiDA plugin
- Abinit executable and AiiDA plugin

* Note, AiiDAlab is not included in this image, due to ongoing changes in its implementation, but is intended for a future release.
* [Simon: Could you explicitly list the changes causing the delays, please?]
* By the changes I mainly mean the ones that we have already discussed about inverting control of dependencies to the individual plugins and use of separate venv's for aiida/jupyter. In the aiida role I have now I hope I have now "solved" this splitting of the envs (plus ensuring Python 3.7). See [[#53](https://github.com/marvel-nccr/ansible-role-aiida/pull/53)]
* Simon: I guess I am confused about the wording "ongoing changes", there have been no changes as of late that I know of that would be blocking; if there are would be good for us to know.
* Chris: well hopefully aiida lab will be out of beta soon, as that tends to suggest there are ongoing changes 😉, but it is mainly that there **has** been a lot of changes, so it was not possible to just include the existing role version.
* This release also follows a major refactoring of the build process, testing and documentation, to provide a more user-friendly experience.
* Full GitHub Actions CI build that takes ~2.5 hours!

* Released disk-objectstore on conda-forge

* PRs awaiting review/revision:
* Plumpy: Add full typing, type checking and test coverage [[#180](https://github.com/aiidateam/plumpy/pull/180)]
* Found/fixed a bug with Fail state (don't think it affects aiida-core directly)
* Nitpick: type your code!
* refactor `ci/` folder [[#4565](https://github.com/aiidateam/aiida-core/pull/4565)]
* Add deletion of nodes in groups [[#4578](https://github.com/aiidateam/aiida-core/pull/4578)]

* Next steps
* v1.6 help out on asyncio, migration issues, and continue archive work (`verdi archive`)
* Rework `verdi config`: proposal in [[#4584](https://github.com/aiidateam/aiida-core/issues/4584)]

Marnik:

* Work on BIG-MAP tutorial: PR's [[#278]](https://github.com/aiidateam/aiida-tutorials/pull/278), [[#282]](https://github.com/aiidateam/aiida-tutorials/pull/282), [[#284]](https://github.com/aiidateam/aiida-tutorials/pull/284) [[#288]](https://github.com/aiidateam/aiida-tutorials/pull/288), [[#290]](https://github.com/aiidateam/aiida-tutorials/pull/290)
* TODO:
* Fix issues raised by Leo (thanks, Leo!)
* Closed PR's:
* core: Add options for transport tasks: [[#4583]](https://github.com/aiidateam/aiida-core/pull/4583)
* core: Fix command for getting config options [[#4587]](https://github.com/aiidateam/aiida-core/pull/4587)
* Open PR's
* core: Docs: Add "How to extend workflows" [[#4562]](https://github.com/aiidateam/aiida-core/pull/4562)
* qe: `PwCalculation`: Add parsing of up/dw Fermi energy [[#622]](https://github.com/aiidateam/aiida-quantumespresso/pull/622)

Simon:

* Helped debug issue related to conda dependencies as part of [[#4317]](https://github.com/aiidateam/aiida-core/pull/4317)
* Participated in AiiDA Roadmap discussion (v1.6, v2.0, v3.0).
* Reviewed pull request on AiiDA core dependency update [[#4593]](https://github.com/aiidateam/aiida-core/pull/4593)
* Setup AiiDAlab k8s cluster for BIG-MAP tutorial
* Updated AiiDAlab QE app in preparation for BIG-MAP tutorial

Giovanni:
* Organisation of BIG-MAP tutorial tomorrow (thanks a lot to Simon, Marnik, Francisco, Seb for help in VM/AiiDAlab preparation, and content!)
* Will run on AiiDAlab on OpenStack -> Suggestion for Simon to organise an explanation of how this works for the future
* Will need to put back the material online for "offline use" - Marnik will coordinate that. Marnik suggests to have "independent modules" so it's easy to organise new tutorials by picking the relevant ones, I like it
* Important: for these short tutorials, important not to lose the perspective of Materials Science (audience wants to learn about that)
* Tested new AiiDA engine, there are some issues, reported in [#4595](https://github.com/aiidateam/aiida-core/pull/4595). As discussed: great if everybody can have a test-set (heavy from a AiiDA point of view, e.g. many workflows, but maybe not too much CPU, so e.g. unconverged parameters) for future testings
* Some issues seen for "intense" use in AiiDA (1.5):
* [#4596](https://github.com/aiidateam/aiida-core/pull/4596) -> Weird error with SQLA+DbAuthInfo! Everything was working a few days ago... Any idea? E.g. Casper?
* [#4598](https://github.com/aiidateam/aiida-core/pull/4598) -> when submitting *many* calculations and making the machine slow
* [#4603](https://github.com/aiidateam/aiida-core/pull/4603) -> Memory leak: after running ~2000 WFs, the 8 daemon workers were still using ~15GB of RAM even if no more operation was needed, and no tasks were in the RMQ queue (restarting the daemons cleared the memory, but we should fix this) -> Chris?

Casper:
- Continuously updating [`aiida-optimade`](https://github.com/aiidateam/aiida-optimade).
- TODOs:
- Need to get back to the REST API POST endpoint PR ([#4337](https://github.com/aiidateam/aiida-core/pull/4337)) as it is very relevant for Flaviano and our efforts in INTERSECT.
- Develop and implement an OPTIMADE DB importer for AiiDA Core.

Francisco:
* Work on BIG-MAP tutorial
* Transfer Calcjob [#4194](https://github.com/aiidateam/aiida-core/pull/4194)
* Fix on dict node [#4577](https://github.com/aiidateam/aiida-core/pull/4577)
* Calcjob stash => review? Yes

Leopold:
* discussed with Flaviano - we'll move the REST API presentation to the ~next AiiDA meeting
* Went through tutorial & added a couple of suggestions
* Worked with Dominik on [aiida-testing](https://aiida-testing.readthedocs.io/en/latest/) - thinking of replacing part of the mock-executable logic with a monkey-patch in AiiDA's execmanager in order to get better error messages
In general - the mock executable can be very useful for CI testing of executables you don't want to compile on CI

Sasha:
* Contributed to aiida-code-registry:
* Merging info into a single file ([#17](https://github.com/aiidateam/aiida-code-registry/pull/17))
* Add CI for updating GitHub pages. ([#18](https://github.com/aiidateam/aiida-code-registry/pull/18))
* Setting '.yaml' as the only accepted extension of the config files ([#20](https://github.com/aiidateam/aiida-code-registry/pull/20))
* Issue: on AiiDAlab AiiDA configuration gets deleted (on high load). Not sure how to reporduce.