# AiiDA Team Meeting 2020-11-04

###### tags: `team meetings`
###### time: 4pm CET

[TOC]

### Present

* Chris
* Marnik
* Leopold
* Giovanni
* Francisco
* Simon
* Casper
* Sasha
* Sebastiaan
* Espen

### Catch-up round

*Max. 3 minutes each*

Chris:
* Archive refactor:
* The refactoring goal is to pave the way for the implementation of a new archive format ([ aiidateam/AEP005](https://github.com/aiidateam/AEP/pull/21))
* Three abstract+concrete interface classes are defined; writer, reader, migrator, which are **independent of the internal structure of the archive**. These classes are used within the export/import code (tested performance regression).
* The code in `aiida/tools/importexport` has been largely re-written, in particular adding `aiida/tools/importexport/archive`, which contains this code for interfacing with an archive, and **does not require connection to an AiiDA profile**.
* The export logic has been re-written; to minimise required queries (faster), and to allow for "streaming" data into the writer (minimise RAM requirement with new format). It is intended that a similiar PR will be made for the import code.
* A general progress bar implementation is now available in `aiida/common/progress_reporter.py`. All corresponding CLI commands now also have `--verbosity` option.
* Additionally, group extra export has been added and the code is all fully typed (mypy compliant)!
* Merged PRs (thanks to Leo for the reviews):
* Refactor export archive ([#4448](https://github.com/aiidateam/aiida-core/pull/4448) & [#4534](https://github.com/aiidateam/aiida-core/pull/4534))
* Refactor import archive ([#4510](https://github.com/aiidateam/aiida-core/pull/4510))
* Refactor migrate archive ([#4532](https://github.com/aiidateam/aiida-core/pull/4532))
* Add group extras to archive ([#4521](https://github.com/aiidateam/aiida-core/pull/4521))
* Refactor cmdline progress bar ([#4504](https://github.com/aiidateam/aiida-core/pull/4504) & [#4522](https://github.com/aiidateam/aiida-core/pull/4522))
* Next steps:
* Release in v1.5.0
* Integrate in new object store ([#4345](https://github.com/aiidateam/aiida-core/pull/4345))
* Propose new format for DB entity storage (probably jsonlines vs sqlite)
* Move CLI commands to `verdi archive`?
* Work with aiidalab in quantum mobile ongoing. [aiidalab#138](https://github.com/aiidalab/aiidalab/pull/138) moves plugin imports to extras, and so will be more flexible for integration with QM. Will assess/implement in next few days.
* Slack vs https://github.com/aiidateam/aiida-core/discussions. In the same way we are now using hackmd, I feel it would be beneficial to move some discussions out of the (private) AiiiDA#general space (especially when it is discussing the resolution to a mailing list question!)

Leopold:
* To discuss (archives):
* ``--extras-mode``, ``groups-mode`` for ``verdi import`` [#4383](https://github.com/aiidateam/aiida-core/issues/4383) - no reply from Sasha yet, so I propose to postpone
* special treatment of node attributes/extras in archive file - was there a particular reason?
* To discuss (CLI):
* Reconsider default user filter on CLI `list` commands [#4525](https://github.com/aiidateam/aiida-core/issues/4525)
* cli-wide `-v/--verbosity` option [PR](https://github.com/aiidateam/aiida-core/pull/3896): let verbosity option set the level of the AIIDA_LOGGER (and thus of all dependent loggers for that python process)
* To discuss (AEP):
* [draft status PR](https://github.com/aiidateam/AEP/pull/23) - should "draft" include an endorsement from the AiiDA team or should it be nothing more than a snapshot of a PR?

Sebastiaan
* Released `kiwipy==0.7.0`
* **[RabbitMQ SSL]:** Finally fixed bug to allow SSL connections to RabbitMQ server. Will require update of `topika` library. This will be obsolete once the migration to `asyncio` is complete, but since this won't be until `aiida-core==1.6.0`, I wanted to make a patch version `v1.4.3` so that `v1.4` and `v1.5` have SSL capability for RabbitMQ as well. Since `topika` hasn't been worked on in a long time, it's taking quite some time to first get the CI up and running which was still running on Travis. (Side note, us moving to GHA was a good call, Travis is a real mess right now for OSS.)
* **[Migrating to `asyncio`]:** Worked with [@muhrin](https://github.com/muhrin) and [@unkcpz](https://github.com/unkcpz) on the migration from `tornado` to `asyncio`. Critical bug was discovered during production testing and the problem was identified. A solution is currently being tested in [this PR](https://github.com/aiidateam/plumpy/pull/177). We are discussing on final details and hope to merge this soon into `plumpy` and make a new release. Then we can update the [PR in `aiida-core`](https://github.com/aiidateam/aiida-core/pull/4317) and perform the final testing. The aim is to release this feature with `aiida-core==1.6.0`.
* **[New repository:]** Improved the repository/database migration to reduce the memory footprint by migrating the repository in 256 chunks, one chunk per top level shard. Unit tests are passing, but test on production SqlAlchemy database failed. However, on a Django database it worked fine. Currently investigating. Once fixed, this branch should merely be updated to include the refactoring of @chrisjsewell to the export/import code. Then it is ready for extensive testing, where I will require from the team to test the branch on their existing databases.
* Merged PRs
* Fix `UnboundLocalError` in `aiida.cmdline.utils.edit_multiline_template` [[#4436]](https://github.com/aiidateam/aiida-core/pull/4436)
* Open PRs:
* `CalcJob`: add `additional_retrieve_list` [[#4437]](https://github.com/aiidateam/aiida-core/pull/4437) (Ready for review)
* `CalcJob`: add the option to archive files after job completion [[#4424]](https://github.com/aiidateam/aiida-core/pull/4424) (Needs discussion on API, next meeting is being scheduled)

Simon:

* **Usability**: The installation flow for Python 3.9 appears to be working; in the process of preparing [draft PR #4301](https://github.com/aiidateam/aiida-core/pull/4301) for review.
- Question: The CI workflow is currently very slow for py39, because some dependencies must be compiled; delay testing as part of `ci-code`?
* **Docs**: The revision of the "Get started flow" [#4496](https://github.com/aiidateam/aiida-core/pull/4496) is completed and currently under review.
* **Process**: Participated in discussion on AEP "draft" status description [AEP:#23](https://github.com/aiidateam/AEP/pull/23#pullrequestreview-523324204)
* Reviewed/contributed: [#4536](https://github.com/aiidateam/aiida-core/pull/4536), [#4466](https://github.com/aiidateam/aiida-core/pull/4466)

Casper:
- Add tests for Hill notation in `get_formula()` [#4536](https://github.com/aiidateam/aiida-core/pull/4536).

Francisco:
* Preparing the release of `v1.5.0` [#4536](https://github.com/aiidateam/aiida-core/pull/4536).
* **Usability:** Full Type Count via RestApi [#4277](https://github.com/aiidateam/aiida-core/pull/4277) (WIP, problems with how the API deals with non-registered processes)
* **Docs:** closing up the work of the last weeks for the release.
* *"Get started flow"* ([#4496](https://github.com/aiidateam/aiida-core/pull/4496)) - who is reviewing it?
* *"Extend Workflows"* ([#3993](https://github.com/aiidateam/aiida-core/pull/3993), [#1927](https://github.com/aiidateam/aiida-core/pull/1927)) - is this in process?
* *"Topics: Data types"* ([#4469](https://github.com/aiidateam/aiida-core/pull/4469)) - changes requested.

Marnik:

* Open PR's:
* **Usability**: Add `--delete-nodes` flag to `verdi group delete` ([#4425](https://github.com/aiidateam/aiida-core/pull/4425)).
Question on deprecation of `--clear` flag ([#4357](https://github.com/aiidateam/aiida-core/pull/4357)).
* **Usability**: Add `verdi group move-nodes` command ([#4428](https://github.com/aiidateam/aiida-core/pull/4428#pullrequestreview-510337360)).
Question on behaviour or removing nodes from a `Group`: currently there is no warning in case the node the user is trying to remove is not in the `Group`. ([#4438](https://github.com/aiidateam/aiida-core/issues/4438))
* **Docs**: Move data types to Topics section ([#4469](https://github.com/aiidateam/aiida-core/pull/4469)).
* Closed PR's:
* **Docs**: Add `verdi process status` command to tutorial ([#4508](https://github.com/aiidateam/aiida-core/pull/4508))
* `aiida-quantumespresso`: Remove parser information from output_parameters ([#597](https://www.theguardian.com/law/2020/oct/26/wisconsin-mail-in-ballots-supreme-court-election-day))

Giovanni:
* **Docs**: Reviewed [Marnik's PR](https://github.com/aiidateam/aiida-core/pull/4469)
* **Workflows**: Interacting with Miki Bonacci and Nicola Spallanzani, providing feedback for the aiida-yambo automatic convergence workflow
* **Core libraries**: [disk-objectstore](https://github.com/aiidateam/disk-objectstore): I will add a changelog to [this PR](https://github.com/aiidateam/disk-objectstore/pull/103) so it can be merged, v0.5 can be released and Sebastiaan's branch can depend on it.
* **Coordination/outreach**: AiiDA and Materials Cloud presented (me + Sebastiaan) to the new TREX CoE
* **Organisational note**: Reminder (for the EPFL team) of the BIG-MAP meeting on Dec 2, everybody please mark in your calendar

Sasha:
* **Docs**: scheduler plugin creation instructions ([#4476](https://github.com/aiidateam/aiida-core/pull/4476))
* **Plugins**. Worked together with Carlo Pignedoli ([#18](https://github.com/nanotech-empa/aiida-nanotech-empa/pull/18)) on avoiding using `Int` objects in the workchain metadata and on adding small examples for the automatic testing ([#19](https://github.com/nanotech-empa/aiida-nanotech-empa/pull/19)).
* Modify aiidalab for a better compatibility with Quantum Mobile ([#138](https://github.com/aiidalab/aiidalab/pull/138)).

### Specific Agenda Items