# AiiDA Team Meeting 2021-02-10

###### tags: `team meetings`
###### time: 4pm CET

[TOC]

### Present

* Leopold
* Casper
* Francisco
* Giovanni
* Chris
* Sasha

### Catch-up round

*Max. 3 minutes each*

Leopold:
* Created AiiDA [GSOC2021 ideas page](https://github.com/aiidateam/aiida-core/wiki/GSoC-2021-Projects) + [discussion thread](https://github.com/aiidateam/aiida-core/discussions) where students can say hello
* [Memory leak fix merged](https://github.com/aiidateam/aiida-core/pull/4699) - to be tested in the real world + together with chris's new plumpy alpha release
* REST API postgres connections:
On `sqlalchemy` profiles, the number of connections on the postgresql side correctly reflects the settings of the `QueuePool`. By switching to a `NullPool`, the number of permanent connections drops to zero.
Alternative: `QueuePool` with `pool_size=1` and `max_overflow=10` (1 permanent connection)
`django` profiles: different behavior, still to investigate
* I noticed that on `sqlalchemy` profiles the *first* request to, e.g. `api/v4/nodes` requires at least 2 concurrent connections (hangs with `pool_size=1` and `max_overflow=0`) and results in connections being checked out from the queue pool and put back 5 times. Subsequent requests only require one such checkout.

Simon:
* Any items for next NumFOCUS newsletter?
* If released: **1.6 release**
* Supported Marnik with setup of [AiiDAlab cluster for the ABC tutorial](https://aiidalab-abc-tutorial-2021.materialscloud.org/)
* Required substantial improvements to the documentation within the [aiidalab-k8s](https://github.com/aiidalab/aiidalab-k8s) repository.
* Marnik is documenting organization-specific details of the process in Google doc in addition to the public documentation as part of the [aiidalab-k8s](https://github.com/aiidalab/aiidalab-k8s) repository.
* Required minor updates to address deprecated terraform versions and components.
* Otherwise no major issues.
* Dependency management
* Updated the compatibility timeline on https://www.aiida.net/download/
* Reviewed PR for drop of support of Python 3.6 [#4701](https://github.com/aiidateam/aiida-core/pull/4701)
* No longer test the pip 2020 resolve feature (is default) [#4689](https://github.com/aiidateam/aiida-core/pull/4689)
* Reviewed PR on dropping the setuptools upper version constraint [#4725](https://github.com/aiidateam/aiida-core/pull/4725)
* Increase `test-install` timeout (forgot to merge...) [#4697](https://github.com/aiidateam/aiida-core/pull/4697)
* Investigated issues with respect to conda package 1.5.2 ([conda-forge/aiida-core-feedstock#46](https://github.com/conda-forge/aiida-core-feedstock/issues/46))
* Made suggestions on how to avoid failure to update dependencies on conda-forge recipe [aiida-core-feedstock#48](https://github.com/conda-forge/aiida-core-feedstock/issues/48)
* Require `ipython~=7.20` [#4715](https://github.com/aiidateam/aiida-core/pull/4715)
* Reviewed and supported PR to require `sqlalchemy-utils~=0.36.0` [#4724](https://github.com/aiidateam/aiida-core/pull/4724)
* CI:
* Notify slack channel (#dev-aiida-core) on specific failures of the `test-install` GA workflow [#4690](https://github.com/aiidateam/aiida-core/pull/4690)
* Reviewed PR on jenkins docker image [#4714](https://github.com/aiidateam/aiida-core/pull/4714)
* Harmonize matrix dimensions across CI workflows [#4713](https://github.com/aiidateam/aiida-core/pull/4713)
* Investigated issue regarding random test timeouts [#4731](https://github.com/aiidateam/aiida-core/issues/4731)

Chris:
- Passed Casper in contributions 😝 (https://github.com/aiidateam/aiida-core/graphs/contributors)
- Merged PRs
- ♻️ REFACTOR: verdi export/import -> verdi archive [[#4710](https://github.com/aiidateam/aiida-core/pull/4710)]
- ♻️ REFACTOR: `.ci/` folder [[#4565](https://github.com/aiidateam/aiida-core/pull/4565)]
- 🔧 MAINTAIN: drop setuptools upper pinning (#4725) [[#4725](https://github.com/aiidateam/aiida-core/pull/4725)]
- 🐛 FIX: Standardise transport task interrupt handling [[#4692](https://github.com/aiidateam/aiida-core/pull/4692)]
- As an example, this fixes a known bug, whereby the upload task could not be cancelled via `CTRL-C` in an ipython shell.
- May fix more?
- ✨ NEW: Add `verdi database summary` [[#4737](https://github.com/aiidateam/aiida-core/pull/4737)]
- Prints a summary of the count of each entity
- Plumpy:
- 🔧 MAINTAIN: Remove frozendict dependency (#204)
- 👌 IMPROVE: Terminal states do not reference process (OPEN)
- released https://pypi.org/project/plumpy/0.18.5a0/ for testing
- Should reduce memory for daemons
- OPEN PRs (ready for review):
- 📚 DOCS: Add `BaseRestartWorkchain` how-to [#4709]
- ♻️ REFACTOR: configuration management API and CLI [#4712]
- 🧪 TESTS: Add pytest `requires_rmq` marker [#4739]
- Working on now:
- Engine asyncio issues

Casper:
- Looked into the issues of PostgreSQL connections ([#3867](https://github.com/aiidateam/aiida-core/issues/3867), [#4374](https://github.com/aiidateam/aiida-core/issues/4374)), possibly found a (temporary) solution for Materials Cloud to use: [PgBouncer](https://www.pgbouncer.org/).
- Fixed an issue with `aiida-optimade`, where it would alter the DB (updating the `optimade` extras) for any simple GET request [aiida-optimade#198](https://github.com/aiidateam/aiida-optimade/pull/198).

Francisco
* Currently working on the migration of the attributes of old InlineCalculation nodes ([issue 4401](https://github.com/aiidateam/aiida-core/issues/4401)). A hand with alembic and SQLAlchemy?
* AiiDA presentation for the [love data week](https://memento.epfl.ch/event/love-data-week-2021-at-epfl/).
* Next: release v1.6

Sasha
* Worked on updating the aiida-prerequisites Docker image
* Simplified the conda setup [#25](https://github.com/aiidateam/aiida-prerequisites/pull/25)
* Add start ssh-agent at user's startup [#26](https://github.com/aiidateam/aiida-prerequisites/pull/26)
* New release 0.3.0 [#27](https://github.com/aiidateam/aiida-prerequisites/pull/27)
* Update the base image in aiida-core Docker [#4738](https://github.com/aiidateam/aiida-core/pull/4738)
* Reviewed a couple of PRs:
* verdi database summary by Chris [#4737](https://github.com/aiidateam/aiida-core/pull/4737)
* verdi export/import -> verdi archive by Chris [#4710](https://github.com/aiidateam/aiida-core/pull/4710)
* verdi group remove-nodes`: Add warning when nodes are not in Group` [#4728](https://github.com/aiidateam/aiida-core/pull/4728)
* Question:
* Can we add a feature such as `group.node_pks`? This would allow to simplify and (I believe) to speed working with groups:
* `query.append(Node, with_group='group', filters={'id': {'in': node_pks}}, project='id')`
* `set(group.node_pks).intersect(node_pks)`
* `set(group.node_pks).difference(set(node_pks))`
* Can we add python hashing (maybe being equal to the node pk)

Giovanni
* [PR](https://github.com/aiidateam/aiida-core/pull/3787) on SSH deadlocks ready: just decide if it's ok to break this type of backward-compatibity + merge (Jenkins tests failing though?)
* [SqlAlchemy-utils version](https://github.com/aiidateam/aiida-core/pull/4735) updated; also suggested to drop frozendict in plumpy (Chris removed it) and to [fix reentry](https://github.com/DropD/reentry/pull/58) for py3.10 (collection.abc imports). When this is done and dependencies are updated, we should not have any more warnings in the code because of these old imports. @Simon: now (or soon) it might be good to check if [this wiki section on problematic dependencies](https://github.com/aiidateam/aiida-core/wiki/AiiDA-Dependency-Management#problematic-dependencies) can be updated
* [Fix hanging direct scheduler+ssh](https://github.com/aiidateam/aiida-core/pull/4735): in the end the issue could be fixed with two parentheses in the right place :-) I also managed to add a test: ready to review and merge.
* Worked with Leo in [reducing the number of warnings in the tests](https://github.com/aiidateam/aiida-core/pull/4719). Actually, fixed a couple of bugs (e.g. fixing some files that might have remained open during execution; also, now most `verdi daemon` commands try to return non-zero error codes if the operation does not succeed). Most of the remaining ones are [asyncio ones (great to check (@Chris?))](https://github.com/aiidateam/aiida-core/issues/4722) as often they are pointing to actual bugs. (And I reactivated some ResourceWarnings - some are still there, but it was important to spot issues).
* Tutorial in July -> will probably still be in remote. To plan organisation "soon" (in a month?).

Marnik:
* Work on tutorial (updated to `aiida-quantumespresso` v3.3.0)
* verdi group remove-nodes`: Add warning when nodes are not in Group` [#4728](https://github.com/aiidateam/aiida-core/pull/4728)
* Reviewed `CalcJob`: add the option to stash files after job completion [#4728](https://github.com/aiidateam/aiida-core/pull/4424#event-4313188139)
* Seems in good state, I agree with Seb's final remarks; @Gio Seb still asked for sign-off from your end before we merge.

### Specific Agenda Items