# AiiDA Team Meeting 2021-04-21 ###### tags: `team meetings` ###### time: 4pm CET [TOC] ### Present * Sebastiaan * Marnik * Giovanni * Chris * Casper * Leopold * Ninad * Francisco * Simon * Espen ### Catch-up round *Max. 3 minutes each* Participant: * Item Leopold: * GSOC 2021: Welcome, Ninad Bhat! 🎉 * codecov security incident: [summary and to-do items for aiidateam org](https://docs.google.com/document/d/1PuiWxxJ3LmCP3xLZpSn7z-pCXVlbdfUbqwxxGdJVguA/edit?ts=607b44bc) (you may want to do the same for other orgs & your own repos) * [Daemon "burst" issue](https://github.com/aiidateam/aiida-core/issues/4841): after a couple of minutes of submitting calculations, the interaction of the daemon with rabbitmq becomes more and more "burst-like" with long periods of inactivity. This significantly slows down job submission & result retrieval. * export with aiida 1.6.1: ```Export: RuntimeError: File size unexpectedly exceeded ZIP64 limit``` werid, since I estimated archive size ~2GB * Since we've been discussing about ansible roles vs conda recipes for building codes in Quantum Mobile, and how to best get code developers involved, here an (ongoing) example of adding a code (Sirius) to conda-forge * [I reached out to developers](https://github.com/electronic-structure/SIRIUS/issues/621#issue-859996282); some dependencies were missing on `conda-forge` * [they prepared an initial recipe](https://github.com/electronic-structure/SIRIUS/issues/621#issuecomment-821958782) for one of the dependencies * [I created conda-forge PR and gave them access to my fork](https://github.com/conda-forge/staged-recipes/pull/14605#issuecomment-822337099) * [they fixed the build](https://github.com/conda-forge/staged-recipes/pull/14605) and made the dependency (SpFFT) available via conda * We'll continue in a similar vein for the other missing dependency (spla) and for sirius itself Marnik: * **Agenda:** Louis Ponet show-and-tell. 🎤 * **Agenda:** AiiDA July tutorial. 🙌 * **PR:** CI: install default `postgresql` instead of specific version [[#4860](https://github.com/aiidateam/aiida-core/pull/4860)] -- *Merged!* 🎉 * **PR:** CLI: Use DB `proxy_command` for `verdi calcjob gotocomputer` [[#4761](https://github.com/aiidateam/aiida-core/pull/4761)] -- *Merged!* 🎉 * **PR:** DOCS: Add "How to extend workflows" section [[#4562](https://github.com/aiidateam/aiida-core/pull/4562)] -- *Merged* 🎉 Sebastiaan * Participated in the WorkflowsRI Community summit of April 7th * [WorkflowsRI](https://workflowsri.org/) is a community of workflow management system developers and users * Joined the breakout session on Workflow Systems' Interface with Legacy and Emerging HPC Software and Hardware Stacks * [This video](https://www.youtube.com/watch?v=jz6FZC48G1s&list=PLAtmuqHExRvOfsFTDdtGDrhovi1oQWLgP&index=4&ab_channel=workflowsRIProject) contains an introductory video and here [are notes with a summary of the breakout session](https://docs.google.com/document/d/1H2fTv61vcGtyfvy0z4GQXNBPzmnmNyC_H67uY51ROJs/edit) * One big point that was raised (by me, but also some others) is that authentication at HPC centers is moving towards 2FA and poses problems for WfMS. * One person mentioned that it is not just problematic for automated systems, but also for users in general. If every center uses their own solution, it becomes difficult to manage multiple resources for a user * NERSC is phasing out NEWT (which was their REST interface) and is actually considering adopting FireCREST from CSCS. Their representative said that they were looking for solutions and found FireCREST to be the only real candidate. * Support for LLNL is ongoing, contact person is Nathan Keilbart, post-doc in group of Brandon wood * They are actually using AiiDA with `aiida-quantumespresso` for scientific work on their systems now * He has found colleagues that are interested in running NWChem (I have brought them into contact with Conrad Johnston who is helping them add missing features) * **PR**: [IMPROVE: Add `account` option to `LsfScheduler` (#4832)](https://github.com/aiidateam/aiida-core/pull/4832) - *Merged* * **PR**: [Dependencies: update sqlalchemy dependency requirements](https://github.com/aiidateam/aiida-core/pull/4855) - Waiting for [conda forge release](https://anaconda.org/conda-forge/sqlalchemy-utils) * **PR**: [New repository implementation](https://github.com/aiidateam/aiida-core/pull/4345). Added the final piece, linking a repository to a database. The tests are failing because the fixtures do not yet perform the linking operation, but the functionality should be working. Testing is much appreciated! * **PR**: [Respect nested output namespaces in `Process.exposed_outputs`](https://github.com/aiidateam/aiida-core/pull/4863) - Waiting for review: tested in the wild by Espen and JPChico and confirmed to solve the issue * **PR for `aiida-common-workflows`**: [CI: add testing for common relax input generator and input spec](https://github.com/aiidateam/aiida-common-workflows/pull/199): added some much needed testing of workchains but tests are failing because at some point the connection to RabbitMQ is refused. This seems to be completely unrelated to the actual test, but rather a case of overloading. If I run the tests individually there is no problem. I suspect it is because we recreate the `Connection` in every test. I wonder why we don't have this problem in `aiida-core` tests though. Can't imagine we have less connection creations. Anyone any ideas? Francisco * Outstanding issues/PRs to discuss: * **Addition of `remotecat` command ([#4861](https://github.com/aiidateam/aiida-core/pull/4861)):** it relies on the ssh access methods of `RemoteData` which were going to be subject of change [Aproved in principle]. * **Double quotes in AiiDA scripts for env variables ([#4836](https://github.com/aiidateam/aiida-core/issues/4836)):** is not super consistent with how we handle codes, for example (requiring absolute path instead of relative), although those get stored in a node. Still, it is an "incomplete" reference in the data that gets stored. * **Restart failed steps of calcjob submission ([#4857](https://github.com/aiidateam/aiida-core/issues/4857)):** similar to how workflows restart steps on their outline (using caching?). Worth persuing? Feels like a lot of work for marginal gains (parsing and transport?). * Argument: *"Particularly for HPC users with finite usage budgets, resubmitting entire calcjobs may not be viable."* * **Consumption of CPU in connection errors ([#4827](https://github.com/aiidateam/aiida-core/issues/4827)):** we discussed this morning and initially have no idea what this might be about, but seems like an important issue to prioritize. Any others have experienced this? Could it be related to new async system? Giovanni * Who should take care of the notifications of dependency security issues? * Time of future meetings? Simon: * Mostly worked on AiiDAlab (app registry) * Resolved issue that prevented the installation of aiida-core with all extras ([#4851](https://github.com/aiidateam/aiida-core/pull/4851)) * Created tool to analyze dependency constraints and point out where we are no longer compatibility with new releases ([#4852](https://github.com/aiidateam/aiida-core/pull/4852)) * Reviewed PR on updating sqlalchemy (related) dependencies ([#4855](https://github.com/aiidateam/aiida-core/pull/4855)) * Removed slack integration and secrets from aiida-core repository secrets (codecov hack) ([#4862](https://github.com/aiidateam/aiiad-core/pull/4862)) * Reviewed misc. PRs: [#4838](https://github.com/aiidateam/aiida-core/pull/4838), [#4854](https://github.com/aiidateam/aiida-core/pull/4854) * Any NF updates? Chris: * Michele mentioned issue of migrating config to v1.6.1 (config version 5) stripping `broker_` keys. Known issue with "reverse migration", but she did not give the impression that what she had done. * Released new Quantum Mobile, also looking to move to Conda but some issues with `conda activate env` * Ongoing work with aiida-lammps and Sandvik Coromat (JPchico et al): https://github.com/aiidaplugins/aiida-lammps/discussions/21 * Nothing else of significance (the calm before the storm) Casper: - Have been working extensively on the [OPTIMADE gateway](https://github.com/Materials-Consortia/optimade-gateway). I.e., nothing related to AiiDA. - Did get a notice about security issues with `pyyaml` and `pygments` Python packages? - Notice: I'll be leaving the group, and CH, by the end of July. :sob: I'll be moving to a position in SINTEF Industry in Trondheim, Norway, where I'll continue to work with similar EU projects and possibly assist Espen Flage-Larsen in Oslo with AiiDA-specific (and other) things (if he asks nicely). ### Specific Agenda Items * Event for AiiDA developers and power users in second half of 2021 * **Louis Ponet show-and-tell**: Louis, one of the newest additions to THEOS, has apparently written some rudimentary scripts/code for running calculations for his work during his PhD. He thought it might be nice to do a little demo of some of the features, since there are some usability aspects that we could apply to AiiDA. Should I invite him for the next meeting? He said it should only take 10 mins. * **AiiDA july tutorial**: * [Advertisement](https://docs.google.com/document/d/1tk90RDwBGYXYNio2zeSceARWWwHFEwdlUppoJGK7Ktk/edit#) ; [Application form](https://docs.google.com/forms/d/1jM6KoBQpbFjZTeXUMmatQICwrnsUuAqnjqGxu7wgUs8/edit) ; [Schedule](https://docs.google.com/spreadsheets/d/1taYiFCfnmBtuydd-8Yc_ifSuyOZ-Om4OcbnBlwastPU/edit#gid=64673141) ; [Website](https://docs.google.com/document/d/1vit5svzvh3Oo6zo8TbYLEBFCm2V6Fi87Wb43qtzgdIg/edit#) * Idea: Virtual poster session via gather.town * Idea: Virtual social event -> Scientific speed dating? * Gather.town cost ~ $300 = acceptable? (might be able to get a better deal for academia.) * Francisco and I will make sure the module structure is prepared by **14th of May**, and will open issues for each module that you can assign yourselves to. * We ask that everyone checks their module and makes corrections where necessary (Open PR linked to issue) by the AiiDA meeting on the **1st of June**.