AiiDA coding week 18-22 Nov 2024

# AiiDA coding week 18-22 Nov 2024 ###### tags: `event` [toc] ## Information ### Location The coding week will take place from 18-22 November 2024, at the Central Résidence hotel in Leysin: https://hotelcr.ch/home Address: Route du Belvédère 20, CH-1854 Leysin, Switzerland Nearest train station: Leysin-Feydey For participants travelling from outside of Switzerland, there are good train connections via Paris/Milan (see https://www.sbb.ch/en), the closest airport is Geneva. ### Stay We'll take care of booking the hotel + conference rooms. The cost per night per person is: CHF 169.00 per person in single room CHF 139.00 per person in double/twin room I've been in touch with each of you to confirm if you want a single room or not. Two important points: * The hotel cannot send separate invoices for everyone, instead they will ask each participants to pay separately upon arrival. **If you need to have an invoice beforehand, an exception might be possible. Let me know if this is the case.** * If you want to still change between a single or double room **let me know by Nov 1st**. ### Schedule A preliminary version of the schedule can be found here: https://docs.google.com/spreadsheets/d/1iTZIC_3vX_ONKqPJwa9NRj58bvUerNKgnrBc5GNNvjM/edit?gid=64673141#gid=64673141 ### Social Dinner Order: 1. Cheese fondue * Marnik * Giovanni * Miki * Kristjan * Elliott * Julian * Xing * Carlo * Timo 2. Chicken-cheese quesadilla * Edan * Andrés * Sasha * Riccardo * 3. [Porc Milanese](https://duckduckgo.com/?q=pork+milanese&atb=v406-1&iax=images&ia=images) * 4. Not joining: * Ali * Adrian ### Coding projects Each project needs to have a clear milestone that should be possible to reach by the end of the week. At the beginning of the week the milestone is presented to everyone in a group session and at the end the progress is reported (What has been achieved and what not). At the end of every day every group project can report problems they have to the whole group so people from other group can maybe help or give feedback to solve them. We also separated the different projects into groups that could work in the same room since they do similar things and might need some exchange. - workgraph group (Xing, Natasha, Julian, Sasha, Junfeng) -> breakout room - Move PythonJob out from aiida-workgraph (_Xing_) - Drop `node-graph` from aiida-workgraph (_Xing_)? - Create Wannier90 workflow using WorkGraph (_Xing_, Natasha and Junfeng) - Missing tasks for the stable release version 0.5 (_Xing_, Julian) - Implement some workflow using WorkGraph, either from CP2K, or other workflow from EMPA (_Sasha_, Xing) - quantumespresso/atomistic group (Marnik, Timo, Michael, Miki, Ifeanyi) -> breakout room - Solving various issues in aiida-quantumespresso, aiida-pseudo, aiida-common-workflows (_Marnik_, Timo, Michael) - AiiDA-QE - (Timo) fixes do already exist for the following issues, just need to open PRs: - [#821](https://github.com/aiidateam/aiida-quantumespresso/issues/821), [#939](https://github.com/aiidateam/aiida-quantumespresso/issues/939), [#944](https://github.com/aiidateam/aiida-quantumespresso/issues/944), [#976](https://github.com/aiidateam/aiida-quantumespresso/issues/976) - Finalize the PDOS PR [#764](https://github.com/aiidateam/aiida-quantumespresso/pull/764), it's just the failing test - Get [#1011](https://github.com/aiidateam/aiida-quantumespresso/pull/1011) merged - (Timo): Maybe spend one morning to finalize the previous ones, and get the following small ones done: - [#1035](https://github.com/aiidateam/aiida-quantumespresso/issues/1035), ... - Related to usability: - [Extend protocol concept to CalcJob classes](https://github.com/aiidateam/aiida-quantumespresso/issues/947) - Adding `tools` to `WorkChain`s: would fix [merging of trajectories](https://github.com/aiidateam/aiida-quantumespresso/issues/937#issuecomment-1566076506) and [interrupted bands (discuss with Andres)](https://github.com/aiidateam/aiida-quantumespresso/issues/1025) - [Automatic setup of QE](https://github.com/aiidateam/aiida-quantumespresso/issues/1037) - Robustness - [New PwRelax logic](https://github.com/aiidateam/aiida-quantumespresso/pull/952) - [Improved handling of SCF convergence](https://github.com/aiidateam/aiida-quantumespresso/pull/987) - AiiDA-CWF - [Improved handling of overrides](https://github.com/aiidateam/aiida-common-workflows/issues/238) - Expose the inputs of the common relax interface to the actual common WorkChains, e.g. EOS. Try to fix/continue the [code agnostic relax WC](https://github.com/aiidateam/aiida-common-workflows/pull/261) (also addresses the overrides before) - Review `CommonBandsWorkChain`s - common phonons by Lorentzo Bastonero - Lower priority: Add interface for `CommonDOSWorkChain` - Put `aiida-atomistic` into production (_Miki_, Marnik) - Query of properties - Replace old StructureData type in`aiida-quantumespresso` and `aiida-pseudo` (basically done, but missing test migration) - Deprecate the old `StructureData` in `aiida-core` + mainly test migration - Consolidate the docs + define clear guidelines for plugin migration - Complete the properties which will be supported + complete interface with ASE and Pymatgen (no loss of information) - Checking the overlaps/issues with the [pydantic AiiDA migration](https://github.com/aiidateam/aiida-core/pull/6255) (see above) - Will work on aiida-quantumespresso, Miki knows what (Ifeanyi, Miki): NEB - miscellaneous group (Edan, Kristjan, Alex, Sasha, Carlo, Daniel, Andreas, Julian, Jason, Ali) -> main room - Finish migration to pydantic into AiiDA (Edan) - [ORM: Use pydantic to specify a schema for each ORM entity](https://github.com/aiidateam/aiida-core/pull/6255) - Bring new aiida-restapi up to date so we can replace old one (_Kristjan_, Alex) - [AiiDA >= 2.5 support](https://github.com/aiidateam/aiida-restapi/issues/70) - [Implement missing endpoints](https://github.com/aiidateam/aiida-restapi/issues/69) - AiiDAlab (Sasha, Carlo, Daniel Hollas, Andres, Edan) - Merging [lazy-loading PR](https://github.com/aiidalab/aiidalab-qe/pull/802) - Integrate weas-widget to AiiDAlab. - Migrate to ipywidgets 8.x - Migrate to python 3.11, include AWB in base image - Working on demo server - AiiDA website (*Edan*, Julian) - Revamp the website! (POSTPONED) ~~- The extent of this task is to be determined~~ ~~- Might be content-focused only~~ ~~- Great if tech-stack is also addressed/standardized~~ - Refactoring engine parts where the inheritance are over-used (*Jason*, Ali, Alex) - The inherit pattern are over-used in engine part and lead to messy code. - Three of us can pick one class each and refactoring to composition and switch in turn to review to make sure the style is in agreement. - I personnally like composition way more than inheritance, here is [a guidance](https://python-patterns.guide/gang-of-four/composition-over-inheritance/) on how to achieve it. - Finish incomming changes to support asynchronous Trasnport plugin (Ali) - [link to the fork](https://github.com/aiidateam/aiida-core/compare/main...khsrali:aiida-core:async-without-cwd) ### Discussion sessions: These sessions will happen at a specific day and time. They will be written out on the spreadsheet and announced each morning. The schedule for the meetings will decided on site. - (_Xing_, Natasha, Edan, Julian, Giovanni, Alex, Marnik) Feedback session for people having used workgraph - (_Jason_) Presentation of ideas on refactoring kiwipy to replace RabbitMQ to reflect on the design decisions. RabbitMQ is used in AiiDA to create an asynchronous communication between the different processes (daemon and workers) so AiiDA can submit multiple jobs and asssess their state in parallel. While we need a message broker + queueing system due to Python's GIL, RabbitMQ is not a suitable solution for AiiDA as it contains a rich number of features that we do not need while not providing features that we do need like a priority queue system. - AiiDA responsibilities team meetings: Each team of one responsibility goes through the issues that is part of their responsibility (marked by the topic), discuss how the issue should be solved and assigns it to one the team members - Sound public APIs for core/developers/plugin developers/users. Decide which should be in the outermost API, which should be the abstract methods for plugin write. #### Misc - hashlib switch algo: https://github.com/aiidateam/aiida-core/issues/6611 ## Notes for organizers * Check with Nadja/hotel how to arrange separate bills, since we have to use multiple sources of funding and external people can provide their own. Please ignore the part below for now, once it is fleshed out you will find here a link to a spreadsheet. ### Participants From the MSD group: * Giovanni * Alex * Julian * Ali * Marnik * Edan * Jason * Xing * Miki * Timo * Michael * Natasha * Kristjan (13) Externals: * Sasha (Empa) - Probably (from Tue) * Andres (Empa) - Probably (3 days) * Carlo (Empa) - Maybe 1 day only * Daniel Hollas - * Sebastiaan - maybe he decides last minute and comes 1, max 2 days * Ifeanyi (Parma) - Yes * Adrian (Parma) - Yes * Pietro Bonfa' (Parma) - Maybe * Elliott Kasoar (STFC UK) - yes * Antimo and his group (Trieste) - Maybe ## Wrap-up [MBx] Please write down what has been accomplished in the past week. - Updated `qe-tools` to latest Python versions, provided continuous deployment and did the v2.3.0 release. - Worked on [restructuring `qe-tools` PR of Guillaume from Matgenix](https://github.com/aiidateam/qe-tools/pull/80). First working version of new parsing API + basic documentation and design notes. - Full test of the AiiDAlab QE app and AiiDAlab demo server toward open release, several fixes reported in the GitHub project QE: - Fixed parsing issues for SIRIUS and discussed future design decisions on parsing and moving it to qe-tools - Implement a CLI command for a quick setup of a QE code, similar to `verdi presto` JG: - Resolved some issues for stable release: Implemented shorter inputs/outputs syntax for WorkGraph task specification ([PR 345](https://github.com/aiidateam/aiida-workgraph/pull/345)), improvements to the visualization, e.g. being able to specify the namespace depth for task visualization ([PR 346](https://github.com/aiidateam/aiida-workgraph/pull/346), [PR 347](https://github.com/aiidateam/aiida-workgraph/pull/347)), fixed a bug when running `aiida-shell` through WG ([PR 351](https://github.com/aiidateam/aiida-workgraph/pull/351)). Quite some tasks remain though, for the stable release ([see here on GH](https://github.com/orgs/aiidateam/projects/10/views/1?query=sort%3Aupdated-desc+is%3Aopen&sortedBy%5Bdirection%5D=asc&sortedBy%5BcolumnId%5D=Assignees)) - Created [PR](https://github.com/aiidateam/aiida-core/issues/6435) for `get_size_on_disk` method of `RemoteData`. - Looked into issues raised by Zisen Liu for `contains` operator of PSQL ([PR 6617](https://github.com/aiidateam/aiida-core/pull/6617), still WIP), will continue with him on this. MBo: - Improving of the new `StructureData` API, mainly queries, kind generation, immutability robustness/warnings and relative documentation (see closed issues [here](https://github.com/aiidateam/aiida-atomistic/issues?q=is%3Aissue+is%3Aclosed), label is `coding-week`) - Started: migration of `aiida-core`, `aiida-quantumespresso`, `aiida-pseudo`, submission examples and migration guide for plugin developers - `aiida-atomistic` documentation is [here](https://mikibonacci.github.io/aiida-atomistic/structuredata) Edan: - Merged MVC + lazy-loading PR to aiidalab-qe 🎉 - Picked up Pydantic PR from Sebastiaan - Discussed with Alex and Kristjan - Currently in testing phase - Started working on list of tasks for AiiDAlab demo server Riccardo (workgraph): - try to port a part of a real world messy workchain to workgraph to see what are the difficulties to do the porting and have a feedback. (https://github.com/rikigigi/aiida-QECpWorkChain/blob/test-workgraph/aiida_QECpWorkChain/workflow3.py) Found: to_context api improvement, existing builder re-usage, would be very nice to have functional-like operations on list with tasks (for example Map). - https://github.com/aiidateam/aiida-workgraph/issues/355 - https://github.com/aiidateam/aiida-workgraph/issues/356 - https://github.com/aiidateam/aiida-workgraph/issues/357 Alex (new REST API): - Implemented most endpoints that were supported by the old REST API in the new one. To implement further endpoints Kristjan - Migrated the REST API to work with latest aiida-core (https://github.com/aiidateam/aiida-restapi/pull/75) - Looking into Sebastiaan's Pydantic PR for all ORM classes (https://github.com/aiidateam/aiida-core/pull/6255) and moving towards replacing the pydantic models of the rest api. Jason * [Using pytest-xdist for running tests](https://github.com/aiidateam/aiida-core/pull/6620) in parallel which cut the time of running test a lot. With many cores some part of [tests failed randomly](https://github.com/aiidateam/aiida-core/pull/6627) because of test states sharing, I am keep on trying to fix up all. * Test with Timo and Natasha on the other possibility of Quantum Mobile that is spinned up using container and more lightweight and support both amd64 and arm64. * [Check global variables used in aiida-core](https://github.com/aiidateam/aiida-core/pull/6610), try to not use them if not needed and using singleton pattern in need. It can help to move forward to thread-safe and multiprocess-safe AiiDA. * Try refactoring the scheduler module to provide the clear interface to plugin developer. * Discuss and clear the path on how to improve the communicator part of engine (AEP#42) Xing: - WorkGraph - Discuss with the WorkGraph group and collect the [feedback](https://hackmd.io/Ojxram5qQdisG2bna6H6Bw). - Start to move PythonJob into a separate [repository](https://github.com/aiidateam/aiida-pythonjob), currently empty will push when the first draft ready. - Fixed bug reported by a Siesta user. [PR](https://github.com/aiidateam/aiida-workgraph/pull/350) - Fixed bug for the entry point, [PR](https://github.com/aiidateam/aiida-workgraph/pull/352) - AiiDAlab - [PR](https://github.com/aiidalab/aiidalab-qe/pull/939), Refactor the computational resource setp in QEApp, , this is the first large work after the MVC design, gain some experience and also try to give feedback. Natasha - Implemented MinimalWannier90WorkChain as a WorkGraph - Started working on a cli command to copy RemoteData files to a different computer. - Started working on a simple demo script wrapping a Julia code to run mid-throughput on a remote computer (using aiida_shell) MM: - Extended the `aiida-hyperqueue` plugin to support multi node job submission based on the work of Timo and Jason. - I worked on the PR #987 for improve the handling of SCF convergence issues. - Performed analysis of the effect of the magnetic moment initialisation. Sasha: * Finished [aiida-qe PR](https://github.com/aiidateam/aiida-quantumespresso/pull/1029) to add `parse_data_files` and `keep_data_files` options to `pp.x` calculation. * Implementation of [cubehandler](https://github.com/nanotech-empa/cubehandler) and corresponding plugins/aiidalab widgets * Migration of aiidalab-widgets-base to ipywidgets 8.x Adrian: * Added to [aiida-muon](https://github.com/positivemuon/aiida-muon) an new input for the DFT+U calculations, a dict with the U values for each atomic species. * Started working on maintaining and expand aiida-muon's testsuite. Elliott: * Ported high throughput geometry optimisation WorkChain to WorkGraph * Generalised WorkGraph builder to convert all aiida-mlip calculations (singlepoint, MACE descriptors, etc.) to high throughput calculation (in review: https://github.com/stfc/aiida-mlip/pull/169) * Started fine tuning WorkGraph (download data -> calculate descriptors -> filter data -> calculate DFT energies/forces/E0s -> preprocess -> fine tune) Ali: - New changes in Transport & Engine, and developed AsyncTransport plugin [PR 6626](https://github.com/aiidateam/aiida-core/pull/6626), After this PR, daemon won't get block anymore. - Reviewed Julian's PR - Many useful discussions with Jason about design of the new AsyncTransport plugin. - Planned: - Test the AsyncTransport PR, with some daemon blocking senarios - Discussed with Edan, about testing the pydantic PR. Ifeanyi: - Implemented in the QE-NEB (NebCalculation CalcJob) the possibility to provide and run calculations with the intermediate structures (in review: https://github.com/aiidateam/aiida-quantumespresso/pull/1049) - Improved the validation of the inputs. - Completed test cases on (materials of interest) with the CalcJob. - Initiated the NebWorkChain using the BaseRestartWorkChain. - Design of the NEB implementation in Aiida-muon. Daniel H:· #### Improving testing infrastructure - `aiida-testing`: (pytest fixtures for testing CalcJobs and WorkChain) - renamed to aiida-test-cache, preparing for publishing on PyPI, other chore tasks - still TODO: make it compatible with aiida-core >=2.3 - original (unfinished plan): https://github.com/aiidateam/aiida-test-cache/issues/74 - aiida-core: implemented new pytest option '--db-backend', switched default to sqlite (https://github.com/aiidateam/aiida-core/pull/6625) #### AiiDAlab - working on aiidalab-home (bugfix for 3.11 image)