# AiiDA coding week 01-05 Dec 2025
###### tags: `event`
[toc]
## Daily kickoff
### Monday
__JG__
- during pr review, found two "issues" in disk-objectstore:
- method to get size of `packed` during creation uses `stat` -> Before `flush`, this gives wrong size, use instead `tell()` to get number of bytes in `packed`
- `update` callback called on every loose deletion, rather than every pack creation
__MBo__
- feedback for SData, working on performances
- discussion on blog post for 3 usage applications (relax, phonons, Wannier, ...)
__CP__
- Discussion with MB GP for PR of pausing BaseRestartWorkchain, MB updated MBx who will be working on it on wednesday
- Prepared PR of fixed atoms in QE APP to be revised by XW https://github.com/aiidalab/aiidalab-widgets-base/pull/716 and https://github.com/aiidalab/aiidalab-qe/pull/1440
- Trying to understand if we can use MFA CSCS app also for CINECA
__MBx__
- Got review from DH on: 🐛 TrajectoryData: Add `pbc` [[#7079](https://github.com/aiidateam/aiida-core/pull/7079)]. (should be close 🤞)
- Opened draft PR on: 💥 `BaseRestartWorkChain`: add `on_unhandled_failure` input [[#7116](https://github.com/aiidateam/aiida-core/pull/7116)]. Still draft because I need to adapt/add documentation.
- Discussion on `BaseRestartWorkChain`: pause at `max_iterations` (issue [#7095](https://github.com/aiidateam/aiida-core/issues/7095)): I think we reached agreement on proposed API, plan to provide implementation tomorrow. Might make sense to add it to [#7116](https://github.com/aiidateam/aiida-core/pull/7116).
### Tuesday
__JG__
* [[DO #200] Fix "wrong" pack sizes due to missing flush by passing known size via `tell`](https://github.com/aiidateam/disk-objectstore/pull/200) -> merged
* Opened [[aiida-core #7118] Fix progress bar stuck during archive import](https://github.com/aiidateam/aiida-core/pull/7118)
* Field-tested [[aiida-core #7061] Fix race condition in `JobsList`](https://github.com/aiidateam/aiida-core/pull/7061) by Ali again, and seems to solve the issue. Merged :rocket:
__MBx__
* Nothing much to report today! Had to work on the 🇦🇺 project, still working on `BaseRestartWorkChain`: pause at `max_iterations` (issue [#7095](https://github.com/aiidateam/aiida-core/issues/7095)).
__CP__
* Thanks to XW ready to plug into siesta workchain results the Bands and PDOS viewers we use in the QE app
* Checking with AY the PR of pywidgets 8 https://github.com/aiidalab/aiidalab-widgets-base/pull/644#issuecomment-3602581828 It is not easy to keep compatibility with 7. we propose to migrate apps and abandon compativbility to 7.
* Trying to improve widget to slice cube files https://github.com/nanotech-empa/aiidalab-empa-surfaces/pull/263
__XW__
* WorkGraph and Airflow integration
* The essential features are all supported, see [docs](https://node-graph-engine.readthedocs.io/en/latest/engines/airflow.html#engines-airflow)
* data provenacne
* async function task (monitor)
* `run` and `submit`
* nested workflow (thus support if, for, while)
* features nice to have in the future
* Load workgraph data from Process checkpoint (which will be cleaned when process finish), this can reduce the size of the DAG files.
* Expose graph inputs of WorkGraph as the input parameters of the DAG, so that user can confige the inputs in the GUI.
__AK__
- Racing conditions on StashCalculation, TransferQueue, and (merged) JobsList
__KE__
- Looking into aiida-wannier90-workflows
- REST API
__EK__
- Additional aiida-workgraph examples
__EB__
- Working on AiiDAlab peripheral bugs/issues
- Cleaning up old aiida-restapi PRs and migrating
### Wednesday
* Meeting on AiiDA kubernetes deployment on CSCS
*
__JG__
* Planning to merge [[DO #193] Delete loose files per pack creation incrementally](https://github.com/aiidateam/disk-objectstore/pull/193)
__EB__
- Working on REST API endpoints
- Meeting with @eimrek @mikibonacci @superstar54 to walk through pydantic/RESTAPI PRs
- Great meeting! See [PR review](https://github.com/aiidateam/aiida-restapi/pull/93) for more details
__MBo__
- StructurueData, meeting with GP
### Thursday
__JG__
* `aiida-core` 2.7.2 release
* review [[# 197] Add dev dependency group for uv, update CI workflow](https://github.com/aiidateam/disk-objectstore/pull/197) by DH
*
__EB__
- Following up on reviews
- Adding remaining endpoints to REST API
__GP__
- `verdi process list` show also computer
- Review DO PR
-
__MBx__
- As discussed, still trucking away at the pausing the `BaseRestartWorkChain` upon unhandled failures.
__AY__
- Upgrade to ipywidgets 8 -> aiidalab widgets base major release
__ZL__
- Stream directly to packed PR
__CP__
- Progress in the connection to CINECA
- Discussion with [GP, AY] for ploicies in teh demoserver
- Discusison with [XW,GP] on how to implement in QE App features related to pausing BaseRestartWorkChain
***
### Friday
__MBx__:
- 💥 `BaseRestartWorkChain`: add `on_unhandled_failure` input [[#7116](https://github.com/aiidateam/aiida-core/pull/7116)]: ready for review! Mainly updated/added documentation, I still looked into changing the process state to "Waiting", but that didn't seem too trivial (see [this comment](https://github.com/aiidateam/aiida-core/pull/7116#issuecomment-3614978716)).
- 👌 `SshTransport`: avoid use of deprecated `getcwd` method [[#7128](https://github.com/aiidateam/aiida-core/pull/7128)]. I've added this to the v2.7.2 project, because I think these deprecation warnings make `core.ssh` rather user-unfriendly.
__EB__
- Decoupling REST API node responses from attributes/extras/etc. is ongoing
- This raised some issues with the pydantic PR, which has been fixed
- [KE] has raised an intersesting design idea for the pydantic model system for clarity and GET/POST model symmetry. Will open issue to investigate down the road
- [KE] [#7131](https://github.com/aiidateam/aiida-core/issues/7131)
- [EB] May merge [#7131](https://github.com/aiidateam/aiida-core/issues/7131) with [#7093](https://github.com/aiidateam/aiida-core/pull/7093), the latter aiming to refactor the model system out in a non-nested way. See simliar work for Entity collections in [#5183](https://github.com/aiidateam/aiida-core/pull/5183)
- Will review [AY]'s PRs w.r.t AiiDAlab upgrades, AWB upgrades, etc.
__Kristjan__
- Looked into pydantic & restapi PRs
- Wannier workflows - docs should be improved
- An important issue: maybe the API doc should be restored, I'm occasionally using it. ([e.g.](https://aiida.readthedocs.io/projects/aiida-core/en/stable/reference/apidoc/aiida.orm.nodes.data.array.html#aiida.orm.nodes.data.array.trajectory.TrajectoryData))
__CP__
- After discussions with [GP] yesterday, we set "before Christmas" as a temptative deadline to release aiida-core 2.8, [AY] pywidgets 8? python 3.11? I will stay in touch with [MBx] for tetsing the BaseRestart after max iteratuons and with [XW] for the QE App adaptation
- Discussed with [AY], let's try to have [AY,EB]Finalize images and submission letter for AiiDAlab2 manuscript and try to submit before Christmas [AY] update of the journal template
__Ali__
- [Fixed a deprication warning in core.ssh](https://github.com/aiidateam/aiida-core/pull/7130), thanks to Marnik and Julian
- Stashing
- [Discussed a race condition with Michail, Alex, Edan](https://github.com/aiidateam/aiida-core/pull/7115)
- [A backward incompatible behaviour requested by Michail](https://github.com/aiidateam/aiida-core/issues/7120)
- Engine
- [The race condition in JobList merged](https://github.com/aiidateam/aiida-core/pull/7061)
- TransportQueue
- [Unexpectedly closes connection when using ssh_async](https://github.com/aiidateam/aiida-core/issues/7119)
- Review ed/ing
- [Run type checking on engine.processes module](https://github.com/aiidateam/aiida-core/pull/7114)
- [Adding the verdi computer goto command](https://github.com/aiidateam/aiida-core/pull/7124)
__JG__
_core_
* [[# 7132] Release v2.7.2](https://github.com/aiidateam/aiida-core/pull/7132)
* should be (almost) ready: test release, changelog, ...?
* what about conflicts during cherry-picking?
* [[# 7118] Fix progress bar stuck during archive import](https://github.com/aiidateam/aiida-core/pull/7118)
* fixed!
* [[# 6417] Stream directly to `packed` during `verdi archive import`](https://github.com/aiidateam/aiida-core/pull/6417)
* Discussion with ZL and GP: know a way how to get this in -> _unsafe_ method in disk-objectstore that doesn't return file handles individually through context manager
* [[# 6659] Storage: Add `extend_json` Tag for Enhanced JSON Field Handling in `bulk_update` Operations](https://github.com/aiidateam/aiida-core/pull/6659)
* with ZL, fixed issue of failing test due to db hanging, injection place into db, and load time
_disk-objectstore_
* [[# 200] Fix "wrong" pack sizes due to missing flush by passing known size via `tell`](https://github.com/aiidateam/disk-objectstore/pull/200) -> merged
* [[# 193] Delete loose files per pack creation incrementally](https://github.com/aiidateam/disk-objectstore/pull/193) -> merged
__MBo__
* no time for blogpost after the discussion
* reviewed a couple of small sub-PRs of the pydantic one by EB
* working on the comments by GP about StructureData (performance, database vs repository, plugins integration and other stuff), should finish by monday
__TR__
Weekly/daily wrap up
* `aiida-core`: https://github.com/aiidateam/aiida-core/pull/7129
* `aiida-pseudo`: https://github.com/aiidateam/aiida-pseudo/pull/201
* Install all functionals and protocols at once, i.e., don't need to specify PBE+efficiency, PBE+precision, PBEsol+efficiency and PBEsol+precision
* `aiida-pseudo install sssp --all`
* `aiida-pseudo install pseudo-dojo --all`
* `aiida-qe`:
* Improving automatic code setup:
* new `--all` https://github.com/aiidateam/aiida-quantumespresso/pull/1222
* `aiida-quantumespresso setup codes localhost -a` instead of `aiida-quantumespresso setup codes localhost pw.x projwfc.x dos.x ...`
* Further small improvements https://github.com/aiidateam/aiida-quantumespresso/pull/1223
* Fix related to `HubbardStructure`: https://github.com/aiidateam/aiida-quantumespresso/pull/1224
__XW__
* WorkGraph and Airflow integration, [docs](https://node-graph-engine.readthedocs.io/en/latest/engines/airflow.html#engines-airflow)
* Together with Elliott
* check the the tutorial workflow for aiida-mlip: trainning the MACE model.
__EK__
* `aiida-workgraph`/`node-graph` task inputs
* https://github.com/scinode/node-graph/issues/131, resolved by https://github.com/scinode/node-graph/pull/132 (merged)
* https://github.com/aiidateam/aiida-workgraph/pull/732 (merged)
* `aiida-pythonjob` trajectory input/outputs
* https://github.com/aiidateam/aiida-pythonjob/pull/63
* `aiida-mlip`
* Redeveloped fine-tuning workgraphs e.g. https://github.com/stfc/aiida-mlip/pull/260 with XW
__AY__
* Migrated aiidalab widgets base (AWB) to ipywidgets 8.x ([PR](https://github.com/aiidalab/aiidalab-widgets-base/pull/644)).
* Home app migration [PR](https://github.com/aiidalab/aiidalab-home/pull/207).
* Moving some of the notebooks [from awb](https://github.com/aiidalab/aiidalab-widgets-base/pull/720) to [home app](https://github.com/aiidalab/aiidalab-home/pull/211).
* Had some fun on Thu afternoon trying to catch bitcoin miners.
__AG__
[aiidateam/aiida-core/pull/7133](https://github.com/aiidateam/aiida-core/pull/7133): Implementation of custom daemon replacing [circus](https://github.com/circus-tent/circus) dependency and offering support for single process services (like api servers) and worker services (like aiida workers). My idea is now to use it in the [airflow-provider-aiida](https://github.com/aiidateam/airflow-provider-aiida) package to stress test it, because creating edge case tests for IPC applications is quite hard.
## First kickoff
- Working towards new features for 2.8 release: https://github.com/orgs/aiidateam/projects/18/views/1 [all]
- work on 3 usage applications (relax, phonons, Wannier, ...) -> implement needed features [all]
__JG__
- v2.7.2 release: https://github.com/orgs/aiidateam/projects/24/views/1 [JG, Ali]
- Finalize open disk-objectstore PRs [GP, JG]
- wrap up Zisen's PR [GP, JG]
- Improve Renku integration
__GP__
- Finalize work on pausing BaseRestartWorkchains on double unhandled error, extend to pausing on 5 handled errors [GP, CAP, MBx; MBo?]
- Add new user features (computer goto, improve calcjob process status message) [GP]
- Quantum Mobile for Arm processors [GP, NM]
__CP__
- As above with [GP]
- Discussed with [GP] hwo we could mofify the QE app to inform efficietly teh user of the paused workchain and facilitate actions, we should invole [XW] and or [EB]?
- With help of [GP] step forward in connection to CINECA, will contact CINECA directly to invetsigate further possibilities
- Completed cube slicer
- Plugged into siesta calculations viewer the solution of [XW] to visualize bands. TBD PDOS
__EB__
- Finalize pydantic work [EB]
- Finalize/test REST API [EB]
- Refactor storage backend implementation? [JG, EB]
- Brainstorm on keeping Data plugins inheritance vs. encapsulation [GP; EB?]
__Ali__
- Solve a [bug in stashing](https://github.com/aiidateam/aiida-core/issues/6915) [Ali]
- [Fix race condition in `JobsList`](https://github.com/aiidateam/aiida-core/pull/7061)
- PR opened: https://github.com/aiidateam/aiida-core/pull/7115
__DH__
- Review TrajectoryData: Add `pbc` [[#7079](https://github.com/aiidateam/aiida-core/pull/7079)].
- Fix script that autogenerates `__all__` attributes in `__init__.py` files. [#7091](https://github.com/aiidateam/aiida-core/pull/7091)
- Test ipywidgets 8.x upgrade (AiiDAlab), needed for upgrade from 3.9 to 3.11
- (stretch goal) Support Python 3.14 (and drop 3.9?) [DH]?
__MBo__
- Finalize aiida-atomistic [MBo, ... GP?]
- Review pydantic work by Edan [MBo,Associated educational platforms XW]
- Pausing of BaseRestartWorkchains (if I have time)
__Nataliia__
- Quantum Mobile [NM, support by GP and MBx]
__XW__
- WorkGraph and Airflow integration.[XW, AG]
- improve the performance: currently, create a dag file in the DAG folder, and wait for the scheudler to pick up the DAG. This leads to delay.
- How to integrate ontology-awared semantics with AiiDA provenance [XW, EB]
- While AiiDA structural provenance captures how data was created, AI
agents require semantic clarity on what the data represents. AI-ready provenance
- handle realistic workflow (dynamic)
__MBx__
- 🐛 TrajectoryData: Add `pbc` [[#7079](https://github.com/aiidateam/aiida-core/pull/7079)]. -> Is Daniel in the house by any chance? :sweat_smile:
- Pausing [`BaseRestartWorkChain`] upon unhandled error. [[#7069](https://github.com/aiidateam/aiida-core/pull/7069)] -> my branch [here](https://github.com/mbercx/aiida-core/tree/new/pause-unhandled).
- Transports: Testing `core.ssh_async` some more, help Ali with [[#7061](https://github.com/aiidateam/aiida-core/pull/7061)] if still needed. Also fix deprecation warnings for `core.ssh` [[#7113](https://github.com/aiidateam/aiida-core/issues/7113)].
- [Optional] Re GPs comment: `BaseRestartWorkChain`: pause at `max_iterations` [[#7095](https://github.com/aiidateam/aiida-core/issues/7095)]. My suggestions are [in this comment](https://github.com/aiidateam/aiida-core/issues/7095#issuecomment-3519246167).
- [Optional] 🐛 `FileNotFoundError` related to monitors [[#7086](https://github.com/aiidateam/aiida-core/issues/7086)]. -> Very tedious if you are running monitors. Encountered with `aiida-vasp`, so first investigate if the issue lies with their monitors somehow. Even in that case, maybe we can make it easier for users/developer to identify the problem.
__Empa__
- AiiDAlab: migration to Python 3.11 (3.12) [AY, EB, DH] above they mention 3.14 let's list what would be teh problem to go beyond 3.12
- Fixed atoms in QE App [CAP, AOG]
- Minimal AiiDAlab siesta app [CAP]
- Finalize CDXML importer [CAP, AY]
- Check bug importing AiiDAlab QE app claculations from aiida-core 2.5 to more recent e.g. 2.7.1
- Investigate and solve issue with ssh key field permisisons reset in CSCS Kubernetes [AY]
- Cubehandler tool [AY, CAP]:
- finish rendering functionality
- The problem is [here](https://github.com/oschuett/appmode/issues/80).
__KE__
- Reviewing PRs
- https://github.com/aiidateam/aiida-core/pull/6884
- https://github.com/aiidateam/aiida-restapi/pull/88
- https://github.com/aiidateam/aiida-restapi/pull/87
- https://github.com/aiidateam/aiida-restapi/pull/93
- Hiding the pydantic docs bloat. https://github.com/aiidateam/aiida-core/issues/6997
- sqlite_zip SQLite decompress caching issue: https://github.com/aiidateam/aiida-core/issues/7100 [KE, AK]
- test/run real-world workflows (wannier)
__BM__
- Improvements to AiiDA explorer (mutlistep view? + performance improvements)
- Test new Edans rest api to see if integrating into Explorer is ready.
- pagination / convenience endpoints etc.
__EK__
- Updating aiida-mlip workgraph examples/entry points
- Improvments to aiida-workgraph docs
- https://github.com/aiidateam/aiida-workgraph/issues/723
- https://github.com/aiidateam/aiida-workgraph/issues/486
## Discussions
- Replacing circus with our own process supervisor [AG]
- Airflow WG integration [AG, XW]
- How to clean up the CLI towards v3? (along with airflow integration) [AG]
- AiiDA kubernetes deployment on CSCS [RH, JG, AG, MBo, AY]
- aiida-tui [RH]
- How to integrate ontology-awared semantics with AiiDA workflow [XW, EB]
- Dropping Immutability with versioned nodes [AG]
- `aiida` meta package [AG, all]
- `aiida-explorer` demo/feedback [BM, KE, ...]
### aiida-tui
- EB will continue working on this, as well, MM for feedback, also BM to coordinate together with aiida-explorer
- Refreshing?
- Re-rendering the full table seems to be snappy enough
- Option to manually refresh, but also auto-refresh frequency
- trogon vs. textual (former is a dependency of the latter)
- GP: browse groups via ls interface
- agree on list of features between aiida-explorer and aiida-tui
- More things to show: retrieved files, input files, call root, ideally, full tree, group organization/navigation, use aiida to fetch file contents done or running (similar to outputcat), or jump to file
- action to go to web ui for a node?
- github.com/aiidateam/aiida-tui
### Real-world workflows
- Finalize/review [[#1207] Add a documentation page on inputs customisation](https://github.com/aiidateam/aiida-quantumespresso/pull/1207)
- First draft of blog post(s?): setup the same for all three workflows
- Python snippets for each workflow (make 3 packages consistent, functionality in aiida-wannier90-workflows only should be in aiida-qe)
- Include also some result of the analysis in the blog post, as that is what's interesting
### aiida metapackage
We decided to keep the meta package in sync with the aiida-core version but to not pin any dependencies except aiida-core. This way people can install the metapackage like aiida-core `pip install aiida==<AIIDA_CORE_VERSION>`. The other packages we leave unpinned, allowing flexibility in the the versions of the other packages, and more importantly we do not need to release a new metapackage for each new version of a dependency. With each new release we also want to provide a pip constraint file (basically a lock file) with hard pinned dependencies to offer an official stable production environment that can be installed with `pip install aiida --constraint https://raw.githubusercontent.com/aiidateam/aiida/constraints-<AIIDA_CORE_VERSION>/constraints-<PYTHON_VERSION>.txt`
. It is stable in a sense that we recommend it and hopefully most people use it so we can make it stable.
By default we only include packages the extend capabilities how to write workflow. In the extras we add tools, UI, and other things in the future.
```
dependencies = [
'aiida-core==2.7.2',
'aiida-workgraph',
'aiida-shell',
'aiida-pythonjob',
]
[project.optional-dependencies]
tools = [
'aiida-submission-controller',
'aiida-project',
]
ui = [
'aiida-gui'
]
full = [
'aiida[tools,ui]',
]
```
### kubernetes setup
EMPA CSCS AiiDA setup:
- one service (psql, rabbitmq) per user
- User authentication to aiidalab works with jupyterhub
- Job submission is done over MFA (for SSH job submission) https://github.com/aiidalab/aiidalab-mfa-cscs
- https://github.com/aiidalab/aiidalab-docker-stack
- for EMPA CSCSC: kubernetes extremely slow on (maybe due to HD, no SSD)
W&C CSCS AiiDA setup:
- shared service (psql, rabbitmq)
- rabbitmq service can be easily shared because profile uuid is put into communcation
- psql service for multiple users requires the postgres admin to create an aiida database for each new user once (need to check if multiple profile can lie in one psql database but should work)
- TODO ricoh, write about the kuberenetes infrastructure
--> Can Rico share a Helm chart for this setup
## Rooms
| Date | Morning | Afternoon |
|--------|-----------|-------------|
| 01/12 | OHSA/B17 | OHSA/B17 |
| 02/12 | OVGA/200 (OTLA/120) | OVGA/200 (OTLA/120) |
| 03/12 | OVGA/200 (OFLG/401) | OHSA/E13 |
| 04/12 | OHSA/E13 | OVGA/200 (OTLA/120) |
| 05/12 | OHSA/B17 | OHSA/B17 |
https://pocket.psi.ch/psimap
---
OHSA/B17
OTLA/120
OFLG/401
## Planning of participation
| Name | Participation | How | Days |
|------|---------------|------|----|
| Giovanni | ✅ | |
| Alex | ✅ | |
| Ali | ✅ | in-person except Thursday which I'll be online |
| Edan | ✅ | in-person | arriving late M/Th (~11am); working from home Tu
| Xing | ✅ | |
| Miki | ✅ | |
| Timo | ✅ | |
| Michael | ✅ | |
| Natasha | ✅ | |
| Julian | ✅ | |
| Kristjan | ✅ | discussions, but can work in room
| Bud | ✅ | discussions, but can work in room |
| Marnik | ✅ | remote (daily kickoff) | |
| Jason | ❌ (new job) | |
| Austin | ❌ | |
||||
| Elliott Kasoar (STFC UK) | ✅ | in-person | all |
| Sasha (Empa) | ✅ | in-person | all, except We |
| Carlo (Empa) | ✅ | in-person | Mo, Tu, We |
| Rico Haeuselmann | ✅ | in-person | Mo & We |
| Zisen | ✅ | in-person | one day |
| Daniel Hollas | ✅ | remote | |
| Ifeanyi John Onourah (Parma) | ✅ | remote | |
| Nataliia Manko (Antimo Marrazzo group) | ✅ | remote |
| Davide Bidoggia (Materys) | ✅ | remote |
| Fabio (Empa) | ❌ | |
| Riccardo Bertossa (Materys) | ❌ | |
| Sebastiaan | ❌ | |
| Adrian Gomez (BIFI) | ❌ | |
| Pietro Bonfa' (Parma) | ❌ | |
| Junfeng | ❌ | |
| Andres (Empa) | ❌ | |
## Todos JG
-
Sign in with Wallet
Connect another wallet