# AiiDA-core days

**What**: (try and) block some time to work on AiiDA-core.

**When**: By default: Thu-Fri each second week of each month, check the calendar. Kickoff on Thu at 10am and wrap-up Fri at 16:00.

**Where**: In person at PSI if possible, otherwise on the AiiDA zoom (see calendar).

## Apr 2024

Participants:

* Marnik
* Miki
* Julian
* Ali
* Sebastiaan
* Kristjan
* Xing

### Topics

* [`verdi init`](https://github.com/aiidateam/aiida-core/pull/6315) -> Dogfooding and feedback, let's get this baby released.
* [Dumping `CalcJob`s/`WorkChain`s](https://github.com/aiidateam/aiida-core/pull/6276)
* General topic: Testing [`verdi storage backup`](https://github.com/aiidateam/aiida-core/pull/6069): changes have been made, good to do another round of testing.
* [Add the `SshFabricTransport` plugin](https://github.com/aiidateam/aiida-core/pull/6154): I'm still a huge fan of getting this merged, but needs more testing.

### What are you working on? (The progress is updated after `->` symbol for people can not join the wrap up meeting)

* Marnik: Four topics above, in that order.
* Julian:
* Resolve some minor issues in the dumping so that it's ready for dogfooding -> Ready for trying out: https://github.com/aiidateam/aiida-core/pull/6276
* Have a short look at the changes to StructureData and using AiiDA in `koopmans`, both with Miki -> Will focus next Friday on the StructureData, will work on implementing running the kcw workflow in `koopmans` with AiiDA; for pw.x it already works (thanks, Miki!)
* test verdi init -> See my comment, and long discussion this morning

* Miki
* test storage backup -> done
* test verdi init -> done
* work with Julian&Ali on StructureData & Koopmans@AiiDA ->
* useful discussion on StructureData, we will reserve the next friday (19th) as focus day (mainly me, Ali, Julian and Marnik) to work on it;
* Koopmans: scaffold of `from_ase_to_pwbaseworkchain`, working on the results part trying to combine `verdi dump` and ASE io modules.

* Sebastiaan
* Finalize the storage backup PR and get it merged
* Discuss the interface/behavior of `verdi init`
* Finalize the changes in caching behavior, see PRs
* https://github.com/aiidateam/aiida-core/pull/6215
* https://github.com/aiidateam/aiida-core/pull/6328

* Ali:
* Questionnaire
please comment and review my [PR](https://aiida-website--60.org.readthedocs.build/sections/science.html)
* `aiida-registry`
will do: Add scheduled CI workflow to detect and fix the plugin entry issue #310
* test `verdi init`

* Xing:
* Improve AiiDA-WorkTree
* [Improve decorator](https://github.com/superstar54/aiida-worktree/pull/36): Add built-in sockets
* [Simply the while syntax](https://github.com/superstar54/aiida-worktree/issues/34)
* [Support appending worktrees](https://github.com/superstar54/aiida-worktree/issues/31)
* **Wrap-up**
* [Built-in sockets: _outputs and _wait](https://github.com/superstar54/aiida-worktree/pull/36): Two new built-in sockets: 1)`_wait` input and output sockets. User will allow the user to see the dependency and running order of nodes in the GUI. 2) `_outputs` sockets, which allows the user to pass the entire outputs of a AiiDA node to the input of another node.
* [Support appending worktree](https://github.com/superstar54/aiida-worktree/pull/37): This PR allows user to append a worktree to another. Thus it is easy to combine worktree (group).

* Kristjan:
* [`verdi storage backup`](https://github.com/aiidateam/aiida-core/pull/6069) should be fully ready.
* Discussion on the [next-gen aiida explorer](https://github.com/materialscloud-org/issues/issues/33)?
* We'll probably have a student working on this from 27th of May.
* schedule for later
* **Wrap-up**
* Good feedback on the [Backup PR](https://github.com/aiidateam/aiida-core/pull/6069) (esp. from Marnik)
* Restore procedure was not easy with the previous state, so it was decided to keep re-add the config.json in the top level of the backup folder
* didn't have time to finish this today, will work on it next week.

* Jason:
* Test restore from backup (of Kristjan backup PR)
* Restore works fine but requires more instructions.
* Some tips added to documentation for conda postgresql (https://github.com/aiidateam/aiida-core/pull/6344)
* Minor improvements on `aiida-registry`.
* https://github.com/aiidateam/aiida-registry/pull/317 (PR open)
* Update qe-input-generator with new docker stack.

### Discussions

* Fri 9:30 - `verdi init`
* Fri 10:30 - new `StructureData`

### Notes

[Mbx] backup:

* I see the plan is to add support for the `core.sqlite_dos` backend in a later PR. Would this still be _before_ the release? It seems a shame that we provide new tools that use the SQLite version of the database but the backup isn't supported.
* Nitpick: default reporting seems excessive. Do we need to show all this information? Addition: My `aiida_loglevel` is set to `REPORT`, but I seem to be getting the `INFO ` level for the `storage backup` command?
* Do we think the `--keep` option is potentially dangerous? It would remove older backups that were _not_ made with this setting as well.
* Maybe we can also point users to how to restore a backup after performing one.

---

## Feb 2024

Participants:

* Marnik (running late; thanks SBB)
* Julian
* Ali
* Jason
* Miki
* Sebastiaan
* Kristjan
* Edan
* Xing

### Topics

* General topic: Testing [`verdi storage backup`](https://github.com/aiidateam/aiida-core/pull/6069). @Kristjan: what's the status here? Is it ready for more testing?

### What are you working on?

Julian:
* Finish WorkChain input/output dump [[#6276]](https://github.com/aiidateam/aiida-core/pull/6276)
* Bring up issue with jinja fields in [[#6294]](https://github.com/aiidateam/aiida-core/issues/6294)
* DOCS: Update `aiida-tutorials` to be compatible with AiiDA v2.5 [[#469]](https://github.com/aiidateam/aiida-tutorials/pull/469)
* DOCS: Better/quicker highlighting of useful resources [[#6290]](https://github.com/aiidateam/aiida-core/issues/6290)

Ali:
* `aiida-core` documentation: opened a PR with my suggestions on to linking `aiida-tutorials` and aiida cheat sheet [[#6288](https://github.com/aiidateam/aiida-core/pull/6288)]
* Reviewed PR: "Jusong's DOC: Add more instructions on how to use docker image"
* `aiida-tutorials`: opend an issue of ambiguities [[#470](https://github.com/aiidateam/aiida-tutorials/issues/470)]

Jason:
* ~~Test fabric PR~~
* Second round of testing backup PR
* Finalize [docker image Doc PR #6249](https://github.com/aiidateam/aiida-core/pull/6249)
* Loose the hashing computing without include core version
#6215
* Assistant Julian with computer/code template setup with verdi CLI.

Miki
* Complete the last two design principles of *atomistic* `StructureData` (see [this](https://github.com/aiidateam/aiida-atomistic/issues/3#issuecomment-1958988591) and [this](https://github.com/aiidateam/aiida-atomistic/issues/4#issuecomment-1957549568))
* Missing: aiida-quantuespresso version with the *atomistic* package + tests
* Missing: Plugin-developer guide for the migration

Sebastiaan
* Work with Kristian on finalizing the backup PR
* Available for discussion on various open PRs/issues
* Discussion on proposed reorganizing of tutorials on AiiDA docs

Kristjan:
* `verdi storage backup` is ready for testing. https://github.com/aiidateam/aiida-core/pull/6069
* Rsync progress PR: https://github.com/aiidateam/disk-objectstore/pull/166
* wrapup:
* How to make sure different profiles are not backed up to the same location
* root-level config.json or profile_uuid?
* Should we impose the same subfolder management for all storage backends?

Edan:
* Finalize QueryBuilder PR [#6245](https://github.com/aiidateam/aiida-core/pull/6245)
* Cover all operations
* Address "contains" concern
* Document
* Finalize `aiida-core-dev` docker image
* Concern regarding daemon in container
* Resolve issue with `verdi storage backup` against a network drive

Xing:
* Add more use cases/examples to aiida-worktree docs
* My own workflow for XPS calculation of adsorbed molecule.
* Maybe Marnik's?

Marnik:
* Reviewing: `CalcJob`: Allow to define order of copying of input files [[#6285](https://github.com/aiidateam/aiida-core/pull/6285)].
* `pydantic`-related AEP and PRs:
* Add AEP: Add a schema to ORM classes [[#40](https://github.com/aiidateam/AEP/pull/40)].
* ORM: Use pydantic to specify a schema for each ORM entity [[#6255](https://github.com/aiidateam/aiida-core/pull/6255)].
* ORM: Switch to pydantic for code schema definition [[#6190](https://github.com/aiidateam/aiida-core/pull/6190)].
* Migrate Transport to use pydantic for configuration [[#6198](https://github.com/aiidateam/aiida-core/pull/6198)]

## Jan 2024

Participants:

* Marnik
* Sebastiaan
* Kristjan
* Jusong
* Edan
* Miki

### Topics

Suggestions from Marnik:

* [Add the SshFabricTransport plugin](https://github.com/aiidateam/aiida-core/pull/6154). I've already done some testing here, but came up with an issue for Piz Daint, and it would be great if others could also give it a spin.
* [verdi storage backup](https://github.com/aiidateam/aiida-core/pull/6069): have been meaning to test this one for a while. Easy backups would be great to get in the next release.
* [Allow copying between different computers with same hostname](https://github.com/aiidateam/aiida-core/issues/6136): I've been running into this issue quite often on LUMI. Already have a rudimentary solution that just checks the full hostname, but as already discussed during a previous meeting it would be better if we can somehow indicate that different computers share the same file system.

Suggestions from Seb:

* [ORM: Switch to pydantic for code schema definition](https://github.com/aiidateam/aiida-core/pull/6190): This PR replaces the custom model spec for AbstractCode and its subclasses to use pydantic just as the StorageBackend does since v2.5.0. I have also [started a PR](http://migrate%20transport%20to%20use%20pydantic%20for%20configuration/) to do the same for Transport but that is a bit more involved and needs a bit more work. Here it won't be possible to be completely backward compatible I fear.
* [Add orm.Entity.fields interface for QueryBuilder](https://github.com/aiidateam/aiida-core/pull/6245): This is an old PR from Chris that is taken over by @Edan Bainglass. It adds a more programmatic interface to constructing querybuilder filters. I think this is great functionality, I just wonder if the definition of the model attributes could be joined with the concept of pydantic as applicable in the PR in the item before this. Would be great to discuss and experiment
* [Scheduler: Refactor interface to make it more generic](https://github.com/aiidateam/aiida-core/pull/6043): This is the PR to provide support for what is necessary for FirecREST. What is the status on this work?

### What are you working on?

Marnik:

* SSH-`fabric`: testing and figuring out failure for Piz Daint.
* Testing `verdi storage backup`
* If I have time: [Allow copying between different computers with same hostname](https://github.com/aiidateam/aiida-core/issues/6136).

Sebastiaan:

I would very much like to discuss the following:
* [ORM: Switch to pydantic for code schema definition](https://github.com/aiidateam/aiida-core/pull/6190/files): This PR replaces the custom model spec for AbstractCode and its subclasses to use pydantic just as the StorageBackend does since v2.5.0. I have also started a PR to do the same for Transport but that is a bit more involved and needs a bit more work. Here it won't be possible to be completely backward compatible I fear.
* [Add orm.Entity.fields interface for QueryBuilder](https://github.com/aiidateam/aiida-core/pull/6245): This is an old PR from Chris that is taken over by @Edan Bainglass. It adds a more programmatic interface to * constructing querybuilder filters. I think this is great functionality, I just wonder if the definition of the model attributes could be joined with the concept of pydantic as applicable in the PR in the item before this. Would be great to discuss and experiment
* [Scheduler: Refactor interface to make it more generic](https://github.com/aiidateam/aiida-core/pull/6043): This is the PR to provide support for what is necessary for FirecREST. What is the status on this work?

Kristjan:
* migrating [aiida-optimade](https://github.com/aiidateam/aiida-optimade) to pydantic 2, aiida-core 2.5, optimade-python-tools 1.0
* finalizing [disk-objectstore backup](https://github.com/aiidateam/disk-objectstore/pull/161)
* finalizing [aiida-core backup](https://github.com/aiidateam/aiida-core/pull/6069)
* writing also documentation on this

Jason:
* Resume the [aiida-core-i18n](https://github.com/unkcpz/aiida-core-i18n) and there is now full translated zh_CN documentation.
* Finalize the remove version in hashing computing:
https://github.com/aiidateam/aiida-core/pull/6215
* Docker image: improve documentation [PR open](https://github.com/aiidateam/aiida-core/pull/6249)

Edan:
* Deal with [`QueryBuilder` PR](https://github.com/aiidateam/aiida-core/pull/6245)
* Consider switch to `pydantic`
* Implemented by Seb :pray:
* Tested - all good - Chris's original features still operational
* Minor concern regarding inheritance synchronization (resolved?)
* Needs testing
* Consider restructuring to support `or` operations
* Done - works as expected. Can present at some point.
* Test `verdi storage backup`
* Please review [`qe-tools` PR](https://github.com/aiidateam/qe-tools/pull/73) 🙏
* Thanks Seb! Updated MaRDA PR
* When do we plan to release qe-tools next?
* Create and document new aiida-core dev image
* Plan discussed with Jason
* Will get to it in coming days

Miki:
* Implementation of the StructureData in `aiida-atomistic` package:
* pushed on github the new implementation, contained in a [jupyter notebook](https://github.com/aiidateam/aiida-atomistic/blob/new_Data/aiida_atomistic/data/structure/new_implementation.ipynb).
* `properties` attribute
* immutability
* backward-compatibility (structure.pbc and structure.kinds)
* pbc property implemented (structure.properties.pbc) as most simple example
* missing: custom properties support
* missing: decision on how to store properties (db or repo)
* missing: adapting `aiida-core`, `aiida-quantumespresso`, `aiida-pseudo` to support the new StructureData. (however, I already have something implemented which should work fine)

## Dec 2023

### Topics

#### Message from Sebastiaan

Hi everyone, unfortunately something has come up and I won't be available too much tomorrow. I may be able to respond to urgent questions on my phone though, and I will be back on Friday to look at PRs, feedback, questions and any other comments. From my side, I think it would be good to work on finalizing the upcoming release. I have updated the branch to the latest state of main today. Here are some points of action that need to happen before the release:
* [This PR is blocking](https://github.com/aiidateam/aiida-core/pull/6202). A feature was added to the release branch that can break certain processes. Please have a look and give feedback on its proposal if you have any (if you agree with the suggestion, that would also be useful to know). For background on why this is necessary, see [the discussion on Discourse](https://aiida.discourse.group/t/pause-a-process-when-launching-it/184/4?u=sphuber).
* Please have a look at [the changelog](https://github.com/aiidateam/aiida-core/blob/release/2.5.0/CHANGELOG.md) that I prepared. Feedback on things that are unclear/wrong are very welcome. Or if you have other features that would merit a more detailed expose, please suggest them
* Give the new verdi profile setup command a go. Specifically, try the new core.sqlite_dos storage backend with verdi profile setup `core.sqlite_dos`. This will setup a new profile with a storage backend using SQLite database and disk-object container, so all it needs is a single folder on disk. This makes it easy to setup. Please try creating it and try using it a bit with for some actual calculations/queries.
* In the light of the new core.sqlite_dos backend [I made a PR](https://github.com/aiidateam/aiida-core/pull/6207) to add a section to the documentation on the various storage backends. It gives [a succint overview](https://aiida--6207.org.readthedocs.build/projects/aiida-core/en/6207/topics/storage.html) of their strengths/weaknesses and suggestion when to use them. This is just an initial mock-up and it would be great if we can further improve it. We should then probably also update the Get Started page to suggest the core.sqlite_dos backend for some cases, or at least link to this new topic, with a quick tutorial on how to create a profile (which should be 1 command now).

Thanks a lot in advance for your help in testing and finalizing the release! If all goes well, we should be able to release it early next week. Please ping me on Slack or Github if there are any questions. I will see you all on Friday to catch up on progress.

#### Some additionals topics to work on:

Testing new features/APIs:

* Add the `SqliteDosStorage` storage backend [[#6148](https://github.com/aiidateam/aiida-core/pull/6148)]
* Add the `SshFabricTransport` plugin [[#6154](https://github.com/aiidateam/aiida-core/pull/6154)]
* Add verdi profile setup [[#6023](https://github.com/aiidateam/aiida-core/pull/6023)]
* CLI: Various improvements concerning user details [[#6173](https://github.com/aiidateam/aiida-core/pull/6173)]
* Xing's `WorkTree`

Potential coding:

* AiiDA temp profile in shell and Jupyter-notebook [[#6070](https://github.com/aiidateam/aiida-core/pull/6070)]
* SqliteZipBackend.create_profile should automatically define the default_user_profile [[#6165](https://github.com/aiidateam/aiida-core/issues/6165)]

### What are you working on?

* Marnik:
* Testing the [new `engine.submit` signature](https://github.com/aiidateam/aiida-core/pull/6202) ✅
* Reviewed https://github.com/aiidateam/aiida-core/pull/6207 ✅
* Test the `SqliteDosStorage` feature 🔴
* Test `verdi profile setup` 🔴
* (looked into) New `StructureData`
* (looked into) Xing's AiiDA-light: https://github.com/aiidateam/aiida-core/pull/6070
* (did not manage) Test the `SshFabricTransport` transport

* Miki:
* aiida-atomistic. I will produce a draft for the documentation of the StructureData API, based on the [user stories](https://hackmd.io/QncWOzUcTni6Mlg0VgJknA). Results: [documentation of aiida-atomistic package about StructureData](https://github.com/aiidateam/aiida-atomistic/blob/new_Data/docs/source/user_guide/structuredata/structuredata.md). The correct branch to compile the docs is [new_Data](https://github.com/aiidateam/aiida-atomistic/tree/new_Data). More discussions and details are provided [here](https://hackmd.io/IN19UdRDRICDhyDGMdsUUQ). In particular user stories, that I am using to write the documentation.

* Kristjan:
* https://github.com/aiidateam/disk-objectstore/pull/161
* waiting for review
* https://github.com/aiidateam/aiida-core/pull/6069
* adapted accordingly
* could this even go to 2.5.0?

* Jusong
* SSH with `fabric` in AiiDAlab
* Docker image: Check documentation; improve UX 🔴
* Caching: Remove version.
* https://github.com/aiidateam/aiida-pseudo/pull/168
* https://github.com/aiidateam/aiida-core/pull/6215 discussion require

* Xing:
* Update the [WorkTree engine](https://github.com/superstar54/aiida-worktree/wiki/WorkTree-Engine) to support real `Dependency-based Execution`. Thus imporve the performance compared to the WorkChain (Hybrid Sequential-Parallel Execution)
* Support modifing WorkTree
* [x] Restart: load a finished worktree and modify the inputs, then submit the worktree
* [x] Continue a finished worktree by adding new nodes.
* [ ] Pause and play the worktree
* [x] Modify a running worktree on the fly.
* Add `worktree` cli to inspect the worktree

### Discussion

* MBx: Enforce keyword usage for `wait` and `wait_interval`, see [this comment](https://github.com/aiidateam/aiida-core/pull/6202#discussion_r1420194854).
* MBx: Usage of "temp" and "sqlite" storages.
* MBx: SSH-fabric should become the default over time? Do we need the legacy?
* JY: Discussion on caching -> remove versions.
* SPH: Enabling deprecation warnings by default again