# AiiDA coding days July 2024

###### tags: `event`
###### time: 15 Jul 2024 - 19 Jul 2024

[TOC]

**Participants**:

* Marnik [MBx]
* Julian [JG]
* Alex [AG]
* Ali [AK]
* Miki [MBi]
* Edan [EB]
* Jusong [JY]
* Xing [XW]
* Sebastiaan [SPH]
* Kristjan [KE]
* Giovanni [GP]

## Topics

### The `WorkGraph`

**Note**: The `aiida-workgraph` repository has been transferred to the aiidateam organization. This change may affect your forked repositories. Please check, @Marnik and @Julian.

Docs: https://aiida-workgraph.readthedocs.io/en/latest/

The AiiDA `WorkGraph` has already seen quite a bit of changes since we first started testing it, and its API is getting closer to converging. However, not everyone has had a chance to test it thoroughly ([MBx] e.g. me), and the API has changed over time, so a solid round of dogfooding is in order. :dog2:

People who should write an example workflow (if not done already) using the `WorkGraph`:

- [ ] Marnik
- [ ] Julian
- [ ] Alex
- [ ] Ali
- [ ] Edan
- [ ] Jusong
- [ ] Giovanni

[XW] suggest reading this example [Transfer WorkChain to WorkGraph](https://aiida-workgraph.readthedocs.io/en/latest/howto/transfer_workchain.html).

Once everyone has had the chance to work with the `WorkGraph`, we can have an extensive discussion on which (if any ^^) changes to the API we'd like to see.

#### Main Goal: Enhance and Stabilize the Current API
All participants are encouraged to select one or more topics to focus on. Below are key areas identified for improvement:

- [ ] [PythonJob](https://aiida-workgraph.readthedocs.io/en/latest/built-in/pythonjob.html): An important feature requiring attention. Consider spending a day to enhance aspects such as:
- [ ] Try and give feedback on the current implementation.
- [ ] [Data serializers](https://aiida-workgraph.readthedocs.io/en/latest/built-in/pythonjob.html#Define-your-data-serializer): Discuss the feasibility of a common data serializer repository.
- [ ] [Namespace output improvements](https://aiida-workgraph.readthedocs.io/en/latest/built-in/pythonjob.html#Namespace-Output)
- [ ] Improve for virtual environments on remote computers.
- [ ] Simplify the usage of `PythonJob` outside WorkGraph, e.g., inside a WorkChain.
- [ ] Compare with [aiida-dynamic-workflows](https://github.com/microsoft/aiida-dynamic-workflows/tree/main). If possible, evaluate both aiida-dynamic-workflows and PythonJob to understand their strengths and limitations.
- [ ] Data handling with Pickle.

- [ ] [Calling WorkGraph within a WorkChain](http://127.0.0.1:8000/howto/workchain_call_workgraph.html): There may arise scenarios where integrating a WorkGraph workflow within a WorkChain. Try and give feedback on the current implementation.

- [ ] [ShellJob](https://aiida-workgraph.readthedocs.io/en/latest/built-in/shelljob.html): Plese check the docs, if there any important (new) features from latest `ShellJob` is not supported yet. @Sebastiaan or others who interested. Address issue [128](https://github.com/aiidateam/aiida-workgraph/issues/128) if you want.

- [ ] [Restart WorkGraph](https://aiida-workgraph.readthedocs.io/en/latest/howto/error_resistant.html) Enhance the basic error handler in WorkGraph to improve robustness. Try and give feedback on the current implementation.

- [ ] [While WorkGraph](https://aiida-workgraph.readthedocs.io/en/latest/howto/while.html): Test WorkGraph's handling of while loops with realistic examples to identify and address limitations.

- [ ] [Modify and restart a WorkGraph](https://aiida-workgraph.readthedocs.io/en/latest/howto/restart.html) and [Continue a finished WorkGraph](https://aiida-workgraph.readthedocs.io/en/latest/howto/continue_finished_workgraph.html): Explore the flexibility of modifying, restarting and continuing finished WorkGraphs. This differs from WorkChain behavior and feedback on this feature is valuable. Document experiences and suggestions.

- [ ] [Web GUI](https://aiida-workgraph.readthedocs.io/en/latest/gui/index.html): Address issues [119](https://github.com/aiidateam/aiida-workgraph/issues/119), [117](https://github.com/aiidateam/aiida-workgraph/issues/117).

- [ ] Web backend (REST API): The current backend using REST API is a minimum working version. There are a lot space for improvement, e.g., issues [159](https://github.com/aiidateam/aiida-workgraph/issues/159), [146](https://github.com/aiidateam/aiida-workgraph/issues/146)

- [ ] **Protocol** and **get_builder_from_protocol** is commonly used in WorkChain. Can we implement similar concept for WorkGraph? Here is a simple [protocol](https://aiida-workgraph.readthedocs.io/en/latest/howto/protocol.html), but would be great we implment a better one.

- [ ] Any issue outlined in [Github issue page](https://github.com/aiidateam/aiida-workgraph/issues) if you are interested in.

#### Use Cases

- [ ] It's would be great if you can share your future WorkGraph examples in the [workgraph-collections](https://github.com/superstar54/workgraph-collections) repository.

### The new `StructureData`

GitHub: https://github.com/aiidateam/aiida-atomistic

This one has been in the works for quite a bit as well. Miki still needs some time to work on the final changes (and might need some help here? [MBi]: yes, I guess at least another laptop could be helpful), so we'll work on this later in the week.
Representation of the new implementation is [here](https://www.figma.com/board/keVsAZb9FHzH1UIBMRCx7I/StructureData?node-id=0-1&t=dAPeuDlBVD94S9Nx-0)

### AiiDA-shell

Docs: https://aiida-shell.readthedocs.io/en/latest/index.html

This package is already more or less established, but I [MBx] for one have not used it too extensively. I think doing some dogfooding here would be good as well, in combination with the `WorkGraph`.

### New aiida-explorer

Repository: https://github.com/aiidateam/aiida-explorer

Discussion and demo Monday at 13:00 CET.

Notes: https://hackmd.io/@materialscloud/B1UW6UGOA/edit

Recording of the meeting: https://drive.google.com/drive/u/1/folders/1K9AR6rkxrDJko4xr_HKbKPkdxB5T3QKL

### Bits and pieces

**PRs**:

* Add the `SshFabricTransport` plugin [[#6154](
https://github.com/aiidateam/aiida-core/pull/6154)]: This plugin has potentially a lot impact, and should not be too much work to test and finalize.
* [MBx] 👌 Allow remote copying/symlinking between computers with same hostname #6196 (this is also important for `aiida-shell` usage, imo).
* [MBx] 👌 Return frozendict for stored Dict nodes #6185

### Other suggestions

* Discussion on comparison with other workflow engines.

## Discussion

What we'll be working on:

- Edan:
- Implementing a `WorkGraph` example for a Python package
- Test `PythonJob`'s functionality/usability
- Check Seb's [Pydantic PR](https://github.com/aiidateam/aiida-core/pull/6255)
- Julian & Marnik: Work on EOS example from workflow comparison: make simplest example.
- Julian: Down the road, look at `SshFabricTransport` PR
- Miki: Focus on `StructureData`: implementation of solution to immutability discussion.
- Alex:
- Xing: just be available for discussion/questions.
- Jusong:
- Review aiida firecrest PR by Ali.
- Check the old example I have with workgraph.
- Ali:
- Testing the RESTapi with `sqlite_zip` backend
- Firecrest PR
- `SshFabricTransport` PR.
- Kristjan:
- Demo/presentation of AiiDA-explorer
- On holiday Tue/Wed
- Thursday: work with Ali on testing the REST API with `sqlite_zip` backend
- Giovanni:
- Test WorkGraph for old phonon-mode-follower workflow

### Non-workgraph-related issues

#### SQLite DB locked when udpating last process state change
* In [this Discourse thread](https://aiida.discourse.group/t/getting-database-is-locked-error/432) we discuss a possible workaround to this apparently very common problem in the SQLite DB.
* JG will investigate if he can reproduce the issue. Will also open an issue on GitHub to report this. The current working hypothesis, somehow also supported by the [SQLite docs](https://www.sqlite.org/cgi/src/doc/begin-concurrent/doc/begin_concurrent.md), is that the error with the `process_state` change in the `DbSettings` table is the most frequent, because all processes are changing the very same row in the DB. When submitting, in general, instead, we typically append new rows (or changes to the same node are often serialized, in practice - two daemons would typically not change the same node). So we get no issues with DB locking (we are using WAL mode [journaled], and at commit time if writes are on different DB pages, the commit is performed without any complaint).

### `WorkGraph` API

#### [MBx] (How) should we integrate `calcfunction` and `pythonjob`?

With the addition of `PythonJob`, we now have two ways of executing Python code in AiiDA:

##### `calcfunction`

The `calcfunction` decorator allows the user to convert a Python function into one that is tracked in the provenance by AiiDA. Some notes:

1. Inputs and outputs have to be AiiDA `Node`s. For the inputs serialization from Python types is possible if implemented, but the outputs will always be AiiDA nodes.
1. Inside the body of the function, the input arguments will be `Node`s. This often means the code in the body of the function should be adapted.
1. As the `Node`s are stored upon execution of the `calcfunction`, the code in the body can never mutate on of the inputs.
1. The `calcfunction` is designed to run Python code locally, it cannot be used to execute Python code on a remote computer.

##### `PythonJob`

[docs](https://aiida-workgraph.readthedocs.io/en/latest/built-in/pythonjob.html)

The `PythonJob` was originally designed to run Python function on remote machines and track the inputs and outputs. However, it also does a few other things:

1. Inputs and outputs of the `PythonJob` can be both mutable ([XW]: raw Python data)
2. Automatical data serialization.
3. The body of the function is a normal Python function. ([MBx] thanks for already completing this a )

##### What I'd like to see

I'm wondering if we can somehow integrate the two tools into one. One tool for all execution of Python code, both local and immediate, as well as remote calculations.

from aiida_workgraph import WorkGraph, ShellJob


def add(x, y):
    return x + y

wg = WorkGraph("first_workflow")
wg.add_task(add, x=1, y=2)

echo_task = wg.add_task(ShellJob, command="echo")
wg.submit(wait=True)

* When passing e.g. `PwCalculation` to `add_task`, the inputs of `PwCalculation` become directly as inputs exposed to `add_task` directly -> Make this an explicit `inputs` directory.

wg.add_task(
    PwCalculation,
    name=<foo>, # This relates to the Task, not PwCalculation, and shadows PwCalculation name attribute 
    structure=<bar>,
    ...
)

Instead, we'll do:

wg.add_task(
    PwCalculation,
    name=<foo>, # This relates to the Task, not PwCalculation, and shadows PwCalculation name attribute 
    inputs = { # or task_inputs
        structure=<bar>,
        ...
    }
    ...
)

# Print out the result:

print("\nResult: ", date_task.outputs['stdout'].value.get_content())

##### Notes during discussion

Do we have to decorate the Python function? Can this not be done on the fly as the task is added to the `WorkGraph`?

### Waiting for a different process

Refer to: https://hackmd.io/7NvhuX1CTxql8AoQM2ua6A

### Syntax of adding task
Refer to: https://hackmd.io/erEjRpYrSUWEv3iRECSZdA

### Scrambled notes JG

https://aiida-workgraph.readthedocs.io/en/latest/concept/task.html#Decorator

GP:
1. Should we recommend WorkGraph to the broad community -> By the end of the week
2. Core subset of features to be released we all agree on -> XW will think about it and make a list of these features
3. Provide WorkGraph as a core feature -> Which things do we put in the core? If it's separate package, it might be difficult for people to find, and they need to install a new package.

SPH (Improvements of WG):
- Easy to take existing AiiDA entities, and chain them together -> No class required that needs to be discoverable by the daemon
- Better concurrency

## Final check-in

Note: I can't access some of the HackMD files above.

### Work today - next week

* Marnik:
* Finalise some open PRs:
* Add the `SshFabricTransport` plugin [[#6154](
https://github.com/aiidateam/aiida-core/pull/6154)]: Tested again and seems to work fine. Ali already gave a review, and might still be nice to get some more clear feedback as he suggested.
* 👌 Allow remote copying/symlinking between computers with same hostname #6196 (this is also important for `aiida-shell` usage, imo).
* 👌 Return frozendict for stored Dict nodes #6185