# AiiDA team last supper (backup 2)

_Please, all put the main topics you will be working on until the end of (MARVEL) funding. What are the main goals we want to achieve until then? What is feasible?_

_How much time can everybody realistically dedicate to what???_
_Expected person-months (PM)? Who would work on it? How would we release it? Technical implications, AEP, modules touched, etc.?_

## Actions (for myself)

- Create team compass issues for the things discussed here, the main topics
- Show this project at beginning of aiidateam meetings
- Assign/comment issues on team compass

## Keep in mind

- Usability (for users)
- Communication
- Blogs
- Reaching out during conferences
- Maintainability of the code (for us devs)
- Coding standards
- Team dynamics
- Communication in the team, between devs from different (non Computer Science) backgrounds
- Compatible with future workflows that have more focus on ML?

## AiiDA Team Planning: Main topics

### WorkGraph Release
- **Timeline:** Release end of Aug
- **Assigned:** **Xing**, Julian, Alex, Edan
- **Actions/Steps:**
1. Two weeks coding week (07.07. - 11.07., second week independent depending on availability)
1. Stable (API) release by end of summer, "as maintainable as possible"
- **Notes:** [AEP](https://github.com/aiidateam/AEP/pull/44)

### Engine Refactor and DSL
- **Timeline:** Technical assessment until mid Sep
- **Assigned:** Jusong, Alex, Edan*, (Ali - just as an observer)
- **Actions/Steps:** TBD
- **Notes:** Xing and team make sure to organize the required meetings

### Use Cases: Relaxation, Wannier, Phonons
- **Timeline:**
- 11/07: Primary discussion
- 17/07: Common list of issues
- 18/07: Issues on repos ready
- Implementation of deliverables (blog posts/tutorials)
- **Assigned:**
- Relax: (Kristjan + Mike + Timo)
- Wannier: (Julian + Natasha)
- Phonon: (Ali + Miki)
- **Actions/Steps:** Dedicated meetings between developer/user - go through the steps of using it and take note of user comments to address
- **Notes:** Good for marketing when discussion with commercial entities; Develop blogs/doc-examples

### AiiDA GUI (with REST API)
- **Timeline:** In two weeks (~12/07)
- **Assigned:** Xing, Edan, Kristjan, Bud
- **Actions/Steps:**
- Assess "plugin" infrastructure
- Provenance explorer as a subtask/milestone
- MC should use its common features, particularly the REST API
- **Notes:** If possible, keep [AEP](https://github.com/aiidateam/AEP/pull/45) minimal

### AiiDA Process Scheduler
- **Timeline:** TBD
- **Assigned:** Xing
- **Actions/Steps:**
1. Decide a name
2. Move out from WorkGraph
3. Ask people to try
- **Notes:** Assess if GUI is needed, or core functionality can be separated, no focus on that in the next few weeks until Sep

### Collaborative Work (Push/Pull)
- **Timeline:** Prep-work by end of Summer
- **Assigned:** **Edan**, Miki, Ali, Julian
- **Actions/Steps:**
- Consolidate docs
- Investigate feasibility
- **Notes:** TBD

### Documentation Accessibility + Tutorial Update/Preparation
- **Timeline:** TBD
- **Assigned:** **Ali**, Julian
- **Actions/Steps:** TBD
- **Notes:** Concise Overview & Basic Tutorial rework, tutorial by November; things should be known

### Usability
- **Timeline:** End of July project board in an updated state, discuss with people on issues, most important priorities by Aug, user priorities
- **Assigned:** **Julian**, Natasha (discussions)
- **Actions/Steps:** TBD
- **Notes:** Coordinate effort

### MFA
- **Timeline:** Ready by end of September.
- **Assigned:** Ali, Alex
- **Actions/Steps:** TBD
- **Notes:** TBD

### Website Re-design
- **Timeline:** Landing page by end of Summer
- **Assigned:** **Edan**, Ali
- **Actions/Steps:**
- Design meeting
- Assess tech stack upgrade
- Focus on landing page
- **Notes:** TBD

### Domain-Agnostic Core
- **Timeline:**
- StructureData by the end of august
- Plugin support by end of Sept
- **Assigned:** **Miki**
- **Actions/Steps:** TBD
- **Notes:** v3? Decide by Sep; possibly move out, but make `aiida-atomistic` a dependency of `aiida-core`, support of QE; use cases phonons/wannier/relax should support it

*TODO* How to organize the work? GitHub project, milestones?

| Person | July | August | September |
| --- | --- | --- | --- |
| Alex | WorkGraph Release (2 weeks), Sirocco (1 week), Engine refactor(1 week) | Engine refactor | Engine refactor, MFA (1 week) |
| Ali | Usability: phonons (20%), Open projects (80%)| collaborative-AiiDA(25%), Documentation (75%) | Documentation (25%), MFA (20%), Website Re-design (20%) |
| Edan | AiiDA GUI (20%); WG release (20%); DSL (10%); collab-AiiDA (10%); Website-redesign (20%); Non-AiiDA (20%) | AiiDA GUI (20%); WG release (20%); DSL (10%); collab-AiiDA (10%); Website-redesign (20%); Non-AiiDA (20%) | Non-AiiDA (100%) |
| Julian | aiida-workgraph (50%), Use cases (20%) | Usability (50%), Push/Pull (20%) | Usability (50%), Tutorial (30%) |
| Jusong | Engine refactor (50%) | Engine refactor(60%) | Engine refactor(70%) |
| Kristjan | use cases | - | REST api & provenance browser |
| Marnik | use case support | aiida-qe | aiida-qe |
| Miki | use cases | SData | collab work |
| Xing |WorkGraph and pyfunction, AiiDA GUI |AiiDA workGraph, AiiDA GUI | AiiDA Scheduler, AiiDA GUI |
| Natasha | use cases | use cases | use cases |
| Michael | use cases | - | - |

## Requirements

### Use cases: relaxation, wannier, phonons
Workflows already exists. People in our lab know how run it (Kristjan + Mike + Timo + Natasha). Create a form of documentation how to run it (Julian). If we decide to make blog post, it needs to only run for 2.7 (Julian). If feasible transform it into a stable form of documentation in the CI to keep thex eample stable (Ali).

### AiiDA GUI
Assessment of maintenability of plugin system (Edan).

### AiiDA Process Scheduler
Discussion should happen here https://github.com/aiidateam/AEP/pull/46
- Does it dependend on aiida-gui or can it be in aiida-core?
- Concerns about if it is neeeded. Michael said its added benefits are not convincing enough to change from the existing submisssion controller

## Pre-discussion

- How do we go forward with the workflow system: WG vs. Domain-Specific-Language (DSL), or both?
- engine refactor (JY & AG), and DSL -> Material/Presentation (10 Min + 20 Min)
- [xw] does the AEP avaiable for the DSL? It would be good that you already have some docs to share before the meeting. we people already get the idea, and ad comments, and have a better discusion tomorrow.
- Why do this? Native multi-threading, workflow validation, remove many dependencies
- Discuss a possible timeline tomorrow
- Can we decouple the new engine from the DSL?
- workgraph release (feature set?)
- AEPs
- [Workgraph](https://github.com/aiidateam/AEP/pull/44)
- [PythonJob](https://github.com/aiidateam/AEP/pull/43)
- [xw] Need at least two weeks to fix bugs, update docs, clean codes.
- [eb] How to integrate into aiida docs? Currently still WC in aiida-core? Probably we can keep it separate for now.
- API has to be stable before we start really publicizing it.
- Timeline
- Ask users which features they require, which are dependencies?
- Compare basic features between two approaches? What's the vertical slice/goal for each approach? And what are the technical requirements?
- aiida-gui: check this [AEP](https://github.com/superstar54/AEP/blob/aiida_gui/010_aiida_gui/readme.md).
- Xing, Edan, Kristjan, Bud
- core package [aiida-gui](https://github.com/aiidateam/aiida-workgraph-web-ui) is ready, with core layout, basic pages, basic API.
- plugin system is ready, and the first plugin [aiida-gui-workgraph](https://github.com/superstar54/aiida-gui-workgraph) also works.
- Todo
- Leverage the existing aiida-restapi endpoints wherever possible.
- Provide component Package: aiida-react-components-base, and publish it to npm -> shared by aiidalab, materialscloud, aiida-gui
- JupyterHub Integration
- If it's only our team, do we need a plugin system? Does plugin system make maintenance simpler? How much effort to make aiida-gui extendable to integrate additional parts?
- Focus on core features? What are the core featers -> AEP. Provide API entry points.
- Plugins can provide their own REST API endpoints -> Check if these are general enough to be moved to `aiida-restapi`?
- aiida-process-scheduler check this [AEP](https://github.com/superstar54/AEP/blob/aiida_scheduler/010_aiida_scheduler/readme.md).
- (make part of aiida-gui? extra package?)
- [xw] it should either be an independent package, (or part of aiida core).
- Timeline! Who works on this? Can we finalize this?
- pyfunction
- aiida-explorer
- Collaborative work with AiiDA, push/pull mechanism (GP, EB) -> EB present?
- EB as much time as possible over the summer, MB interested, Ali also, JG, as well
- EB, pls add text here
- making aiida-core domain-agnostic (new `StructureData`, MB)
- smaller tasks? (ideas and AEPs -> Which things we want/can get **done**??)
- Simplify computer/code setup
- `verdi computer search`
- MBx: `verdi backup restore` -> Is this feasible?
- General, concise overview of AiiDA in the docs
- Our docs lack examples
- pyiron, angular, react -> Website/landing page: What they are, scroll, how to install, how to get up and running
- Rework basic tutorial -> Move provenance to the end; Remove `verdi shell` (REPL)
- Problem with MFA. What to do long-term? -> Ali: We have a proper plan to implement that, first [PR #6761](https://github.com/aiidateam/aiida-core/pull/6761) has to merge. Then the implementation will follow.
- website design
- Edan and Ali to setup a meeting

## People's proposals

Xing:
- [WorkGraph](https://github.com/aiidateam/aiida-workgraph) release
- Xing, Julian, Alex, Edan
- Decide which syntax to keep supporting
- User feedback, what about the people we reached out to?
- Update docs (workchain to wg, new users, people with workflows already)
- **AiiDA GUI**, check this [AEP](https://github.com/superstar54/AEP/blob/aiida_gui/010_aiida_gui/readme.md). On top of current [aiida-workgraph-web-ui](https://github.com/aiidateam/aiida-workgraph-web-ui) repository, but make the WorkGraph part as a plugin. Use aiida-restapi as backend, test and update aiida-restapi.
- Xing, Edan, Kristjan, Bud
- Plugin-based system: AiiDA provenance explorer, scheduler plugin (for monitoring)
- **Event‑based, multi‑instance process scheduler**, check this [AEP](https://github.com/superstar54/AEP/blob/aiida_scheduler/010_aiida_scheduler/readme.md). Implement the Scheduler for high-throughput calculatioin. Generalize the [Scheduler](https://github.com/aiidateam/aiida-workgraph/pull/275) idea from WorkGraph to AiiDA.
- [aiida-pythonjob](https://github.com/aiidateam/aiida-pythonjob) release.
- **pyfunction**, check this [AEP](https://github.com/superstar54/AEP/blob/pyfunction/010_pyfunction/readme.md). Add a decorator and helper, pyfunction, that turns a plain function into an AiiDA process without rewriting the function body.
- pythonjob independent of wg
- Stabilize API
- Timeline? How much work? Who works on it?
- Can we integrate these features into calcjob/calcfunction? Are we losing something if we make calcfunction behave like pyfunction?

Alex:
- slides https://docs.google.com/presentation/d/1ZsPJkuGmvfXwZ5YGFo3pxeR3mCHFWuZZBWSw-iozKJo
- Refactor engine (plumpy/kiwipy) with centralized mulitthreading architecture
- Put `Waiting` state into plumpy which requires to extend workflow intermediate representation by transport/scheduler and monitor

Kristjan:
- aiida-explorer GUI -> Part of aiida-gui
- easily deployable in different contexts (also just spin up locally)
- visualize provenance at different levels/hierarchies
- local inputs/outputs per node
- "call stack"/workflow hierarchy: for a node, show all the parent callers
- global provenance
- ...

Julian:
- There are still some smaller modifications open that help usability (compare `verdi presto` [PR](https://github.com/aiidateam/aiida-core/pull/6351): ~250 lines added, provide great benefits)
- Simplify computer/code setup (see [AiiDA team compass](https://github.com/aiidateam/team-compass/issues/4), [this issue](https://github.com/aiidateam/aiida-code-registry/issues/63), and [this issue](https://github.com/aiidateam/aiida-core/issues/4680))
- Allow for AiiDAlab jinja placeholders in `aiida-core`
- Ignore non-supported sections, rather than crashing
- Merge `aiida-code-registry` and `aiida-resource-registry` (deprecate the former)
- Add `verdi computer setup search` to query repo and install Computer/Code directly from registry on GH
- `verdi backup restore` command

***

## AiiDA team compass

_Which issues do we tackle? Which are solved already? Which are not as important? Which are not feasible?_

- [ ] [Usability: Allow pure folder-based access and storage of AiiDA profiles](https://github.com/aiidateam/team-compass/issues/22)
- GP: I think this would help a lot in making multi-profile access simpler: if many agree, this could be among the usability tasks.
- [ ] [Future: make `aiida-core` fully domain-agnostic](https://github.com/aiidateam/team-compass/issues/21)
- GP: I would limit to the new structuredata (and in the process, we move out the few remaining data classes that are specific, like KpointsData)
- [ ] [Future: make migration of data types defined by plugin packages possible](https://github.com/aiidateam/team-compass/issues/20)
- GP: I think this is very complex. I would not do it, just provide guidelines on how to migrate, from point above.
- [x] [Usability: Make it easy to run Python-based codes](https://github.com/aiidateam/team-compass/issues/19)
- [xw] I think this is solved by [aiida-pythonjob](https://github.com/aiidateam/aiida-pythonjob)
- [ ] [Usability: Allow exporting of (parts of) the provenance graph with just metadata](https://github.com/aiidateam/team-compass/issues/18)
- GP: I would drop/close this. Rather, think at how to allow deletion of some nodes or data in some nodes (in an way that does not make sharing very hard)
- [ ] [Usability: Provide generic mechanism to serialize and deserialize data from and to AiiDA data](https://github.com/aiidateam/team-compass/issues/17)
- This should be solved by `pydantic`?
- [GP] possibly yes - good if e.g. Edan could check the Compass Issue and close it if it's solved
- [ ] [Usability: Make it possible to interact with an AiiDA instance over a web API](https://github.com/aiidateam/team-compass/issues/16)
- GP: I would focus on aiida-restapi in combination with AiiDA-GUI, then Materials Cloud uses only this and we drop the internal one. I would also separate the GraphQL into a separate package (to keep it, but we don't maintain it unless we have usecases)
- [ ] [Future: Allow users to use the optimade structure as first-class citizen that can import/export to AiiDA](https://github.com/aiidateam/team-compass/issues/15)
- GP: shall we close it? (@Jason what do you think)
- [ ] [Defect: Improve the caching policy when the RemoteData is cleaned up](https://github.com/aiidateam/team-compass/issues/13)
- This would require some further discussion, but it's probably not the highest priority. There could however be simple usability improvements on how users can query/control which nodes are being used for caching, clean their cache, etc. If we do it, I would put it under the usability tasks.
- [ ] [Usability: Make computers and codes entities more flexible](https://github.com/aiidateam/team-compass/issues/12)
- GP: this might also be related to the more general topic of allowing file deletion in nodes, for instance
- [ ] [Usability: Make it easier to backup and restore full AiiDA installations](https://github.com/aiidateam/team-compass/issues/11)
- GP: @Kristjan: is it feasible to have a restore command, or should we just point to the docs and close this issue?
- [ ] [Usability: Document AiiDA support of two-factor/multi-factor authentication (2FA/MFA)](https://github.com/aiidateam/team-compass/issues/10)
- [ ] Mainly documentation
- [ ] Does point #1 [here](https://github.com/aiidateam/aiida-core/issues/3929) now work with OpenSSH support?
- [ ] Check out [this repo](https://github.com/aiidateam/aiida-core/issues/3929#issuecomment-975390816) (but no pw in plain text pls) -> Does Archer currently work?
- [ ] [Usability: Allow users to relaunch a previously completed process with minimal required setup](https://github.com/aiidateam/team-compass/issues/9)
- [ ] Only functionality missing: Provide a utility that takes a `ProcessNode` and will relaunch it, while automatically setting up and configuring any needed `Computer` and `Code` instances as much as possible -> This should be fairly simple
- [ ] [Usability: Provide general overview of the AiiDA ecosystem on the docs](https://github.com/aiidateam/team-compass/issues/8)
- GP: would be very important
- [ ] [Fully migrate old roadmap](https://github.com/aiidateam/team-compass/issues/6)
- GP: At this point we should just close it
- [ ] [Usability: Make it easier for users to setup known computers and codes](https://github.com/aiidateam/team-compass/issues/4)
- [ ] Make `aiida-code-registry` and `aiida-resourcce-registry` easier findable and usable
- [ ] Support for OpenSSH makes the non-interactive setup already much easier
- [ ] Easy way to duplicate and setup working Computer/Code from imported archive
- [ ] https://github.com/aiidateam/aiida-code-registry/pull/62
- [ ] [Usability: Give users more control over process submissions](https://github.com/aiidateam/team-compass/issues/3)
- [ ] In principle, solved by the aiida-submission-controller
- [ ] How does XW's aiida-scheduler tie into this?
- [ ] What's the status of limiting number of jobs in `aiida-core`
- [ ] [Usability: Allow multiple calculations to be run as a single scheduler job](https://github.com/aiidateam/team-compass/issues/2)
- Sufficiently solved by `aiida-hyperqueue`?
- What about FirecREST (they are aware of hyperqueue and looking into iff it's possible)
- Can this be closed?
- GP: I this the only thing left is pointing to the plugin in the docs, so that users find it. Then we can close it. Maybe better to close this explaining it, but open a team compass issue (or expand the one on the docs) with clear pointers to these tools from the AiiDA docs.
- TASK: I would also reorganize the aiidateam GH organization keeping in there only things we will maintain as a team, and move to aiidaplugins things that we cannot maintain as a "core" team

## Other notes

- Cover both ends of the spectrum of users
- What to do about the many open issues? -> AiiDA core coding days
- There is a lot to read... AEPs for Xing's tools, team-compass issues, Oxiida documentation, etc. It's non-sensical to have _very_ detailed techincial discussions on these tools without having read thees documentts befohehand
- Move plugins from aiidateam org to aiidaplugins if they are not core plugins we plan to maintain
- All plugins have to be installable via `pip install aiida [aiida-*]`, here `[aiida-core]` would be installed by default. A team compass issue has to be opend on this. (JG: `aiida` meta-package?)

## Points from Marvel review

- Further simplifications in close communication with users outside the development effort
- Of the user interface for installation, and
- Creation of workflows
- Integration into commercial organizations -> Does AiiDA benefit from that??
- In addition to tutorials, give "masterclasses", where people come with a real-live problem and we help them address it using AiiDA
- Turning AiiDA into commercial software runs the risk of having to compete with established commercial products (e.g., apache airflow) -> evaluation with competing efforts must be done to identify what makes AiiDA unique
- Possibilites for commercialization:
- Spin-off
- Licensing that generates revenue from access, providing training, or custom workflows
- "Much of this decision will depend upon the career development plans of the leading developers"
- Possible benefits provided by AiiDA:
- protecting the value of data assets
- facilitating professional management of research data that:
- enables re-use and
- survives staff turnover
- reducing training and induction times for new staff

- GP: Tutorial in Nov: keep an intro tutorial, but then convert into Masterclass as recommended? Or do it in addition? Or skip tutorial (just update it - risk that we don't do it, though) and only do master class? It might be useful to assess in the AiiDA users community who would be interested (send email with registration form?)