# AiiDA team last supper

_Please, all put the main topics you will be working on until the end of (MARVEL) funding. What are the main goals we want to achieve until then? What is feasible?_

_How much time can everybody realistically dedicate to what???_
_Expected person-months (PM)? Who would work on it? How would we release it? Technical implications, AEP, modules touched, etc.?_

## Actions (for myself)

- Create team compass issues for the things discussed here, the main topics
- Show this project at beginning of aiidateam meetings
- Assign/comment issues on team compass

## Keep in mind

- Usability (for users)
- Communication
- Blogs
- Reaching out during conferences
- Maintainability of the code (for us devs)
- Coding standards
- Team dynamics
- Communication in the team, between devs from different (non Computer Science) backgrounds
- Compatible with future workflows that have more focus on ML?

## AiiDA Team Planning: Main topics

### WorkGraph Release
- **Timeline:** Release end of Aug
- **Assigned:** **Xing**, Julian, Alex, Edan
- **Actions/Steps:**
1. Two weeks coding week (07.07. - 11.07., second week independent depending on availability)
1. Stable (API) release by end of summer, "as maintainable as possible"
- **Technical Details:**
- [AEP](https://github.com/aiidateam/AEP/pull/44) available for review
- Need at least two weeks to fix bugs, update docs, clean codes
- API has to be stable before we start really publicizing it
- Decide which syntax to keep supporting
- Update docs (workchain to wg, new users, people with workflows already)
- Integration question: How to integrate into aiida docs? Currently still WC in aiida-core? Probably keep separate for now
- **Notes:** [AEP](https://github.com/aiidateam/AEP/pull/44)

### Engine Refactor and DSL
- **Timeline:** Technical assessment until mid Sep
- **Assigned:** Jusong, Alex, (Ali, Edan - just as an observer)
- **Actions/Steps:** TBD
- **Technical Details:**
- Why: Native multi-threading, workflow validation, remove many dependencies
- Can we decouple the new engine from the DSL?
- [Slides available](https://docs.google.com/presentation/d/1ZsPJkuGmvfXwZ5YGFo3pxeR3mCHFWuZZBWSw-iozKJo)
- **Notes:** Jusong and team make sure to organize the required meetings

### Use Cases: Relaxation, Wannier, Phonons
- **Timeline:**
- 11/07: Primary discussion
- 17/07: Common list of issues
- 18/07: Issues on repos ready
- Implementation of deliverables (blog posts/tutorials)
- **Assigned:**
- Relax: (Kristjan + Mike + Timo)
- Wannier: (Julian + Natasha)
- Phonon: (Ali + Miki)
- **Actions/Steps:** Dedicated meetings between developer/user - go through the steps of using it and take note of user comments to address
- **Technical Details:**
- [Team compass issue](https://github.com/aiidateam/team-compass/issues/23) available
- Workflows already exist, people in lab know how to run them
- Create documentation on how to run it (Julian)
- If blog post decided: needs to only run for 2.7 (Julian)
- If feasible: transform into stable form of documentation in CI to keep examples stable (Ali)
- Requirements: Ask users which features they require, which are dependencies?
- **Notes:** Good for marketing when discussion with commercial entities; Develop blogs/doc-examples

### AiiDA GUI (with REST API)
- **Timeline:** In two weeks (~12/07)
- **Assigned:** Xing, Edan, Kristjan, Bud
- **Actions/Steps:**
- Assess "plugin" infrastructure
- Provenance explorer as a subtask/milestone
- MC should use its common features, particularly the REST API
- **Technical Details:**
- Based on current [aiida-workgraph-web-ui](https://github.com/aiidateam/aiida-workgraph-web-ui) repository
- Make WorkGraph part as a plugin, use aiida-restapi as backend
- Core package [aiida-gui](https://github.com/aiidateam/aiida-workgraph-web-ui) ready with core layout, basic pages, basic API
- Plugin system ready, first plugin [aiida-gui-workgraph](https://github.com/superstar54/aiida-gui-workgraph) works
- Todo: Leverage existing aiida-restapi endpoints, provide component package: aiida-react-components-base (publish to npm), JupyterHub Integration
- Assessment needed: Does plugin system make maintenance simpler? How much effort to make aiida-gui extendable?
- Focus on core features, provide API entry points
- Plugins can provide own REST API endpoints -> Check if general enough for `aiida-restapi`, if so, include there
- **Notes:** If possible, keep [AEP](https://github.com/aiidateam/AEP/pull/45) minimal

### AiiDA Process Scheduler
- **Timeline:** TBD
- **Assigned:** Xing
- **Actions/Steps:**
1. Decide a name
2. Move out from WorkGraph
3. Ask people to try
- **Technical Details:**
- [AEP discussion](https://github.com/aiidateam/AEP/pull/46)
- Event-based, multi-instance process scheduler
- Generalize the [Scheduler](https://github.com/aiidateam/aiida-workgraph/pull/275) idea from WorkGraph to AiiDA
- Key questions: Does it depend on aiida-gui or can it be in aiida-core?
- Concerns: Michael noted added benefits not convincing enough to change from existing submission controller
- Should be either independent package or part of aiida-core
- **Notes:** Assess if GUI is needed, or core functionality can be separated, no focus on that in the next few weeks until Sep

### Collaborative Work (Push/Pull)
- **Timeline:** Prep-work by end of Summer
- **Assigned:** **Edan**, Miki, Ali, Julian
- **Actions/Steps:**
- Consolidate docs
- Investigate feasibility
- **Technical Details:**
- Push/pull mechanism implementation
- **Notes:**
- EB as much time as possible over the summer, MB interested, Ali also, JG as well

### Documentation Accessibility + Tutorial Update/Preparation
- **Timeline:** TBD
- **Assigned:** **Ali**, Julian
- **Actions/Steps:** TBD
- **Technical Details:**
- Tutorial by November; things should be known
- Our docs lack examples
- Rework basic tutorial -> Move provenance to the end; Remove `verdi shell` (REPL)
- General, concise overview of AiiDA in the docs
- **Notes:** Tutorial in Nov: keep intro tutorial, also Masterclass as recommended, do both?

### Usability
- **Timeline:** End of July project board in an updated state, discuss with people on issues, most important priorities by Aug, user priorities
- **Assigned:** **Julian**, Natasha (discussions)
- **Actions/Steps:** TBD
- **Technical Details:**
- Smaller modifications that help usability (compare `verdi presto` [PR](https://github.com/aiidateam/aiida-core/pull/6351): ~250 lines added, provide great benefits)
- Simplify computer/code setup: Allow AiiDAlab jinja placeholders in `aiida-core`, ignore non-supported sections rather than crashing, merge `aiida-code-registry` and `aiida-resource-registry`, add `verdi computer setup search`
- `verdi backup restore` command implementation
- **Notes:** Coordinate effort

### MFA
- **Timeline:** Ready by end of September.
- **Assigned:** Ali, Alex
- **Actions/Steps:** TBD
- **Technical Details:**
- Proper implementation plan exists
- First [PR #6761](https://github.com/aiidateam/aiida-core/pull/6761) has to merge, then implementation follows
- Mainly documentation needed
- Check: Does point #1 [here](https://github.com/aiidateam/aiida-core/issues/3929) now work with OpenSSH support?
- Check out [this repo](https://github.com/aiidateam/aiida-core/issues/3929#issuecomment-975390816) (but no pw in plain text) → Does Archer currently work?
- **Notes:** TBD

### Website Re-design
- **Timeline:** Landing page by end of Summer
- **Assigned:** **Edan**, Ali
- **Actions/Steps:**
- Design meeting
- Assess tech stack upgrade
- Focus on landing page
- **Technical Details:**
- Website/landing page needs: What they are, scroll, how to install, how to get up and running
- Reference examples: pyiron, angular, react
- **Notes:** TBD

### Domain-Agnostic Core
- **Timeline:**
- StructureData by the end of august
- Plugin support by end of Sept
- **Assigned:** **Miki**
- **Actions/Steps:** TBD
- **Technical Details:**
- New `StructureData` implementation
- Move out few remaining data classes that are specific (like KpointsData)
- v3? Decide by Sep; possibly move out, but make `aiida-atomistic` a dependency of `aiida-core`
- Support of QE; use cases phonons/wannier/relax should support it
- **Notes:**

### Python Job & PyFunction
- **Timeline:** TBD
- **Assigned:** **Xing**
- **Actions/Steps:**
- [aiida-pythonjob](https://github.com/aiidateam/aiida-pythonjob) release
- pythonjob independent of WG
- Stabilize API
- **Technical Details:**
- [AEP](https://github.com/superstar54/AEP/blob/pyfunction/010_pyfunction/readme.md) for pyfunction
- Add decorator and helper, pyfunction, that turns plain function into AiiDA process without rewriting function body
- Question: Can we integrate these features into calcjob/calcfunction? Are we losing something if we make calcfunction behave like pyfunction?
- **Notes:** Timeline and work assignment needs clarification

*TODO* How to organize the work? GitHub project, milestones?

| Person | July | August | September |
| --- | --- | --- | --- |
| Alex | WorkGraph Release (2 weeks), Sirocco (1 week), Engine refactor(1 week) | Engine refactor | Engine refactor, MFA (1 week) |
| Ali | Usability: phonons (20%), Open projects (80%)| collaborative-AiiDA(25%), Documentation (75%) | Documentation (25%), MFA (20%), Website Re-design (20%) |
| Edan | AiiDA GUI (20%); WG release (20%); DSL (10%); collab-AiiDA (10%); Website-redesign (20%); Non-AiiDA (20%) | AiiDA GUI (20%); WG release (20%); DSL (10%); collab-AiiDA (10%); Website-redesign (20%); Non-AiiDA (20%) | Non-AiiDA (100%) |
| Julian | aiida-workgraph (50%), Use cases (20%) | Usability (50%), Push/Pull (20%) | Usability (50%), Tutorial (30%) |
| Jusong | Engine refactor (50%) | Engine refactor(60%) | Engine refactor(70%) |
| Kristjan | use cases | - | REST api & provenance browser |
| Marnik | use case support | aiida-qe | aiida-qe |
| Miki | use cases | SData | collab work |
| Xing |WorkGraph and pyfunction, AiiDA GUI |AiiDA workGraph, AiiDA GUI | AiiDA Scheduler, AiiDA GUI |
| Natasha | use cases | use cases | use cases |
| Michael | use cases | - | - |

## Requirements

### Use cases: relaxation, wannier, phonons
Workflows already exists. People in our lab know how run it (Kristjan + Mike + Timo + Natasha). Create a form of documentation how to run it (Julian). If we decide to make blog post, it needs to only run for 2.7 (Julian). If feasible transform it into a stable form of documentation in the CI to keep thex eample stable (Ali).

### AiiDA GUI
Assessment of maintenability of plugin system (Edan).

### AiiDA Process Scheduler
Discussion should happen here https://github.com/aiidateam/AEP/pull/46
- Does it dependend on aiida-gui or can it be in aiida-core?
- Concerns about if it is neeeded. Michael said its added benefits are not convincing enough to change from the existing submisssion controller

## AiiDA team compass

_Which issues do we tackle? Which are solved already? Which are not as important? Which are not feasible?_

- [ ] [Usability: Allow pure folder-based access and storage of AiiDA profiles](https://github.com/aiidateam/team-compass/issues/22)
- GP: I think this would help a lot in making multi-profile access simpler: if many agree, this could be among the usability tasks.
- [ ] [Future: make `aiida-core` fully domain-agnostic](https://github.com/aiidateam/team-compass/issues/21)
- GP: I would limit to the new structuredata (and in the process, we move out the few remaining data classes that are specific, like KpointsData)
- [ ] [Future: make migration of data types defined by plugin packages possible](https://github.com/aiidateam/team-compass/issues/20)
- GP: I think this is very complex. I would not do it, just provide guidelines on how to migrate, from point above.
- [x] [Usability: Make it easy to run Python-based codes](https://github.com/aiidateam/team-compass/issues/19)
- [xw] I think this is solved by [aiida-pythonjob](https://github.com/aiidateam/aiida-pythonjob)
- [ ] [Usability: Allow exporting of (parts of) the provenance graph with just metadata](https://github.com/aiidateam/team-compass/issues/18)
- GP: I would drop/close this. Rather, think at how to allow deletion of some nodes or data in some nodes (in an way that does not make sharing very hard)
- [ ] [Usability: Provide generic mechanism to serialize and deserialize data from and to AiiDA data](https://github.com/aiidateam/team-compass/issues/17)
- This should be solved by `pydantic`?
- [GP] possibly yes - good if e.g. Edan could check the Compass Issue and close it if it's solved
- [ ] [Usability: Make it possible to interact with an AiiDA instance over a web API](https://github.com/aiidateam/team-compass/issues/16)
- GP: I would focus on aiida-restapi in combination with AiiDA-GUI, then Materials Cloud uses only this and we drop the internal one. I would also separate the GraphQL into a separate package (to keep it, but we don't maintain it unless we have usecases)
- [ ] [Future: Allow users to use the optimade structure as first-class citizen that can import/export to AiiDA](https://github.com/aiidateam/team-compass/issues/15)
- GP: shall we close it? (@Jason what do you think)
- [ ] [Defect: Improve the caching policy when the RemoteData is cleaned up](https://github.com/aiidateam/team-compass/issues/13)
- This would require some further discussion, but it's probably not the highest priority. There could however be simple usability improvements on how users can query/control which nodes are being used for caching, clean their cache, etc. If we do it, I would put it under the usability tasks.
- [ ] [Usability: Make computers and codes entities more flexible](https://github.com/aiidateam/team-compass/issues/12)
- GP: this might also be related to the more general topic of allowing file deletion in nodes, for instance
- [ ] [Usability: Make it easier to backup and restore full AiiDA installations](https://github.com/aiidateam/team-compass/issues/11)
- GP: @Kristjan: is it feasible to have a restore command, or should we just point to the docs and close this issue?
- [ ] [Usability: Document AiiDA support of two-factor/multi-factor authentication (2FA/MFA)](https://github.com/aiidateam/team-compass/issues/10)
- [ ] Mainly documentation
- [ ] Does point #1 [here](https://github.com/aiidateam/aiida-core/issues/3929) now work with OpenSSH support?
- [ ] Check out [this repo](https://github.com/aiidateam/aiida-core/issues/3929#issuecomment-975390816) (but no pw in plain text pls) -> Does Archer currently work?
- [ ] [Usability: Allow users to relaunch a previously completed process with minimal required setup](https://github.com/aiidateam/team-compass/issues/9)
- [ ] Only functionality missing: Provide a utility that takes a `ProcessNode` and will relaunch it, while automatically setting up and configuring any needed `Computer` and `Code` instances as much as possible -> This should be fairly simple
- [ ] [Usability: Provide general overview of the AiiDA ecosystem on the docs](https://github.com/aiidateam/team-compass/issues/8)
- GP: would be very important
- [ ] [Fully migrate old roadmap](https://github.com/aiidateam/team-compass/issues/6)
- GP: At this point we should just close it
- [ ] [Usability: Make it easier for users to setup known computers and codes](https://github.com/aiidateam/team-compass/issues/4)
- [ ] Make `aiida-code-registry` and `aiida-resourcce-registry` easier findable and usable
- [ ] Support for OpenSSH makes the non-interactive setup already much easier
- [ ] Easy way to duplicate and setup working Computer/Code from imported archive
- [ ] https://github.com/aiidateam/aiida-code-registry/pull/62
- [ ] [Usability: Give users more control over process submissions](https://github.com/aiidateam/team-compass/issues/3)
- [ ] In principle, solved by the aiida-submission-controller
- [ ] How does XW's aiida-scheduler tie into this?
- [ ] What's the status of limiting number of jobs in `aiida-core`
- [ ] [Usability: Allow multiple calculations to be run as a single scheduler job](https://github.com/aiidateam/team-compass/issues/2)
- Sufficiently solved by `aiida-hyperqueue`?
- What about FirecREST (they are aware of hyperqueue and looking into iff it's possible)
- Can this be closed?
- GP: I this the only thing left is pointing to the plugin in the docs, so that users find it. Then we can close it. Maybe better to close this explaining it, but open a team compass issue (or expand the one on the docs) with clear pointers to these tools from the AiiDA docs.
- TASK: I would also reorganize the aiidateam GH organization keeping in there only things we will maintain as a team, and move to aiidaplugins things that we cannot maintain as a "core" team

## Other notes

- Cover both ends of the spectrum of users
- What to do about the many open issues? -> AiiDA core coding days
- There is a lot to read... AEPs for Xing's tools, team-compass issues, Oxiida documentation, etc. It's non-sensical to have _very_ detailed techincial discussions on these tools without having read thees documentts befohehand
- Move plugins from aiidateam org to aiidaplugins if they are not core plugins we plan to maintain
- All plugins have to be installable via `pip install aiida [aiida-*]`, here `[aiida-core]` would be installed by default. A team compass issue has to be opend on this. (JG: `aiida` meta-package?)

## Points from Marvel review

- Further simplifications in close communication with users outside the development effort
- Of the user interface for installation, and
- Creation of workflows
- Integration into commercial organizations -> Does AiiDA benefit from that??
- In addition to tutorials, give "masterclasses", where people come with a real-live problem and we help them address it using AiiDA
- Turning AiiDA into commercial software runs the risk of having to compete with established commercial products (e.g., apache airflow) -> evaluation with competing efforts must be done to identify what makes AiiDA unique
- Possibilites for commercialization:
- Spin-off
- Licensing that generates revenue from access, providing training, or custom workflows
- "Much of this decision will depend upon the career development plans of the leading developers"
- Possible benefits provided by AiiDA:
- protecting the value of data assets
- facilitating professional management of research data that:
- enables re-use and
- survives staff turnover
- reducing training and induction times for new staff

- GP: Tutorial in Nov: keep an intro tutorial, but then convert into Masterclass as recommended? Or do it in addition? Or skip tutorial (just update it - risk that we don't do it, though) and only do master class? It might be useful to assess in the AiiDA users community who would be interested (send email with registration form?)