###### tags: `usability` [TOC] # Recent improvements in user friendliness in AiiDA Dear users, As we are always working hard to improve your experience with AiiDA, we would like to share with you some of the recent chanegs that have been implemented in the code to improve user friendliness. ## Service-less installation While `PostgreSQL` as database backend and `RabbitMQ` as message broker were initially chosen with high-performance and high-throughput capabilities in mind, we have found over the years that their setup can pose a significant initial hurdle for new users. We estimate that about half of the issues reported on Discourse are related to the setup of these services. In addition, the extra performance gains they provide become really only noticeable for the workloads that power users put on the system. Recent changes to the code base have now made it possible to run AiiDA entirely without external services, effectively shrinking the commands required to obtain a running setup to just two: ```shell= pip install aiida-core verdi init/blitz/presto ``` ### Database This is achieved by providing `SQLite` as alternative database backend. As it does not need to run as an external service no additional setup steps are required. A [dedicated section](https://aiida.readthedocs.io/projects/aiida-core/en/v2.5.0/topics/storage.html) was added to the documentation to give a short overview of the available database backends, and when to use which. ### Message Broker Similarly, the use of `RabbitMQ` was made optional. ### Profile setup In addition, we added the `verdi profile setup` command. Therefore, these two changes allow spinning up a working AiiDA instance as simple as: ```shell pip install aiida-core verdi init ``` no additional services required. ## Adaptation from typical file-system Another hurdle that often complicates the adaptation of AiiDA is the conceptual difference between the hierarchical file-system approach most people are familiar with, and the database-driven data organization implemented in AiiDA. While the raw files are of course available, accessing them is not straightforward: - The local file repository depends on the fairly complex [`disk-objectstore`](https://github.com/aiidateam/disk-objectstore), while - On the remote computer, files are stored in a hierarchical structure with folder names based on the universally unique identifiers (UUIDs) of the corresponding nodes. The second point here nicely illustrates the issue - with high perforamnec in mind, the data storage logic was constructed to be *machine-readable* rather than *human-readable*. As such, a new user is automatically fairly locked into the database-driven approach taken by AiiDA and has to effectively use the `verdi` CLI interface or the Python API, such as the `QueryBuilder` to access the data. For that reason, we have added two functionalities that ease the adaptation from the typical file system approach. ### Converting individual processes to file hierarchies First, we have added the ```shell verdi process dump <pk> ``` command that allows the dumping of `CalcJob`s and `WorkChain`s in a directory tree with meaningful directory names. For instance, for a QuantumEspresso `pw.x` calculation, the corresponding directory tree takes has the form: ```shell dump-PwCalculation-42 ├── README ├── raw_inputs │ ├── _aiidasubmit.sh │ └── aiida.in ├── raw_outputs │ ├── _scheduler-stderr.txt │ ├── _scheduler-stdout.txt │ ├── aiida.out │ └── data-file-schema.xml └── node_inputs └── pseudos └── Si └── Si.pbesol-n-rrkjus_psl.1.0.0.UPF ``` As you can see, the main QuantumEspresso input file, internally named `aiida.in`, as well as the submission script `_aiidasubmit.sh` are contained in the `raw_inputs` directory. The relevant output files generated by the calculation are written to the `raw_inputs`, while additional files, such as the Silicon pseudopotential is contained in the `node_inputs`. A README is also generated for each dump, which contains a more in-depth explanation of the output. The command works accordingly for more complex workflows. For instance, for an exemplary `PwBandsWorkChain` that calculates a material's band structure using `QuantumEspresso`, the following directory structure is obtained: ```shell dump-PwBandsWorkChain-462 ├── 01-relax-PwRelaxWorkChain │ ├── 01-PwBaseWorkChain │ │ └── 01-PwCalculation │ │ ├── raw_inputs │ │ ├── raw_outputs │ │ └── node_inputs │ │ └── pseudos │ │ └── Si │ └── 02-PwBaseWorkChain │ └── 01-PwCalculation │ ├── raw_inputs │ ├── raw_outputs │ └── node_inputs │ └── pseudos │ └── Si ├── 02-scf-PwBaseWorkChain ... ``` Here, the generated directory tree mirrors the logical sequence of steps run in the workflow, and output files are again contained in the folders for each calculation. The functionality allows you to explore all files involved in the execution of a multip-step workflow in the same way as a for a traditional directory tree, using the typical shell tools like `grep`, `sed`, and `awk`. ### Mirroring an AiiDA archive to disk ...to be implemented... In a similar way as the dumping of individual process outlined in the previous section, we have further added the functionality to mirror a collection of (process) files to disk. The command is aptly named: ```shell verdi archive mirror <pk> ``` It creates a directory tree in which all workflows, calculations, and structures are written to disk. This differs from the already existing `verdi archive export`, which exports a collection of AiiDA nodes, that is, the `SQL` database and the file repository, such that it can then be imported by another AiiDA instance. In that case, the resulting `.aiida` archive is not an easily explorable directory tree. ... ## Writing workflows ### WorkTree, SubmissionController, Other? -> Future? ### Automatic serialization of base Python types ## Outlook - Clean-up of setup commands, however, backwards-incompatible, so must wait for v3 --- ## Miscellaneous - New QueryBuilder API by Edan - Live calculation job monitoring? ## Relevant PRs (for own reference) - Add the `SqliteDosStorage` storage backend [[702f88788]](https://github.com/aiidateam/aiida-core/commit/702f8878829b8e2a65d81623cc2238eb40791bc6) - CLI: Add the command `verdi profile setup` [[351021164]](https://github.com/aiidateam/aiida-core/commit/351021164d00aa3a2a78b5b6e43e8a87a8553151)