# Usability: Task farming **Goal**: Running multiple AiiDA jobs as a single scheduler (e.g. SLURM) job. **State**: `proposed` ## Motivation Currently, the AiiDA engine will submit one job to the scheduler for each calculation job. The ability to run multiple AiiDA jobs inside one scheduler job has several use cases: 1. In computing centers that do not allow node-sharing between scheduler jobs, this allows AiiDA users to run multiple calculation jobs that only partially use a node in parallel. 2. Some schedulers might struggle to deal with managing a lot of jobs simultaneously, and hence running many small jobs in high-throughput can become problematic. 3. Packaging multiple AiiDA jobs into one scheduler job means that the AiiDA jobs will only have to queue once. 4. The scheduler configuration might restrict the number of active jobs at one time. ## Desired Outcome Have at least straightforward approach to pack multiple AiiDA jobs in one scheduler job, which is well documented and easy to find. ## Impact Any user that cannot efficiently use a full node on their computing center will benefit from use case [1], and we've already had several users request this feature for this reason. Avoiding queueing times (use case [3]) is beneficial to pretty much all users, especially if they are running workflows with many short steps. Use cases [2] and [4] are especially important to users that run many workflows in high-throughput. ## Complexity Most current approaches to implementing task farming rely on using a meta-scheduler (see Progress below). This requires implementing a new AiiDA scheduler, which depending on the meta scheduler is a matter of a few days work. Since we already have several such implementations, the main work that is left is to properly test/documents these and make sure users can find them by pointing to them from the main AiiDA documentation. ## Background This issue was originally raised in the 2020 AiiDA hackathon in Bologna. Also see https://github.com/orgs/aiidateam/discussions/5112 for a more recent discussion on the topic. The main gist of these conversations is that we want to allow task farming through the use of a suitable meta-scheduler. ## Progress There are already two existing scheduler implementations for dealing with task farming: - [`aiida-fireworks-scheduler`](https://github.com/zhubonan/aiida-fireworks-scheduler) using the workflow manager [FireWorks](https://materialsproject.github.io/fireworks/). - [`aiida-hyperqueue`](https://github.com/aiidateam/aiida-hyperqueue): using the [HyperQueue](https://it4innovations.github.io/hyperqueue/stable/) meta-scheduler. Both approaches can in principle deal with all use cases presented in the Motivation section.