# Tasking System Interview

## Title

Overview of the Queue-less Tasking System

## Description

An interview with Matthias Dellweg, who created the queue-less tasking system to be introduced in pulpcore==3.14. Learn about the motivations for making it, the benefits it provides, and a bit about how it works.

## Intro Text
Matthias joined the Pulp team after contributing to Foreman and Pulp for about three years.
He is mostly contributing to pulpcore, the container plugin and the CLI.
As a physicist, he sometimes brings a unique natural science view to the table.

## Questions

Q: What background info do you want viewers to understand to be prepared for today's conversation?
A: The current system is based on what was used with Pulp 2; monolithic;

Q: Pulp already has a tasking system, why does it need a new one?
A: limited throughput of 2 tasks / sec. Not highly availabile. Reliability issue due to psql+redis
theoretical possibility of non optimal system utilization

Q: When is this availabile for users?
A: 3.14, releasing this week

Q: How do users begin using it?
A: it's enabled by default

Q: Do I have to use the new tasking system?
A: No, users can set the USE_NEW_WORKER_TYPE = False to continue using the old tasking system. Drain your queues first!
Also be prepared that we plan to remove it in some later Release.

Q: How does the queue-less tasking system provide high availability?
A: There is no resource manager anymore, which was the single point of failure. Also RQ never supported clustered redis, and by not using RQ anymore we depend only on services that can be clustered, e.g. psql.

Q: What happened to the resource manager?
A: It went away, it's no longer needed
You may still see a dummy process for easier deployment.

Q: What if I start the resource manager with the queue-less system? Will it error?
A: no

Q: How much faster is the queue-less tasking system in terms of throughput?
A: Basically it gets faster with more concurrent running workers. Also it reduces about half a second of overhead for every task from being processed in the resource-manager.

Q: Is the new queue-less tasking system more reliable?
A: Yes much more reliable for two reasons. 1) We no longer have to keep data synchronized between Redis AND Postgresql. 2) The new system uses psql advisory locks, which are guaranteed to auto-cleanup when connections close unexpectedly.

Q: Will tasks still cancel unexpectedly with the queue-less tasking system?
A:

Q: Does the new tasking system use Redis at all?
A: Not at all. The use of Redis leads to the task data duplicated in two databases. And this has been identified as a cause for may problems.

Q: So does Pulp still need Redis?
A: Yes, the new content caching speedups in 3.14 will use Redis if it's available.

Q: Queues typically are the traditional tool for tasking systems, why isn't Pulp using queues?
A: Pulp has a certain 1st come 1st serve policy on resources. That introduces a dependency graph on the tasks. In the queued tasking system the resource-manager was responsible to satisfy those dependencies by delivering tasks to dedicated worker queues. Once queued for a worker, a task was tied to that worker.

Q: From a high level, how does the algorithm work?
A: Like an team of developers. Whenever some worker is idle, it looks for pending tasks starting at the oldest. It keeps a record of in-use resources, and picks the next task that does not require any of those. It then grabs for an exclusive lock on that task, and starts to work on it.
On failure, it continues with the next task in the list.

Q: How can users get help or give feedback on this?
A: Go to pulpproject.org
Mailinglist
We are also on Matrix/IRC

###### tags: `Tasking System`