# Cron/Scheduler feature

## Use cases

* Monthly billing. Needs a specific time of the day on a specific day of the month when it's executed.
* (Aris) Read sensor data every 30 seconds. Needs sub-minute precision of scheduling, also needs to skip missed ticks as each sensor invocation has an associated cost in the real world, and only the last values are interesting.

## Semantics/User experience

This is the user experience/semantics we settled on:

* User sets up a schedule with the Cron format with second precision. This is an extension of the cron format used by `crontab`, used in a couple of projects such as [Spring](https://spring.io/blog/2020/11/10/new-in-spring-5-3-improved-cron-expressions) and [Quartz](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html).
* The cron is configured using an annotation on the handler, e.g. in Java/Kotlin `@Scheduled(cron = "10 * * * * *")`
* Only regular service handlers can be annotated with the `@Scheduled` annotation. Virtual objects and Workflows handlers cannot use it.
* Through the CLI, a bunch of commands are exposed to observe the list of existing crons, the last X execution failures (if any).
* The default behaviour on missed ticks is to skip the invocation when the endpoint is down. In other words:
* If the endpoint is down, or takes long to reply, and a tick passes, that tick is missed.
* If the runtime is down, and at least one tick passes, the last tick should be executed. In practice, on recovery, Restate needs to figure out whether at least one tick elapsed, and needs to execute this.
* The handler MUST be empty request/empty response.
* On re-discovery, old crons are removed and new ones are registered. The first execution will be the next tick. In-flight invocations will continue to be processed by the old endpoint, as usual.

## Implementation

The rough idea is to reuse the existing timers/scheduled invocation system, and store some additional data in the partition processor to start/stop crons.

### Details

The schema registry stores crons using the `Subscription` entity. During the discovery process, new `Subscription`s for new crons are created, and old `Subscription`s for old crons are removed.

Crons get a unique id associated on creation. This id is used to determine the partition key, thus the partition that needs to process a given cron.

The PP has a task (running only on the leader) that listens schema registry changes, takes the relevant crons and proposes them to the PP.

The PP internally handles the registered crons, and uses the existing persisted timer system to trigger timers. To do so, we need the new following data structures:

* A new table kind in the partition-store called `Cron`, which tracks which crons are running on this partition, and what are the last invocations associated with it.
* A new command `SyncCrons` to update the `Cron` table, containing all the crons associated with the partition, and the current wall-clock. The size of this command might be optimized in future by doing a query to the partition store first to check the `Cron` table content.
* The Timer Kind data structure gets a new variant `CronScheduled`, similar to `Scheduled` invocation containing the invocation id of the next cron invocation, and the cron id.
* The InvocationStatus.Source also gets a new variant `Cron`, with the cron id.
* We also need to add the wall clock to the `InvokerEffect::End` command

**On `SyncCrons` command:**

Look up the `Cron` table, determines which crons are old/new:

* For the old crons, PP simply removes them from the table
* For new crons, PP uses the wall clock time from `SyncCrons` command to schedule invocations (using the `InvocationStatus::Scheduled` and `Timer::CronScheduled`)

**On `Timer::CronScheduled` fired command:**

Check if the `Cron` still exists, if it doesn't then skip the handling.

If the `Cron` exists, then check if the last invocation associated to it is still running. If yes, then skip, otherwise start the invocation (from now on, same semantics of scheduled invocations).

**On `InvokerEffect::End` command:**

If the `InvocationStatus.Source` is cron, then check in the `Cron` table if the cron id still exists. If so, then schedule the next invocation using the cron expression in the `Cron` table, and the wall-clock from the invoker effect (essentially same behaviour as on `SyncCrons`).

## Discarded ideas

* Support an interval format, e.g. `every 10 minutes`. This can get unpredictable when used in conjunction with annotations, as now the start-time now depends on some weird combination of when you perform registration, and what happens within Restate. It gets even more complicated (and unpredictable for the user) if you change the format expression on a service upgrade, or change the handler name in such a way that you can't easily correlate with the previously registered intervals, etc. On top of that, there are some use cases that require cron anyway: the monthly billing as an example, which needs to run at a specific hour of a specific day of the month.
* Implement the cron scheduling outside the PP a'la ingress, or a'la kafka subscription. This works well until we need to sort out the failover procedure, because it requires to find out whether the runtime missed some ticks when it was down, and doing so requires some coordination between this (wannabe stateless) cron scheduler component and the PPs.

## Other considerations

* The current proposal makes registration of crons static and tied to the discovery process, but the proposed implementation it's not tied to this behaviour. We can make the cron registration in future more dynamic, e.g. exposing:
* A CLI/Admin API command to register/remove crons. This is perhaps something we might need rather sooner than later, in case some SDKs will have issues with the annotation approach.
* Even a `Context` API, in this case we can get away without registering them in the schema registry.
* When scheduling the next cron invocation, we can compute the next X cron invocations of the same cron, such that we can compute the retention time to be able to retain the last X invocations and their results.
* On the datafusion side, we should perhaps come up with a view like `sys_invocation` that merges the schema registry info with the `Cron` table stored in each PP.
* All crons will be UTC based at the beginning. We can extend the cron expression format to support other time zones too.
* If a user wants the "burst" behaviour on missed ticks, we can do so by sending a payload to the cron registered handler containing the elapsed time before the last execution and the current execution, such that the user can implement some business logic to compensate for missing ticks.