--- # System prepended metadata title: 'Scheduling: A Tech-Tree' --- # Scheduling: A Tech-Tree This, broadly, is my vision for bevy's scheduling infastructure: 1. Scheduling-related code should be identical and uniform across all platforms 2. Platform-scpecific scheduling restrictions should not leak out of `cfg!` blocks 3. Multithreading should be the default on all supported platforms 4. Users should not have to write custom executors/pools code to get good performance Achiving this goal is going to take a time. There will be small incremental improvements; there will be some large-scale overhauls. If it were a tech-tree in a 4X game, the path before us might look something like this... ``` [Windowing Jobs] | v [Send ECS] | +----> [Non-Send ECS Scheduling] v [In-ECS Window Logic] | v [Decoupled Windowing] | | [Timed Inputs] <--+ | [Frame Pacing] <--+ | [Async ECS] <--+ | [Fix Web Task] | | | v | [Remove ConditionalSend] | | | v v | [ECS Web-Workers] v [Fork/Join] | v [Parallel Iterators] [Prevent System Priority-Inversion] | v [Reduce Scheduler Overhead] | v [System Cache Locality] ``` ## Windowing Jobs As described [here](https://hackmd.io/@bevy/winit_refactorening#11-Main-Thread-Task-Queue). The first step in decoupling the ECS from winit to allow external threads to inject arbitrary work into the winit event loop. ## Send ECS As described [here](https://hackmd.io/@bevy/winit_refactorening#12-Main-Thread-Data-Storage). Making the ECS send allows it to be initalized on the main thread, and then moved into a second thread once windowing starts. This will require removing or altering the current "non-send resources" pattern. In the narrow term we will focus on "main-thread-locked" data; we will store these in thread locals on the main thread, and access them through windowing jobs. ## Non-Send ECS Scheduling Charlotte has pointed out that the plan layed out in the [The Great Winit Refactorening](/CfYTEWOFS1mVA9VZ4yWT2A) will leave something wanting for users who need non-send data that dosn't need to be "main-thread-locked" (such as an virtual machine or interpreter). Ideally, we could continue to give information about non-send attachment to the ECS, and have it schedule systems that _must run_ on the thread where that data lives. In principle, that need not be the main ECS thread, and systems that access non-send attachments need not be exclusive. This is still very much in the ideation phase. ## In-ECS Window Logic As described [here](https://hackmd.io/@bevy/winit_refactorening#13-Windowing-Events-Channel). ## Decoupled Windowing As described [here](https://hackmd.io/@bevy/winit_refactorening#2-Demolishing-WinitAppRunnerState). Decoupled rendering means decoupling the ecs update loop from the windowing update loop; windowing will send the ecs messages, the ecs will schedule tasks on the windowing loop, both happen asychronously. For the web, this will allow us to either run ecs updates in a task concurrently to windowing, or in a web-worker. ## Timed Inputs Timed inputs are a pretty simple extension of decoupled rendering; we simply take a timestamp when we receive an input event from winit, but before queueing it for handling withing the ECS. The lsss work we do within the windowing thread, the more accurate these timestamps will be. This isn't a perfect method for input timestamps by any means, but it is the best we have avalible to us. ## Frame-Pacing Once we have accurate frame timing statistics, and windowing is decoupled from the windowing loop, we can try to delay the start of an ECS update such that we finish rendering the next frame just before the present boundary. ## Async ECS Having decoupled the ECS event loop from windowing, we can start to think about running them concurrently on the web. If we make `World::run_schedule` an async function, then we can make the entire ECS update into a future, which we can pass off to the web's async runtime. To carry this off properly, we would likely need to make `SystemExecutor::run` async as well. I think this would probably help to disentangle system execution from the async runtime, at least somewhat. This might also have benifits for `no_std` targets, where projects like `Embasy` use async extensively. When nessicary, for testing, we can just use `block_on` or something similar. ## Fix Web-Task A `Task` on the web target has totally different semantics from all other targets. On other targets it is provided by the `async-task` crate, and on web we mock this type. But as the `async-task` api has grown, our mocked api has not. What makes this different a real pain is the cancel semantics; dropping a task cancels the assoceated future on all platforms _expect web_. The `cancel(self)` method likewise on web has no effect. This tends to lead to a proliferation of web-specific conditionals throughout mixed-platform bevy codebases. The solution: replace `web-bindgen-futures` with `web-task` (which I wrote for this purpose). The latter is just a rewrite of the former which uses `async-task`, and therefore has identical semantics to desktop platforms. ## Remove ConditionalSend `ConditionalSend` is a trait that is equivilent to `Send` on all platforms _except_ web (where it is meaningless). It exists because `JsFuture` (the future version of a java-script promise) is `!Send`. Since many browser apis return this type, and any future that awaits a non-send future itself becomes nonsend, many people make the assumption that "futures are nonsend on the web", and thus need special allowences. This is wrong. It is trivial to convert a `!Send` future into a `Send` future using tasks. ```rust let task: Task = spawn_local(async { let js_result = nonsend_browser_api().await; MySendType::from(js_result) }); ``` As long as `MySendType` is `Send`, the task will be a `Send` future. We should adopt this "nonsend-conversion" pattern instead of allowing `ConditionalSend` to leak into the wider codebase. This will ultametly allow us to send more work to be performed within web-workers. >[!Note] > While we're on this topic, I'd also like the closely related `MaybeSend/MaybeSinc` traits. These are `Send` and `Sync` only `no_std` targets (and are meaningless on std targets). This limitation exists because the no-std async executor we use has `Send` bounds. > > I think we can probably resolve this by switching (back) to a `LocalExecutor` on `no_std`, but we will need to be somewhat cairful about it. > > I would prefer to nix these too, if I can. ## ECS Web-Workers Once we have elimited `ConditionalSend` and can produce useful send futures on the web, we can start bringing parallel thread pooling infastrucutre into the web. Any thread-pool that supports web-workers will do, and our existing thread pool can probably be made compatable. The multithreaded system executor should be able to run directly on top of this with no changes. ## Fork/Join ## Parallel Iterators ## Prevent System Priority-Inversion ## Reduce Scheduler Overhead ## System Cache Locality