Subpackages / Explicit Extensions

Reference: GitHub Issue #55516

Problem Statement

In Julia, the unit of compilation is a package (or an extension). Having very large packages with many loosely related features causes several issues:

• It limits the level of parallelism available to the precompilation process because large packages are precompiled sequentially.
• The set of transitive dependencies can grow large (many packages to precompile, many artifacts to download).
• For a package that provides, for example, a large set of solvers, any given user is likely to use only a small subset. As a result, most of the time spent precompiling code is for functionality the user will never actually run.

Currently, the "solution" is to split a large package into many smaller ones (see for example this structure in OrdinaryDiffEq.jl). Users can then explicitly add dependencies to the components of the package (e.g., a specific solver).

However, splitting up a package has some drawbacks:

• The namespacing can become awkward, requiring prefixes like the main package’s name to be added to the split packages.
• Typically, these smaller packages rely on internal components of the "main" package or each other. This can make versioning difficult, and one must be careful with compatibility constraints.
• Often, updates to multiple sub-packages need to happen simultaneously, which is not very convenient with the current tooling.

The proposal here is to allow the creation of "subpackages" that are namespaced under the "main package," with their versions locked to that of the main package. To bump the version of the entire "ecosystem," you would only need to update the version of the main package.

The subpackages would:

• Be individually loadable, for example, with using OrdinaryDiffEq::AdamsBashforthMoulton (syntax subject to change).
• Have their own precompile files, enabling parallel precompilation with other subpackages.
• Have their own set of dependencies.
• Require explicit dependency declarations to be loadable.

Questions:

• Should each subpackage have a unique Project.toml file?
  • Should it contain the exact structure of a normal package without a version field?
• Can you start Julia with a subpackage as the active environment?
• Can a subpackage depend on another subpackage (whether from the same main package or a different package)?
• Can different subpackages of the same main package have different compatibility constraints for shared dependencies?
• Can subpackages have their own Artifacts.toml file?
• Does using MainPackage::SubPackage require that MainPackage is already loaded, or can SubPackage function independently? If it is entirely independent of the main package, should it perhaps be a separate package entirely?
• Relatedly, does SubPackage always depend on MainPackage?
• What is the overlap between this and the workspace feature?
• How should CI typically be setup for a package containing subpackages. You probably want an environment where you can load the subpackages in it.
• How do we inform the resolver that it can only select the same version of the main package and subpackages when they are input as individual packages?