PSA: Cache relocation with Julia v1.11

Starting with release v1.11 Julia's precompilation cache files become relocatable. This change will have no effect on standard Julia usage and packages should continue to work as usual. Enabling relocation of cache files, together with other improvements coming in v1.11, paves the way for Pkg.jl to serve cache files in the future.

In this note we discuss how a user can prepare, relocate and utilize relocatable caches locally, and what package authors will have to watch out for to ensure that their code will be relocatable.

How to relocate precompilation caches: an example

Generate a fresh cache file inside a new depot:

julia> # assuming the active environment contains CSV.jl

julia> push!(DEPOT_PATH, "depot")

julia> DEPOT_PATH
3-element Vector{String}:
 "/home/user/depot" # compile cache will be stored here
 ...
 
julia> pkg = Base.identify_package("CSV")
CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b]

julia> Base.compilecache(pkg)
[ Info: Precompiling CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b]
...

julia> Base.isrelocatable(pkg) # verify that the generated cache is relocatable
true

Relocate the depot:

shell> cp -r /home/user/depot /relocated/depot/

Utilize relocated caches:

julia> # new julia session, assuming the active environment contains CSV.jl

julia> pushfirst!(DEPOT_PATH, "/relocated/depot") # or use JULIA_DEPOT_PATH environment variable

julia> using CSV # no precompilation

Notes:

The important rule for relocatability is that both, the cache files and the package code that was used to generate them, have to be made available through a path in DEPOT_PATH before loading, but they don't necessarily have to be located on the same one.
The above example relocated the newly created depot as a whole. But depots might contain more data than needed for cache relocation, e.g. registries, artifacts, scratch spaces, etc. It would have sufficed to just relocate the caches from depot/compiled/v1.XX to relocated/depot/compiled/v1.XX.

For debugging loading issues use the JULIA_DEBUG=loading flag when starting Julia.

Notes for package authors

Changes to package development

To support relocatability of cache files it was necessary to switch from tracking the mtime of package code to hasing the source files. As a consequence, updating the time stamp of source files, e.g. using touch filename on linux, no longer triggers recompilation of packages.

Obstacles for relocation

Don't use absolute paths in your package code. Even usage of relative paths that refer outside a package's root directory should be avoided. In general you should follow Pkg.jl's best practices and expect that packages might get relocated and that their locations might not be writeable. Pkg.jl's Artifacts mechanism as well as Scratch.jl and Preferences.jl can help with writing selfcontained, immutable and relocatable packages.

Use include_dependency(filename, track_content=true) (this is the default). This function is used to declare that filename is a dependency for precompilation; that is, a module that calls this function needs to be recompiled if the content of filename changes. Using include_dependency(filename, track_content=false) is incompatible with relocatability, because it makes the cache depend on mtime(filename). Note: track_content=true causes the contents of filename to be hashed everytime the package is loaded, which can incure slowdowns during verification of very large files.