# PSA: Cache relocation with Julia v1.11

Starting with release v1.11 Julia's precompilation cache files become relocatable.
This change will have no effect on standard Julia usage and packages should continue to work as usual.
Enabling relocation of cache files, together with other improvements coming in v1.11,
paves the way for `Pkg.jl` to serve cache files in the future.

<!--By default, cache files are stored in `~/.julia/compiled/v.X.XX/`.-->
<!--A depot is a folder like `~/.julia` in which Julia stores configuration files,
package code, compilation caches and more. -->

In this note we discuss how a user can prepare, relocate and utilize relocatable caches locally,
and what package authors will have to watch out for to ensure that their code will be relocatable.

# How to relocate precompilation caches: an example

1. Generate a fresh cache file inside a new depot:

julia> # assuming the active environment contains CSV.jl

julia> push!(DEPOT_PATH, "depot")

julia> DEPOT_PATH
3-element Vector{String}:
 "/home/user/depot" # compile cache will be stored here
 ...
 
julia> pkg = Base.identify_package("CSV")
CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b]

julia> Base.compilecache(pkg)
[ Info: Precompiling CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b]
...

julia> Base.isrelocatable(pkg) # verify that the generated cache is relocatable
true

2. Relocate the depot:

shell> cp -r /home/user/depot /relocated/depot/

3. Utilize relocated caches:

julia> # new julia session, assuming the active environment contains CSV.jl

julia> pushfirst!(DEPOT_PATH, "/relocated/depot") # or use JULIA_DEPOT_PATH environment variable

julia> using CSV # no precompilation

Notes:
- The important rule for relocatability is that
both, the cache files and the package code that was used to generate them,
have to be made available through a path in `DEPOT_PATH` before loading,
but they don't necessarily have to be located on the same one.
- The above example relocated the newly created `depot` as a whole.
But depots might contain more data than needed for cache relocation,
e.g. registries, artifacts, scratch spaces, etc.
It would have sufficed to just relocate the caches from
`depot/compiled/v1.XX` to `relocated/depot/compiled/v1.XX`.
<!-- - For a successful validation of the relocated caches a depot containing
the package code that was used for generating the cache must also
appear on `DEPOT_PATH` upon loading.
In the above example, this was achieved by relocating the
whole `depot` which also contained the source code in
`depot/packages`. However, its also possible to split the cache
file and package source code across different depots. -->
- For debugging loading issues use the `JULIA_DEBUG=loading`
flag when starting Julia.

# Notes for package authors

## Changes to package development

- To support relocatability of cache files it was necessary to switch from tracking
the `mtime` of package code to hasing the source files. As a consequence,
updating the time stamp of source files, e.g. using `touch filename` on linux,
no longer triggers recompilation of packages.

## Obstacles for relocation

- Don't use absolute paths in your package code.
Even usage of relative paths that refer outside a
package's root directory should be avoided.
In general you should follow [Pkg.jl's best practices](https://pkgdocs.julialang.org/v1/creating-packages/#Best-Practices)
and expect that packages might get relocated and that their
locations might not be writeable. `Pkg.jl`'s `Artifacts` mechanism
as well as `Scratch.jl` and `Preferences.jl` can help with writing
selfcontained, immutable and relocatable packages.
<!-- - Usage of package global variables that are based on absolute paths
as well as relative paths that reach outside
a package's root directory should be avoided, because these variables
will get baked into the package image during compilation.
To avoid this problem use a constant global `Ref{String}` for each variable and
initialize them once via the `__init__` mechansim. -->
<!-- The usage of absolute paths might be ok if the resulting caches are only meant to be relocated
on the machine on which they were generated. -->
<!-- - Functions that use absolute file paths or relative paths that reach outside
of a package's root directory won't prevent loading of relocated caches,
but the correct functioning of these will likely be restricted after relocation.
Note that `@__DIR__, @__FILE__` also expand to absolute paths. -->
- Use `include_dependency(filename, track_content=true)` (this is the default).
This function is used to declare that `filename` is a dependency
for precompilation; that is, a module that calls this function needs to be recompiled if
the content of `filename` changes.
Using `include_dependency(filename, track_content=false)` is incompatible with relocatability, because it makes the cache depend on `mtime(filename)`.
Note: `track_content=true` causes the contents of `filename` to be hashed
everytime the package is loaded, which can incure slowdowns during verification of very large files.
<!-- - TODO `deps/build.jl` scripts?
I don't know much about that, but probably the same issue with absolute paths? -->