owned this note
owned this note
Published
Linked with GitHub
# Julep: Applications
This Julep discusses support for "apps" in the Julia package manager. <span style="color:blue">Text in blue could potentially be left out from a first implementation.</span> In this Julep, an app is defined as a Julia package with a `@main` function, intended to be run by the user as `appname args to app`. It should feel opaque to the user that the app is written in Julia. It's assumed that Julia is installed and serves as the "driver" to start up the app. This type of app thus differs from:
- PackageCompiler apps, which bundle the Julia runtime, libraries, and artifacts to be completely standalone.
- Statically compiled Julia binaries (`juliac`/StaticCompiler) that run without a runtime. <span style="color:blue">It should however be possible to incorporate static apps into the system presented here as later work.</span>
- JLL applications. <span style="color:blue">These are executables that are provided by JLLs. We also want to seamlessly incorporate these into the system so that one can add e.g. ImageMagick from a jll and use it as a normal executable.</span>
## High-Level Notes
- The entry point of an app can be one of the following:
- `Package.main`
- <span style="color:blue">`Package.SubModule.main`</span>
- <span style="color:blue">`app/app.jl` or `app/App/app.jl` with `module app function @main ... end end` in it.</span>
- This mimics how the code for an extension can exist in `ext/Ext.jl` or `ext/Ext/Ext.jl`.
- <span style="color:blue">Maybe `apps/Project.toml` + `apps/Manifest.toml`.</span>
- For adding extra dependencies to app that are only installed once the package is installed as an app. This might require regisrty support (`AppDeps.toml`)
- Each package has a separate environment for its app(s) (`.julia/environments/apps/PackageName`).
- The "shim" that starts Julia and runs the entry point for the app is installed in `.julia/bin`, which needs to be in the user's `PATH`.
## `Project.toml` Additions
For packages exposing an app to users, a new `[app]` section lists the apps of the package. For example, a package (Rot13.jl) providing the app `rot13`:
```toml
# Project.toml
name = "Rot13"
uuid = "..."
version = "..."
[deps]
...
[compat]
...
ArgParse = "1"
[apps]
rot13 = {}
# This means rot13 -> Rot13.@main(ARGS)
```
<span style="color:blue">
Possible other ways of defining an app with a different entry point than `Rot13.main` could under `[app]` be:
```toml
[apps]
[apps.invrot13]
entry = "Rot13.InvRot13" # submodule of package
[apps.rot26]
entry = "rot13.jl" # file in Rot13/apps/
# or entry = "rot13/rot13.jl"
```
</span>
<span style="color:blue">
When the entry point is a file, it must be precompiled in the same way as an extension. Both submodules and files should have a `@main` function.
</span>
## Pkg Additions
A new submodule `App(s)` is added to `Pkg` as well as new REPL commands `pkg> app command`. The section only describes the new "REPL" commands but the "API"-versions are straightforward to derive from it:
- ### `add`
- `pkg> app add Rot13`
- Rot13, a registered package, is downloaded. The project file is examined for an app section.
- An environment is created in `.julia/environments/apps/Rot13`:
- If `bundle_manifest` is `false` resolve with `Rot13` as the only dependency.
- If `bundle_manifest` is `true`, copy the manifest and do an `instantiate`.
- Update the `.julia/environments/apps/AppManifest.toml` (which is a manifest file containing metadata for all the apps). It should be possible to generate all the shims in `.julia/bin` from the `AppManifest.toml` file.
- In this case, the `AppManifest.toml` is similar to a normal manifest but instead of each package having a list of `deps` they have a list of `apps`. It would look something like:
```toml
julia_version = "1.10.0"
manifest_format = "2.0"
project_hash = "622b0771..."
[[deps.Rot13]]
uuid = "..."
git-tree-sha1 = "8eb7b4d..."
[[deps.Rot13.apps]]
name = "rot13"
julia = "dev"
[[deps.lld_jll.apps]]
name = "lld"
#
```
The Julia path could either be an absolute path to the julia executable or alternatively a juliaup julia version specificer.
- For each `app`, a "shim" bash/batch file is created in `.julia/bin` that launches the app. An example shim:
```bash
# julia_app_version = "1.0" (version of the shim generator)
# Check if $julia_executable_path exists; if not, throw a warning
JULIA_LOAD_PATH=$(homedir)/.julia/environments/apps/Rot13
exec $julia_executable_path \\
--startup-file=no \\
-m $(pkgname) \\
"$@"
```
The app is here run in an unstacked environment and the `-m` flag to start the entry point for a package is used (https://github.com/JuliaLang/julia/pull/52103).
- Either automatically put `.julia/bin` in the users `.bashrc` etc or check that `Sys.which` returns the installed app, if not, warn the user about their `PATH`.
- ### `rm`
- `pkg> app rm Rot13`: Remove all shims for the Rot13 package (findable from the `AppManifest.toml` file) and the `.julia/environments/apps/Rot13` folder
- `pkg> app rm rot13`: Remove the `rot13` shim. If all apps for `Rot13` are removed, remove the environment folder.
- ### `status`
- `pkg> app status`: For each package with app, for each app, show some info about the app e.g:
```
Rot13
=====
rot13: {julia = "/path/to/executable/julia", ...}
...
```
- `pkg> app status Rot13`: Show info for all apps of Rot13 and show all the dependencies. Almost equivalent to `pkg> activate "~/.julia/environments/apps/Rot13"; status`
- `pkg> app status rot13` same as above but only show info for `rot13` app.
- ### `update`
- `pkg app update Rot13`, activate `"~/.julia/environments/apps/Rot13"` and run an `update` on it.
- `pkg app update` Run the above on all packages in `AppManifest.toml`
- `pkg app update Rot13 julia` set the `julia` variable (the julia executable) to the current julia path.
- ### `dev`
- `pkg> app dev Rot13`, download Rot13 in the same way as `pkg> dev Rot13` does, set the `path` entry for `Rot13` in `AppManifest.toml` and point the `JULIA_LOAD_PATH` to the package folder in the shim.
- ### `precompile`
- Need to precompile with command line flags "valid"
- `pkg> app precompile Rot13`
## Registry changes
There are strictly no needed registry changes for the proposal here to function. For things like auto-complete (e.g. `app add Fo<TAB>`) it could be good to have a bit in the registry to know if a package has any apps in it so that only those are auto-completed once you are tab-completing inside an `app` command. An error could also quicker be given if one does `app add` on a package with no apps if that information is present in the registry.
## Code loading changes.
With the exception of allowing `apps/app.jl` in a package, there should be no need for any changes to `loading.jl`, everything is built on top of the existing code-loading system.
If `apps/app.jl` is allowed, there needs to be some code (which will look very similar to the package extension code) that maps an app to a UUID etc so it can be precompiled.
## Environment variables
Environment variables controlling app behavior start with `JULIA_APP_` (similar to e.g. `JULIA_PKG_`).
This environment variable takes precedence over the app-specific one.
- `JULIA_APP_ARGS`: Extra arguments that are passed to the Julia executable that runs the app, for example, `JULIA_APP_ARGS="--optimize=3 --threads=2"`
- `JULIA_APP_JULIA`: Path to a julia executable to run the app with.
- It would be good if `JULIA_APP_JULIA="julia +1.9"` worked for nice compat with `juliaup`.
## Open questions
- Exact interaction with `DEPOT_PATH`
- Should Pkg itself install an app (thereby obsoleting [jlpkg](https://github.com/fredrikekre/jlpkg))
- OS compatability on apps? Currently, no OS compat support is available in Pkg / Registry.
- Should a package be able to be a library (currently all Julia packages are libraries) and at the same time be able to provide apps?
- It would be wasteful to require registering two entities to provide the `Rot13` library and the `rot13` app. On the other hand, there is no need for an arg parser dependency in `Rot13`-library but the app probably wants on so if those are forced to share a module the library would probably need to load the arg parser.
- Avoid the bash/batch shim, hook into juliaup?
- This would avoid the duplication of info from AppManifest.toml and the shim. Something like juliaup could just read the manifest file directly.
- `rot13 ARGS` -> `juliaup launchapp rot13 ARGS` -> `launch julia -m Rot13 ARGS`
- <span style="color:blue">Libraries kicking out app precompile files, annoying? Should there exist `.julia/compiled/apps/...`? Or `_app.ji` which is not kicked out by loading.</span>
- <span style="color:blue">`pkg> app add lld_jll`?</span>
- Tab completion for apps `bash-completion`
# TODOs
- [x] Store julia version in AppManifest
- [ ] Call julia +v if juliaup is installed instead of absolute path
- [ ] Precompile, show precompile in status window.
- [ ] REPL completions (complete installed apps / packages)
- [ ] Add to all commands
- [ ] Make an ExampleApp (similar to Example.jl)
- [ ] Env prefix remove when in app mode
- [ ] Show warning if app not in PATH
Sign in with Wallet
