Extensible MyST Exporters

# Extensible MyST Exporters > [!Caution] > This has migrated to the JB team notes: https://hackmd.io/@jupyterbook/Skq85ZqvWl --- > [!Important] > This proposal builds upon "Universal AST" — https://hackmd.io/q0KUwIf_TLCgMyEORaK8kA?both > [!Note] TLDR > 1. MyST exporters are hard-coded (pdf, typst, etc.) — the list of export formats cannot be extended[^patch]. > 2. Static exports are configured by templates that weave together rendered AST (e.g. raw Typst, raw LaTeX) — the rendering step is not configurable by users[^patch]. > 3. By lifting exporters out of `mystmd` and using the same template mechanism as `site.template`, we can simplify `mystmd` and make exporters customisable. This document provides an async scratch-pad to discuss the a "Universal AST" in the MyST Engine. At a high-level, I am effectively proposing that we recognise the status-quo rather than build out anything massively new. [^patch]: Without patching `mystmd`, that is. ## Context The MyST Document Engine supports a range of export targets, including LaTeX and Typst. These exporters are vendored into the `mystmd` binary, and perform most of the export process, making it difficult for end-users to customise the process. For example, it is not possible to change the way that `mystmd` renders an `aside` directive into Typst, or a `figure` in LaTeX. Whilst export templates provide some support for customisation, this is limited to post-processing the rendered markup, rather than programatic access to the document state. Similarly, it is not trivially possible to add _new_ export targets to the MyST Document Engine. For example, there is an `ipynb` exporter being developed as a PR (https://github.com/jupyter-book/mystmd/pull/1882) but this is blocked on maintainer review. It is already fairly functional, yet due to the requirement that it is merged into `mystmd` its value is not being realised (and un-merged PRs atrophy!) Though a long-term view should be to make common features easily configurable, a near-term improvement to the situation would be to permit power-users to customise this behaviour in code whilst the project builds towards better customisation. In both cases, the pathway to resolving this could include giving the MyST Engine _less_ to do, by generalising and separating the export logic. `mystmd` implements support for different exporters in different ways. To build a site, `mystmd` populates a directory with files that a simple static file server can serve (plus a websocket). The theme then consumes this content via the web server. Meanwhile, the static exports are invoked directly within the `mystmd` binary, running AST specialisation before unvoking the typesetting tools. ## Proposal We have had a number of discussions about the difficulty in using consistent language between MyST Web Applications (themes) and the static document exporters. This is a symptom of the fact that these exports do not share a common pathway. The [Universal AST proposal](https://hackmd.io/q0KUwIf_TLCgMyEORaK8kA?both) ultimately argues that we should establish a "lossless" web-first export for MyST that _other_ export types can build atop of. This paves the way for making the MyST Engine _simpler_ and _easier to extend_, by establishing a handover point between the MyST Document Engine and an "exporter". This document proposes the following: 1. (Re)design a common interface for handing off to an external exporter. 2. Tease out the LaTeX and Typst exporters into new `mystmd` packages. This would permit third-party contributors to easily extend and configure static exports, e.g. to import different LaTeX packages or render `aside` in a different way. I'm _not_ necessarily proposing that `_build/site` is exactly the format that we want. It _might_ be that `_build/site/content` + `config.json` is all we need, though. What might this look like? The implications of this are: 1. PDF Templates like https://github.com/myst-templates/lapreprint-typst become _applications_. 2. The Jupyter Book team will lift the `myst-cli` logic for PDF and Typst exports into new `myst-export-typst` and `myst-export-pdf` packages that use the `myst-to-typst` and `myst-to-latex` packages. These exporters will _continue_ to support the basic JTeX templating features. 3. The existing exporter templates can be extended to support definition of an `exporter` field that points to the `myst-export-XXX` package, or the MyST Document Engine will infer the value as it already does. 4. `myst-export-typst` etc. can define extension points for loading custom AST renderers _without_ building a new application. Incidentally, this helps resolve some of the problems concerning language and architectural asymmetry that we have surrounding the "template" language / system. > [!Important] To Do > Explicitly establish the way that this would work. Will it be painful to publish _templates_ (if we use that mechanism) from `mystmd`? ## Related Issues - https://github.com/jupyter-book/mystmd/issues/576 - https://github.com/jupyter-book/mystmd/issues/2299 - https://github.com/jupyter-book/mystmd/issues/1801 - https://github.com/jupyter-book/mystmd/issues/2296 - https://github.com/jupyter-book/mystmd/issues/2162