# Extensible MyST Exporters

> [!Caution]
> This has migrated to the JB team notes: https://hackmd.io/@jupyterbook/Skq85ZqvWl

---

> [!Important]
> This proposal builds upon "Universal AST" — https://hackmd.io/q0KUwIf_TLCgMyEORaK8kA?both

> [!Note] TLDR
> 1. MyST exporters are hard-coded (pdf, typst, etc.) — the list of export formats cannot be extended[^patch].
> 2. Static exports are configured by templates that weave together rendered AST (e.g. raw Typst, raw LaTeX) — the rendering step is not configurable by users[^patch].
> 3. By lifting exporters out of `mystmd` and using the same template mechanism as `site.template`, we can simplify `mystmd` and make exporters customisable.

This document provides an async scratch-pad to discuss the a "Universal AST" in the MyST Engine. At a high-level, I am effectively proposing that we recognise the status-quo rather than build out anything massively new.

[^patch]: Without patching `mystmd`, that is.

## Context

The MyST Document Engine supports a range of export targets, including LaTeX and Typst. These exporters are vendored into the `mystmd` binary, and perform most of the export process, making it difficult for end-users to customise the process. For example, it is not possible to change the way that `mystmd` renders an `aside` directive into Typst, or a `figure` in LaTeX. Whilst export templates provide some support for customisation, this is limited to post-processing the rendered markup, rather than programatic access to the document state.

Similarly, it is not trivially possible to add _new_ export targets to the MyST Document Engine. For example, there is an `ipynb` exporter being developed as a PR (https://github.com/jupyter-book/mystmd/pull/1882) but this is blocked on maintainer review. It is already fairly functional, yet due to the requirement that it is merged into `mystmd` its value is not being realised (and un-merged PRs atrophy!)

Though a long-term view should be to make common features easily configurable, a near-term improvement to the situation would be to permit power-users to customise this behaviour in code whilst the project builds towards better customisation. In both cases, the pathway to resolving this could include giving the MyST Engine _less_ to do, by generalising and separating the export logic.

`mystmd` implements support for different exporters in different ways. To build a site, `mystmd` populates a directory with files that a simple static file server can serve (plus a websocket). The theme then consumes this content via the web server. Meanwhile, the static exports are invoked directly within the `mystmd` binary, running AST specialisation before unvoking the typesetting tools.

## Proposal

We have had a number of discussions about the difficulty in using consistent language between MyST Web Applications (themes) and the static document exporters. This is a symptom of the fact that these exports do not share a common pathway.

The [Universal AST proposal](https://hackmd.io/q0KUwIf_TLCgMyEORaK8kA?both) ultimately argues that we should establish a "lossless" web-first export for MyST that _other_ export types can build atop of. This paves the way for making the MyST Engine _simpler_ and _easier to extend_, by establishing a handover point between the MyST Document Engine and an "exporter".

This document proposes the following:
1. (Re)design a common interface for handing off to an external exporter.
2. Tease out the LaTeX and Typst exporters into new `mystmd` packages.

This would permit third-party contributors to easily extend and configure static exports, e.g. to import different LaTeX packages or render `aside` in a different way.

I'm _not_ necessarily proposing that `_build/site` is exactly the format that we want. It _might_ be that `_build/site/content` + `config.json` is all we need, though. What might this look like?

The implications of this are:
1. PDF Templates like https://github.com/myst-templates/lapreprint-typst become _applications_.
2. The Jupyter Book team will lift the `myst-cli` logic for PDF and Typst exports into new `myst-export-typst` and `myst-export-pdf` packages that use the `myst-to-typst` and `myst-to-latex` packages. These exporters will _continue_ to support the basic JTeX templating features.
3. The existing exporter templates can be extended to support definition of an `exporter` field that points to the `myst-export-XXX` package, or the MyST Document Engine will infer the value as it already does.
4. `myst-export-typst` etc. can define extension points for loading custom AST renderers _without_ building a new application.

Incidentally, this helps resolve some of the problems concerning language and architectural asymmetry that we have surrounding the "template" language / system.

> [!Important] To Do
> Explicitly establish the way that this would work. Will it be painful to publish _templates_ (if we use that mechanism) from `mystmd`?

## Related Issues
- https://github.com/jupyter-book/mystmd/issues/576
- https://github.com/jupyter-book/mystmd/issues/2299
- https://github.com/jupyter-book/mystmd/issues/1801
- https://github.com/jupyter-book/mystmd/issues/2296
- https://github.com/jupyter-book/mystmd/issues/2162