Blog post on Jupyter, Markdown, MyST and ALeA

During a fun and productive visit of Nicolas Thiéry at Michael Kohlhase's Kwarc group at FAU Erlangen in May 2023, and a joint stay at HIM in 2024, we continued our exploration of the following problematic: could lightweight semantic annotations on Jupyter/Markdown based teaching material enrich the experience for authors and learners?

Let's make it concrete on an simple example: if the definitions of concepts in the course material are explicitly annotated with adequate markup:

:::{definition}

The {definiendum}`cat` is a {symref}`domestic species` of small
{symref}`carnivorous mammal`.

:::

then flashcards can be automatically generated to help students memorize the definitions of the course, using spaced repetition. Or of some subset, like the definitions of all concepts that are required to define what a cat is.

This blog post reports on ideas, progress and first prototypes.

Context

[Vision paper](TODO: add link)

TODO: Nicolas: describe the direction in Orsay

One long term line of work of the Kwarc group is to explore how knowledge can enrich computer systems, touching on domains such as semantic AI and semantic web and using tools such as ontologies or knowledge graphs. Our previous collaboration revolved around computation: can we improve the interoperability between computational software by exploiting or adding semantic information into the systems.

Nowadays the Kwarc group is conducting that exploration on education (work funded by the German Ministry of Research and Education (BMBF): VoLL-KI project 2021-2025), as part of the global movement for assisted learning based on learning analytics.

They are building a system, ALeA for Adaptive Learning Assistant, that takes as input course material split in small annotated units (currently authored in LaTeX), and produces an interactive web site for learners to explore that material. The system is continuously experimented with on a corpus of ongoing large courses, including a 1000 student AI course. You can browse the courses – or even create an account to actually play with them – here:

https://courses.voll-ki.fau.de/ (alias: https://alea.education )

Current features

• Support semantic annotations of the course material, using macros and environments provided by the stex LaTeX extension.
• Domain model: a theory of the abstract and didactic concepts for the course, including things like learning objectives or misconceptions (e.g. the usual beginner's confusion between returning and printing).
• Learning records: an event-based track record of the actions of the user (opened that exercise, …)
• Learners model: for each user and each learning objective of the course, this model estimates the competency of the user w.r.t. that learning objective, using a number of stars between 0 and 5 for each of the six levels of competency of Bloom's taxonomy; for example, I remember the definition of a XXX (4 stars), I understand it to some extent (2 stars), I can apply it in very simple cases (1 star), but am not yet able to analyze, evaluate, or create (0 stars).
• Tasks (TODO: is there a difference between task and learning activity):
  • material to read
  • flashcards (with spaced repetition???):
    • front: concept name; back: definition of the concept;
    • could be front: name of a theorem; back: theorem (a theorem being a concept like another);
    • could be: front: theorem; back: proof of the theorem
  • Multiple Choice Questions
  • examples: what to do with them?
  • exercises
    A task can have two components attached to it:
  • knowledge: TODO: just the task content?
  • didactic annotations:
    • prerequisites
    • objectives: know x, apply foo; understand blah
    • answer traits: attributes that can be set on student answers to classify them in answer classes typically: misconceptions
• guided tours: upon a request of a learner to work on concept XXX, and depending on the learner's current abilities, the system builds a personalized guided tour by extracting from the course material a sequence of learning activities (read that definition; do this exercise; use these flashcards) that will help the user learn the prerequisites of the concept and the concept itself.
• dashboards for the student or the cohort to track and visualize their progress in the course.
  • The main instructor dashboard presents a narrative template with figures filled in, estimating how the cohort is doing (like: 10% are doing very well; …). And visualizations of the progress of the cohort, like 3D graphs of competency(concept, student, time).
  • Quizz dashboards display mumber of exercises attempted, response rate, scores
  • The student dashboard presents a narrative template with figures filled in, what topics they are solid at, how far they are from passing the exam, etc. The student also has access to the details of his learner model, and may overwrite the system estimation of each competency level with their own self-estimation. Such actions are tracked, so it's possible to get both the system and self-assesment for different use cases (recommendation, formative or summative evaluation).
• review widgets, typically meant to appear at the end of a course section (chapter, lesson, …), that show the current learner model for the competencies of the section and recommends tasks; currently all tasks whose objectives match with the current section (and prerequisites are fullfilled?).
  Distinction between Drill vs Revise to be clarified.
• trafic lights: similar to review widgets, these are meant to appear at the beginning of a course section to provide feedback on the student has all the prerequisites, and suggest remedies.
• train widgets: similar to review widgets, but this time it's the user that chooses a bunch of concepts and a target mastery level for each
• scanned exam grading platform (similar to CorrectExam), including grading by answer traits (comments in CorrectExam's jargon). Nice little feature of CorrectExam: possibility to mark excellent and notably bad answers (In order to generate sample answer sheets for the students and discuss the wrong answers with the class).
• course, session, class management. This is mostly administrative, up to the ability for the instructors to specify the current advancement of the course: as of now, items X, Y, Z have been covered and should be learned.
• potential for gamification ? in particular generic collaborative games where groups of student drill together.

How does this work

TODO Dennis / Jonas

• Adjust the short description below to make it correct
• Explain briefly what the domain model is
• Say a word about what piece of software/infrastructure are involved in the learner's model service.
• Screenshots?

The course material is written as a collection of (small) annotated LaTeX files; each of these files typically contain a few (beamer) slides. We will from now on refer to them as the stex files (s for semantically annotated).

A special implementation of TeX, called RusTeX takes as input the stex files and generates as many html files, where the annotations have been encoded (which we will refer to as the sHTML files). RusTeX also has the property of producing sHTML which is visually identical to the pdf that would be produced by the usual toolchain (pdflatex).

sHTML plays the role of an AST (abstract syntax tree) from which to pivot. There are also sHTML importers from word, powerpoint, and markdown.

The sHTML files are imported into the ALeA system. They are processed by MMT to extract the semantic and reason on them: resolve cross-references and references to the domain model. The outcome (processed html files + ???) are then uploaded on the course web server.

When the learner logs into the service, a javascript library running in the learner's browser combines all the bits to pilot the user navigation: displaying slides, navigating among them, querying and updating the learner's model service, displaying flash cards, resolving guided tours, etc.

Currently, the values in the learner models are essentially estimated from:

• Remember: flash-cards + self assessment
• Understand: examples + self assessment?
• Apply: assessement by solving problems;
  • autograded multiple choice questions
  • manually graded exam exercises

Caveat: these self-assessments tend to be heavy-handed: they use up energy and meta-cognition; they also require a good understanding of the learning process. In somes courses, the students are first introduced to this chunk of the BLOOM terminology.

Aims and design principles

VoLL-KI aims at disseminating widely its outcomes, fostering best practices (semantic annotations of course material; pedagogical and ethical exploitation thereof), and reuse of the tools they are developing.

To this end, the following design principles aim at reducing the entry barriers.

Annotating one's course material should be:

• interoperable: the instructor can keep working with his usual authoring toolchain (LaTeX, Jupyter, powerpoint, …), and use standard extension points of these tools. Annotations should degrade properly and be preserved when these extensions are not available for a given tool.
• lightweight and unobstrusive: by default, the annotations should be invisible to readers. In the sources, annotations should be either invisible by default (WYSIWYG/WYSIWYM edition), or minimally obtrusive (text-based edition).
• gradual: annotations can be added gradually, starting from no annotation at all, letting the instructor strike the balance between added work and added benefits, according to the timeline, investment capacity, and needs for the course at hand.
• immediately useful: as an incentive for instructors, each new annotation should bring an immediate benefit in addition to long term benefits. This to fit with the typical continuous improvement workflow of instructors; e.g. casually adding a few annotations on the fly while reviewing one's notes for the next day class. Typically, the annotations should bring consistent and nice visual rendering to advantageously replace manually crafted visual markup (separation of form and content).

About MyST

MyST (markdownly Structured Text) is both an extension of Markdown and a (mostly javascript based) ecosystem of tools to support scientific authoring. Of particular interest for this project are:

• Syntax for defining roles (=macros in latex) and directives (=environment in latex)
• Extensive support for cross-referencing and content reuse, including across documents and even collections of documents (cross-site).
• Production of highly structured web sites enabling easy harvesting of metadata and (snippets of) content. E.g. the ast of the documents.
• Strong emphasis on modularity, standards, accessibility, …
• An active, dedicated and highly competent community organised as the Executable Book organisation.

Due to these qualities, MyST is getting traction, e.g. in AGU's project Notebook Now for a Jupyter+MyST scientific publishing plateform, or overleaf's analog for Markdown curvenote. And for teaching!

• With jupyterlab-myst, one can use MyST syntax in Jupyter notebooks (current limitation: single-document features only and no custom roles/directives).
• With jupytext, one can write notebooks entirely as markdown file (with yaml metadata header). This has many good properties for teaching purposes (ease of writing, human readability, version control, …).
• JupyterBook is progressively being phased out by the MyST toolkit

The brainstorms and sprints

Our main target for the brainstorms and sprints was to explore collaboration opportunities between Alea and the Jupyter / Markdown ecosystem:

• bringing Alea features to instructors (like me!) that use Jupyter/Markdown as authoring tools
• sharing standards
• sharing components
• sharing infrastructure

Bottom-up collaboration opportunities

• Sit down and specify how to make the semantic harvestable from a web (annotated web site, e.g. with sHTML) site; possibly using MyST support for harvestable metadata / AST
  TODO: link to section below; link to sHTML syntax
• Sit down and standardize Markdown syntax for in-document semantic annotations (Denis, Michael, Nicolas) good enough for now; may need refinement as we lea
• Sit down and write down a schema for semantic annotation of tasks / learning object (Michael, Nicolas): good enough for now;
• Sit down and write down a schema for learning records / interaction logs (Jonas, Michael, Abhishek, devs of PLaTon? xAPI experts?)
  • Nicolas: add a link to an example of jupylate's learning record
  • Jonas: same for ALeA
• Compare and normalize grading widgets (with answer classes / traits) (Jonas, Abhishek, Olivier (CorrectExam)):
  • Level 1: standardize data
  • Level 2: a shared reusable javascript component
  • TODO: brief CorrectExam demo
  • TODO: brief feature comparison between the two
• Compare and normalize concept descriptions ???
• Standardize abstract syntax trees? Presumably harder but highly valuable? At least to make sHTML / MyST semantically consistent and easy to convert?
  • TODO: add links to MyST documentation
• TODO: Nicolas/Chiara: Jupylate as reusable component for notebook-based autograded randomized exercises

Top-down collaboration opportunities

• Alea as a service
  • by importing sHTML courses; (TODO add link to section below)
  • as an overlay on an existing site? (à la …)
• ALeA as WebComponents and MicroServices

Target: Standardizing annotations accross authoring syntax

Brief reminder of MyST's syntax for roles and directives

One argument macro:  {foo}`bla`         \foo{bla}
Two arguments macro: {foo}`bla <truc>`  \foo[truc]{bla}

Simple directives:

:::{foo} xxx sdfa asdf 
Lorem Ipsum ...
:::

Directives with simple metadata:

:::{foo} xxx sdfa asdf
:key1: value1
:key2: value2

Lorem Ipsum ...
:::

Directives with YAML metadata header:

:::{foo} xxx sdfa asdf
---
key1: value1
key2: value2
---
Lorem Ipsum ...
:::

Annotations in sTeX

Discussion about markdown syntax for stex: https://github.com/slatex/sTeX-React/issues/281

some sTeX documentation for learning objects

Annotations in the sTeX syntax:

\symname{A} : \symref{A}{A}
+ shorthands for plural, ... 

\definiendum{symname}{blah blah blah}
\symref[pre=prefix,post=postfix]{symname}{bla bla bla}
\sn{bla}   shorthand for \symref{bla}{bla} 
\sn{pre=un,post=ed}{color} -> uncolored
\sr=\symref + syntactic sugar for case/plural 

Environments:
- sdefinition
- sassertion (with styles theorem, lemma, remark, ...)
- sproblem (with styles exercise, ...) (with subenvironments subproblem, solution, ...)
- sproof (with subenvironments sproofitem, ...)
- sparagraph (with title and styles)
- sfragment (a sectionning)
- smodule (grouping of symbol / namespace)

Example:
\begin{sdefinition}

\end{sdefinition}

imports module / use module

- inputref: presumably the analogue of %embed in MyST

Inline versions of these environments: \inlinedef{...} (which generate spans instead of divs in sHTML).

Didactict annotations for tasks are inserted in the task block:

\objective{remember}{concept}
\objective{understand}{concept}
\objective{apply}{concept}

Being able to use a single \objective{understand,apply} could be nicer. Prerequisites are deduced implicitly from the symrefs.

Open questions:

• For prerequisites, there is no distinction on remember/understand/apply. There could be a need to be more precise.
• The frontier between prerequisites and objectives is blury; some prerequisites can be soft.
• Would we want to quantify? E.g. this activity requires a light/strong proficiency with for loops?
• How to annotate the difficulty of an exercise? What would be the semantic of it? Relative to the given objectives? Absolute?
• Annotations to target a particular cohort? E.g. biologist? Just a (soft) prerequisite?

Tentative MyST syntax for annotations

{definiendum}`foo`
{definiendum}`blah blah blah <label>`
{symref}`foo`, {symref}`foo <label>`

:::{prf:definition} Lorem
:label: lorem                  alternative: (lorem)= on the line before the definition

A {definiendum}`Lorem <lorem>` ipsum is ...

:::

:::{admonition} Exercise    (in waiting for :::{exercise})
:::

The label could be presumably be made implicit from the definiendums. Particularly handy when there are several definiendums.

Didactict annotations for activities:

:::{exercise}
---
prerequisites:
  - loop: apply
objectives:
  - accumulator: remember
---
Lorem ipsum
:::

Comments:

• This YaML blurb could be in the front matter of a file as well.
• Should the annotations be grouped in a "semantic" subdict?
• In MyST, to be used in cross-references, labels must be a-zA-Z -_ strings.
• It's not critical to set in stone the exact syntax at this stage; what matters is to standardize on the sHTML export syntax. So we can start with the above and extend along the way by defining appropriate roles/directives.
• Nicolas used the above definition/definiendum syntax to annotage most definitions in his course Info111 (caveat: a few use just definiendum, without a definition environment around)
• No need at this stage for finegrained annotations in MyST; e.g. symbols within math formulae. If needs come, we can just use stex syntax in the latex formulae.

Milestone: ALeA as a service

Proof of concept: importing Nicolas's Info111 courses in ALeA

Dennis and Nicolas had a sprint in 2023, with sHTML export with JupyterBook and roles / directives implemented in Python. This "worked".

The web page still has the sHTML annotations. See for example this page (login: Enseignant, password: Enseignant).

In the meantime Nicolas has annotated all definitions (with definition/definiendum, and sometimes symrefs) in the course (so flashcards should work) with many symbols being partially aligned (at the occasion of a sprint with Michael in July 2024) in a central file.

Sources for the course; Annotation example

• Instructions for Jupyter-Book:

  1. Download this Python plugin for JupyterBook into an extensions subdirectory of your jupyter book directory.
  2. Insert the following in your JupyterBook configuration file _config.yml:
     ​​​​​​​​sphinx:
     ​​​​​​​​  extra_extensions:
     ​​​​​​​​    - sphinx_proof
     ​​​​​​​​  local_extensions:
     ​​​​​​​​    semantic: 'extensions/'
     ​​​​​​​​  
     ​​​​​​​​semantic:
     ​​​​​​​​  # This can be any URI identifying your course; need not be an actual URL
     ​​​​​​​​  namespace: https://Nicolas.Thiery.name/Enseignement/Info111
     ​​​​​​​​  # This is were you can insert alignments for the terminology of your course
     ​​​​​​​​  symdecls:
     ​​​​​​​​    - verbalization: informatique
     ​​​​​​​​      en: computer science
     ​​​​​​​​      wikipedia: ...
     ​​​​​​​​      mathhub: '[smglom/cs]mod?computer-science?CS'
     

     See also this example configuration for Info111.
  3. Install sphinx-proof if not already there, e.g. with:
     ​​​​​​pip install sphinx-proof
     

  4. Insert definiendum's in your book sources
  5. Rebuild your book with jupyter-book

• Javascript Plugin for MySTmd and Mystmd configuration file

Other courses

These three courses share the exact same technological stack as Info111

• IPPJ: Introduction to programming, with Python and Jupyter (repository)
  • Could be of use at FAU for Michael's digital humanities course
    Hence a test bed for how to make multilingual courses (here: fr/de/en)
  • Is being built as a test bed for technology and best practices (for reusability, usage in ALeA style infrastructure, …)
• ISD: Initiation to Data Science (repository)
  Course led by another instructor (with Nicolas's help); hence a test bed for aided adoption by others.
• MethNum: Numerical methods (private repositories) A 3 year course run by and for physicists; hence a test bed for independent adoption by other instructors and usability by a different crowd of students.

Other tasks

• Nicolas: add more annotations, alignments :-)
• Dennis: check if the current sHtml can still be imported in ALeA
• Kwarc: fork and play; note: you can now create accounts on our GitLab. Compilation: install this conda environment and make web. Output will be in _build/html. Alternatively, your continuous integration on GitLab may just work.
• Nicolas: reimplement the roles/directives for the MyST CLI
  Partially done; current blocker: MyST seems to be doing some sanitizing on the produced html, and the <span> arguments are striped.
• Nicolas: use the MyST CLI for building the web site (depends on a few remaining MyST tickets)
• Try it with actual students! Could be for the next Info111 session, September-December 2024
• Explore what MyST could possibly bring to help fetch metadata and content

Milestone: ALeA as WebCompontents and MicroServices

Deliverable: ALeA integration in Jupyter

Note: some misc data currently stored in the ALeA application that do not appear in the current diagram:

• intermediate grades, … before publication to the school information system
• course management (student, course, session information + session progress)

Tasks and dependencies:

• Release sTeX-react
• …

Other potentially related use cases

Mathswitch / mathdatahub

• knowls / hover cards: your website has texts involving certain terms; you would like to have knowls for these terms, where the content would either be provided by metadata of the website, or taken out of some other knowledge database (like wikipedia, slglom, …)
  Looks similar to intersphinx crossreferences, with knowls on top. Would that be a myst feature? Or be implemented by hijiking a chunk of the mystjs library? There seems to be some wikimedia api for this; so it might just be a question of following that api, both as producer and consumer.

Cost network

Potential partners:

• Université Paris-Saclay