# Zettel

An open-source AI-powered modular app builder.

## What is Zettel?

Zettel is a platform to build your customized app in seconds by combining well-tested inhouse or community developed feature-level modules.
It aims to boost custom app development for developers.
The core product has all the basics already implemented, including authentication, access management, storage, mobile-first UI, Offline-first synchronization, search content, etc.
Also there are tons of atomic features available as modules in the store to put on top of that.
There is a powerful AI assistant that helps you find and glue the right set of modules to build your app.
You as a developer will only need to focus on what really matters.

## How does Zettel work?

Zettel models the state of an application as an _array of objects_.
The combination of all the associated modules together determines what those _objects_ are.
A Zettel app is nothing but a selection of modules working together on the base platform.

### Modules

Modules are features that together form the app.
They can extend both the data structure and the UI to serve their aimed feature.

They are hot-pluggable, it means one can add, remove, or upgrade any module anytime, even in the middle of runtime.
Modules are designed in a way that they already know how to communicate to each other, therefore there is no extra effort needed to configure the system; you just pick the right set of modules and they can start working together right away.

Every module represents a feature, like: `text`, `files`, `link-preview`, `comment`, `reaction`, `task`, import/export modules, `like`, `file-preview`, `summarizer`, `link-redirect`, etc.
For example, one can configure a simple Todo list manager just by combining the `text` and `task` modules.

Every module is a full-stack application on its own.
They have a frontend implementation which is served by the Module store (explained below) which takes care of extending the Zettel frontend app, and optionally a backend implementation which serves the client-side running module and can connect to the Zettel backend via the Zettel API.

### Data model

The data model of every Zettel app is an _array of objects_ that we call it a _page of cards_.
Each card is one _unit_ of data.
Every module stores its own representation of the data units on these _cards_.
As a result, each _card_ data practically is a dictionary of synchronized information stored by module name.

Here is a sample _card_ data in the above Todo list manager app:

{
    "text": {
        "data": "Buy milk"
    },
    "task": {
        "status": "NOT_DONE",
        "dueTo": ...
    }
}

Sometimes, the stored data by each module are orthogonal and has nothing in common, like the above example.
In these cases, modules just store and use their data from _cards_ independent from other modules.

However, some other times, they overlap.
For example, if we add `link-preview` to the Todo list manager app above, its data will have some common information with the `text` module:

{
    "text": {
        "data": "Study this article: https://en.wikipedia.org/wiki/Biology and prepare a short report"
    },
    "task": {
        "status": "DONE",
        "dueTo": ...,
        "by": "Alice"
    },
    "link-preview": {
        "linkUrls": [
            "https://en.wikipedia.org/wiki/Biology"
        ]
    }
}

As you can see, the data that `link-preview` stores is actually the extracted URLs out of the data that `text` stores.
Therefore, these two modules on this Zettel app can not work independently, their operations that change the data will affect each other.

The key point here is to making sure these overriding modules data on each _card_ are kept compatible and synchronized.
In other words, if the user changes the `text` data, other related modules like `link-preview` also follow the change and update their own data on the card accordingly.
Only then, the result is always like those modules are fused together as one app.

Zettel takes care of this automatically in the background by leveraging developer-defined or AI-generated extractor functions.
Because of that, in order to configure a Zettel app we just need to select the sub-set of modules that we need, and they can work together right away and effortlessly.

### UI and Extenison API

As mentioned before, a Zettel app data model is an array of units or a _page_ of _cards_, therefore the overall structure of the UI pretty much follows the same model; in the simplest form, you can imagine the end-users scroll and work with a list of card-like items.

However, without any modules added to a Zettel app, there is nothing much to do there.
Modules can extend the UI in all sorts of ways; some modules append/prepend UI components to the _page_, or manipulate the _cards_ layout, while others extend the _card_ or the _card_ edit box.
Modules have read/write access to the data available on the Zettel app, they can display dialogs, show messages, create/modify _cards_, and have custom implemented HTML content to extend all these.

Zettel is very flexible on letting modules to take control of many aspects of the UI, which makes it convenient for module developers to implement the user experience of their module as they see fit.
All of these are available to the modules via the Extension API, which is a TypeScript/JavaScript library that developers use in the client-side implementation of their modules to make access to the UI and its data.

For example, the `text` module, displays the text data on the _cards_ and also appends a text input to the card editor to allow users to add new text _cards_.
On the other hand, the `task` module, appends a checkbox to the _cards_, therfore makes them _items that can be marked as done_.
The `link-preview` module also appends the Open Graph preview of the _card's_ URLs to each card, it uses its backend service to extract this data for each _card_ with URLs.

As you can see in the following sketch, they all form how the Zettel app performs and looks like:


### Architecture

Users can configure multiple Zettel apps, each called a _page_.
Each _page_ can have multiple modules enabled on it, as well as multiple members, who are other users that access the same _page_.
The app data is stored on the _cards_ inside each _page_.
All the modules work on an app in a flat structure and all have the same level of control over the UI and data.
So, it sounds like a list of _cards_ being used and manipulated by a group of peers consisting of members and modules.


A module can not directly access what other modules have stored on a _card_, so modules each have their private storage space on an app.

#### Inter-module communication

Just having multiple modules that all work independently doesn't make you an app!
A very important aspect of having a configuration of modules is to let them connect and communicate.

There are three ways for inter-module communication:

- **Data on _cards_**: Even though modules can not directly access others data on each _card_, they can still kind of communicate via the _card_ data since Zettel automatically keeps the modules data synchronized. In the above Todo list manager app, when the `text` module changes its text in a way that its link URL changes, Zettel reflects it to the `link-preview` module data, and then `link-preview` behaves accordingly.

- **Direct API call**: Modules can provide API end-points for other modules to use, or can use other modules provided APIs. This is the most direct client-side communication way for modules which enables them to build graph-like structures as a system. Zettel takes care of building the right connections between the modules using developer-defined or AI-generated connector functions, therefore again there is nothing here the users need to worry about while configuring their apps.

- **Indirect server-side calls**: Since each module is a full-stack app on its own, they're free to have any kinds of communication on their server-side implementations as the developers see fit. However, this method of module communication has to be implemented by the developers and is not something Zettel can help with.

### Store

All the modules are published to the Zettel's module store to be available to anyone who is looking for it.
While developing your own module, you can have a developing version of the module only available to you and the testers, as well as a published version available to your team or anyone at the same time.

Zettel takes care of ensuring your apps always run the latest published versions of their modules, so as soon as the developers have their new versions of modules published, any of their users will automatically upgrade to use it.

Developers will need a developer account in order to access their Developer Console, so they can publish and manage their modules.
They can also automate the module publishing process by using API keys, which will be very helpful on the development process.

## Limitations

As mentioned earlier, Zettel makes custom application development very convenient by providing a powerful base platform that already takes care of many basic needs of an app, like authentication, etc., and supporting the developers with so many official/community developed feature-level modules can be easily and effortlessly added to your app configuration.
So, all a developer needs to do is to find the right set of modules, and at most develop the very custom part of their app as a module, which can be reused later on other apps as well.

This comes with a limitation:
An application data should be representable as an _array of items_, and it's the one and only condition an application should satisfy to be supported by Zettel.

Currently, this condition is the only thing that limits the overall range of the applications that Zettel can support.
However, we intend to add support for more data models, like multiple arrays of items, graphs, etc. to cover wider ranges of the user-facing application ideas in the future.

--------
NEW DOCUMENT BELOW
-------

# Concepts

In the following sections, we walk through developing a simple module named `task`.
The goal is to become familiar with the main concepts of modules, data model, and Zettel's development process.

## Prerequisites

Let's first make sure we meed the following requirements:

### Access to Zettel Developer Console

Developers first need to make access to the [Zettel Developer Console](https://app.zettel.ooo/developer).
It requires the developers to get their _developer account_ ID and password, currently by the help of the support team.

In the Developer Console:
- We can publish and manage our modules for test, development, and production purposes,
- We can also create **access keys**:
- _Developer access keys_ to help automate publishing new versions of modules to the store, very helpful in the development process,
- _Extension access keys_ to allow authenticated modules to access the Zettel API at runtime.

### Tech stack

We use JavaScript/TypeScript and the web development stack for client-side implementation, no other programming language is supported at the moment.

You may use any stack for server-side implementation, however, the official SDK is only available as NPM packages for now. Therefore, it's recommended to use JavaScript/TypeScript for backend development as well.

We strongly suggest to use TypeScript over JavaScript, since all the packages are carefully and strongly typed.

### Server to serve module's backend

Not all the modules need a dedicated server to serve them, but if a module requires a server-side implementation, it's up to the developers to take care of serving it publicly in a way that all the running instances of the module's client-side on the user's devices can access it.

## Module definition

The `task` module is all about transforming every _card_ in a _page_ to a **task** item that can be marked as "done" or "checked".
It doesn't care much what the other modules are and thus what the _cards_ actually are in the _page_, it just converts those _cards_ to todo items.

For example along with the `text` module, it can form a simple todo list application, or along with the `file` module, it can serve as a selectable document source for LLM calls like Q&A on documents.
It'll be up to the users how and for what purpose to use it, it's just responsible for making _cards_ "markable".

A completed version of the `task` module is currently one of the Zettel's official modules.
It's publicly accessible in the platform and also it's open sourced, so you may access the source code from [here](https://github.com/zettelooo/zx-task).

### Data model

The module definition determines what it needs to store on _cards_.
The first thing we're going to need is a flag that indicates whether the _card_ is checked or not.
We can also store the timestamp of being checked and the user information who checked this _card_.

Here is the schema definition for our module's data model:

interface TaskCardData {
  checked: boolean
  checkedAt?: Date
  checkedBy?: {
    name: string
    userName: string
  }
}

### UI extension

Following the data model, we can imagine the module to extend _cards_ UI by a checkbox and probably the timestamp and the user information:


A similar additional UI is also needed on the _card composer_ at the bottom of the _page_.

> Note: In many situations, the _card composer_ ressembles the _cards_, so we need to extend them both.

## Seed project

Now that we know what we want to develop, let's do some code!

The recommended way to start developing a module, is to start by the [Zettel Extension Seed](https://github.com/zettelooo/zettel-extension-seed) GitHub template repository:


Simply give it a name and create your own module repository based on the seed project, then clone and open it.

> Note: We recommend using VS Code as the IDE, however it's not mandatory.

### File structure

The repository initially is a mono-repo with three projects in it:
- **client**: the client-side implementation which is bundled and published to the Zettel Store to be executed on the users' devices,
- **server**: the optional server-side implementation which is served separately and is responsible to support the client-side code to operate,
- **shared**: all the common type definitions and implementations shared by both **client** and **server** projects.

## Develop our module

### Manifest file

We start by the `manifest.jsonc` file in the `client/public` folder.
This folder holds all the assets that are packed into the bundle before being published to the store.
We modify it similar to the following:

{
  "appId": "task",
  "version": "1.0.0",
  "name": "Task",
  "description": "Task module to make cards tasks with checkboxes",
  "aiDescription": "Display checkbox on every card. Check off tasks as you complete them. Make cards selectable.",
  "avatar": { "file": "checked.png" }
}

The `appId` together with the developer ID form the unique module ID in the Store.
Also, the `"checked.png"` is the avatar file path that we just copied into the `client/public` folder.

### Card data model

We implement our modules data model definition into the **shared** project, so we can refer to it from other projects.
Here we identify the type of data our module is expected to store on each _page_ and _card_:

// shared/src/Data.ts:

import { ZettelTypes } from '@zettelooo/api-types'

export type Data = ZettelTypes.Data.Builder<{
  page: {
    disabled?: boolean
  }
  card: {
    checked: boolean
    checkedAt?: string
    checkedBy?: {
      name: string
      userName: string
    }
  }
}>

Then, we attend to the file `client/src/cardData.ts`.
In this file, we provide an overall description on the data model for the AI assistant's later use.
We also define the data _constructor_ as well as all the important data extractors for our module.
Zettel will later use all of them to provide the effortless communication between the modules of a configured Zettel app:

import { ZettelExtensions } from '@zettelooo/extension-api'
import { Data } from 'shared'

ZettelExtensions.setCardData<Data>({
  // Used by the AI assistant to generate all the missing
  // data extractors between this module and the other modules
  // that are already on the page:
  description: 'This data is a raw text representation of the card. If exists, it contains a string `text` property that contains the raw text value for the card.',

  // Used to initialize/update the card data for this module
  // based on the extractions out of other modules data
  // when it's newly added to the page (with no previous data)
  // or when other modules data is updated (with previous data):
  construct(previousData, extractions) {
    const checked = extractions.reduce(
      (checked, extraction) => checked ?? extraction?.checked,
      previousData?.checked
    )
    if (checked === undefined) return undefined
    if (!checked) return { checked }
    const checkedAt = extractions.reduce(
      (checkedAt, extraction) => checkedAt ?? extraction?.checkedAt,
      previousData?.checkedAt
    )
    const checkedBy = extractions.reduce(
      (checkedBy, extraction) => ({
        name: checkedBy?.name ?? extraction?.checkedBy?.name,
        userName: checkedBy?.userName ?? extraction?.checkedBy?.userName,
      }),
      previousData?.checkedBy>
    )
    return {
      checked,
      checkedAt,
      checkedBy,
    }
  }
})

## ...

...

### Offline-first vs online-first approaches

...