OctoFarm backend & API

Glossary

• API Application Programmable Interface, the interface to communicate with in a predetermined contractual manner.
• OF OctoFarm
• OP OctoPrint
• DTO Data transfer object, an API contract which needs verification and transformation on it's data in order to become data ready for a database CRUD transaction.
• Entity database-first model which can reside in memory.
• CRUD Create/Read/Update/Delete, primitive actions for any persisted entity.
• ORM Object Relational Mapper, a database abstraction layer with Transactions, Unit-Of-Work and repositories.
• TypeScript Javascript-like typing and compilation language supporting ECMAscript - made by Microsoft
• VueJS frontend framework like f.e. Angular, ReactJS, Blazor(WASM) or Svelte.

Architecture

There's quite a lot we can choose whilst developing an architecture, but existing architecture require connecting a lot of dots in order to meet regressive functionality. OctoFarm is quite dependent on external dependencies:

• Users setting MongoDB up properly, but also nuking that data (as we didnt migrate collections before)
• Users having a proper network (static IP's or working uPnP/FQDN's)
• OctoPrint being correctly set-up (API key, CORS, user groups)
• Deployment working (NodeJS or Docker environment)

Therefore we've chosen to migrate from a ExpressJS API server for OctoFarm to something more abstract with the main goal of promoting self-healing behaviour. We'd need a layered architecture for that in order to keep the code overhead manageable.

We aim to use NestJS mainly because it has modularization patterns very much like Angular. The authors also state that they took the patterns from that frontend framework to begin with, and it feels very natural to me because of it.
My goal here is to find what I know to be required middleware for an API and server to get started:

• Authentication (cookies & JWT/access-token)
• Authorization (role-based or claim-based)
• MVC
• ExpressJS packages
• Mapping
• Validation
• Class decorators for auto-validation
• Exception transformation/filtering
• Central logging and sinks/filtering
• Distributed/Local events
• Scheduled work/jobs
• WebSocket, HTTP
• SMTP mail
• Push notifications

NestJS docs
https://docs.nestjs.com/
NestJS reference packages/tools
https://github.com/juliandavidmr/awesome-nestjs

TypeORM

In comes TypeORM, an abstraction over Mongoose. Why? Well with class-validators (a NPM library) and typescript support, database entities have a major advantage over schema's: we abstract ourselves away from the database. We do lose some features like table-capping and mongoose out-of-the-box validators. We gain a lot of validation options as well with much more control and extensibility, so it's rather a pro than a con in that perspective.

No database capping

Database capping is a feature available to Mongoose, but not on TypeORM level as it is not a common rel-db SQL feature. It can easily be added on a service/repository level, but it's advised to look at TypeORM migrations or QueryRunners to do this cleanup job for us. Explicit control.

No underwater commit validation

We must call validate(entity) at our leisure. This can be abstracted by creating and extending a base class Entity. This class then attaches the @BeforeInsert() and/or @BeforeUpdate() to one or more methods. There are also a lot more post-transaction type of decorators. So it's all a huge gain in control really as these validators allow for explicit validation, whereas static decorators do not at all - they allow calling callbacks, but that's messy w.r.t.this binding (dont-repeat-yourself).

We have to take a look at dynamic validation, just because it's fun and see if we can use Dependeny-Injection safe singletons for it. That would open up a world of options:
https://stackoverflow.com/questions/54057006/nestjs-validating-body-conditionally-based-on-one-property

"We do lose some features like table-capping and mongoose out-of-the-box validators. We gain a lot of validation options as well with much more control and extensibility."

DTO <-> Entity mapping

With class-transformer and class-validator under our belt, we should start looking at DTO mapping. This is a process where the data presented to an API or service is mapped/transformed in order to fit the database specification (no redundant or private data stored, like f.e. passwords). The reverse flow is also important: the database should not provide data which is not meant to be exposed, like user-sensitive metadata or password hashes. This can be dynamically or statically determined as well, for example by using user consent or roles assigned to the logged-in user.

Heed of warning
Auto-mapping DTO's against entities is a heated topic as it is well-known that mapping can result in runtime errors and it increases processing of API's, but it can also reduce repeating code multiple times and prevent bugs in that way.
My personal rule of thumb: if you have more than 5 properties in a DTO/entity and you predict to adjust those properties on more than 1 place, consider mapping. In case you can avoid having 5 flat properties - which MongoDB is really good at - consider nesting the properties within a property type.

In typescript or javascript for that matter, the spread operator ({…objectHere}) is often very attractive to use, also because it seems like TypeScript errors out on any missing or superfluient property. One does not have to specify properties to map manually, so it seems like the dream… or not? The problem is that you can never know which data is in the DTO, especially with a ducktyped language like JS. Therefore a developer needs to be EXTREMELY careful in using the spread property.

A mapper implementation should map what data comes in to what model is targeted, not more, but sometimes it can be less (partial updates). I hope you see how this is quite a complex thing and why it is much harder to control in JS/TS.
Please read about NestJS w.r.t. Validation Pipe and property whitelisting, this is an API superfluency mechanism.

[EDIT] I have found the following module for NestJS based on famous AutoMapper:
https://automapperts.netlify.app/docs/nestjs/

We will use this knowledge in a later section NestJS with VueJS interop.

Client state and cache management

The V1 OctoFarm API has major drawbacks:

1. It manages its state in static context - scalability
2. It loses its complete state after a server reboot - persistence
3. It has to 'clean' its state a lot in order to suit client needs - dont-repeat-yourself

A lot of these problems will be tackled in another memo on state management - it needs much more detail and investigation.

OctoPrint WebSockets

The v1.x API uses the ws library succesfully to connect to all OctoPrint WebSocket interfaces. This data is aggregated and streamed to the frontend using Server-Sent Events. This way the frontend does not have to implement a WebSocket client. This is a nice architecture, we should promote.

Question: should we also create a WebSocket server on OctoFarm for future OctoPrint plugins to pull from or is there only client-server communication? This defines whether we need a WebSocket adapter or not. This adapter can also act as a server for the OctoFarm frontend.
Hypothesis: no, avoid mixing different transport layers with the same purpose. Either reuse the existing client connection over WS or make a plugin consume a well-known OctoFarm API. It's in the plugin's interest to reach OctoFarm and not vice versa.

OctoPrint ClientAPI

The v1.x API also connects to the OctoPrint REST interfaces with a ClientAPI class. This class will have to be inflated to a set of http-proxy client services and controllers, so it becomes a normal API to access for the backend (or frontend in some limited cases). It's quite unclear what the current usages are of the OctoPrint API, and this will have to be 'put into more code and cleaner providers'.

I've added the OctoPrint module in NestJS for this reason. It will contain the OctoPrint Core API as a mirror, not just a GET/POST/etc proxy. This will make sure we can leverage the docs of the OctoPrint REST API and make any differences in our code visible throught TypeScript modeling.

https://docs.octoprint.org/en/master/api/

Aggregating OctoPrint data

In V1 thefarmPrinters array is used to query multiple WebSocket connections with a heartbeat. This is a JavaScript prototype implementation, which will need to be transformed into a transient provider. We can then maintain all websocket connections using dependency injection - in other words we can share the websocket connections across our application. It is not really advised to actually manage or consume the connections on multiple places on our server, but we should promote layered management instead. In order to expose the websocket messages, we can use application Events to publish and subscribe messages to in order to achieve that. See the following for NestJS events, which is quite a common server technique:
https://docs.nestjs.com/techniques/events

A small example on how Events can help us 'spread the word' (pub-sub) about printers across our app in an abstract manner:

Lets say we have a printer which is done printing, and its final message of any kind indicates so (state idle or GCode 100%). Our websocket ClientConnectionManager detects this type of message and transforms the message onto the local event-bus as PrinterStateUpdateMessage with the printer metadata required to operate. We've programmed our FileManagementModule, MonitoringModule, and PrintersModule to subscribe to this kind of message. The actions performed could be:

• the FileManagementModule decides to not do anything, the file storage on the client is not full enough or this file is marked as 'persisted'. Everything is OK.
• the PrintersModule marks the printer state as Operational, it updates PrinterRoomData and it starts crunching PrinterTempHistory. After it's done it pushes the statistics to the MonitoringModule, and if succesful it drops the data.
• the MonitoringModule is separately subscribed to the PrinterStateUpdateMessage to calculate the statistics for this print job, cleanup the gathered data and prepares cool-down data gathering on lower pace (just a random placeholder thought). Secondly it checks what alerts to send and it finds that an Admin user required a push-notification on their mobile and desktop device. It does that and logs it.

This means that we will get the following layering:

• ClientHeartbeatWorker background worker calls ClientConnectionService for status updates and reports immediate successes/failures counts to PrintersModule (state). After that it stores failure backoff retries, so a next scheduled call can use that to not fire off too fast.
• ClientConnectionService has the existing printer connections provider injected and/or will call the printer WebSocket implementations in an order of choosing (I'd say, healthiest first). After that step, it will calculate which printer is up for an update (backoff schedule). It will gather all the Promises, so we can keep on running our server (Does this work?) and join the main coroutine once done.
• ClientConnectionService aggregates the returned or timed-out Promises and decided if an the abstract BasePrinterEventMessage should be fired off.

Question: does awaiting websocket timeouts block the NodeJS event-loop? This has shown to at least hinder it, but we should benchmark this.
Question: does the event system not introduce a huge overhead or overkill? And how to test it? We should really consider race-conditions and CPU load if a certain Event can cause multiple heavy tasks to performed.

Further reading of NestJS Event system leads to the documentation of EventEmitter2:
https://github.com/EventEmitter2/EventEmitter2#multi-level-wildcards
Also note that this is a powerful subscription syntax, albeit that message names are strings - thus prone to bugs.

OctoFarm NestJS with VueJS interop

We should cover the shared modeling of backend and frontend. This is why we've chosen to use TypeScript, so we should start thinking about how to use class-validation (and class-transformation where applicable).

PWA asset caching vs cutting-edge API

Sometimes a PWA app can be out of date due to caching. That's not always avoidable, but I do think we should to try to avoid PWA caching here - until our API stabilizes.
We should think about the API contract with either versioning in mind (debug and interop solution) or with really eager loading. Maybe API versioning, or model versioning, in order to prevent calling the OctoFarm API with old code running in the browser.

The API versioning is something I've had trouble with getting right in Angular for a production ready app. My problem was that my API changed faster than I could cope with on the frontend. I was looking for some mechanism which allowed old code to still cooperate with changed API code (without keeping old API interface models intact too long).
Do note that Angular's download size is mostly larger than both ReactJS and VueJS, so our caching dial was turned all the way up. But it's just a thought, written it down for later reconsideration.

Shared models

The big benefit of NodeJS is that it can run most of code, which the jsDOM can run. Therefore validators, transformers are all available in the browser as well. That's huge!

One caveat is that configurable validators/transformers must have the same context as the backend. An example:

Case study: A username can be changed in the profile page. By configurable constant this username is limited by at least 6 characters minimum on the server DTO as well as database entity. Secondly, the username must be unique once changed, or a 'username-uniqueness' mechanism should be fired. First off, how does the frontend know the actual current configuration of this validation? Secondly, how does the frontend know it should reach the backend to validate a new username proposal?
Hypthesis: the property validator for a model to be put into form (CRUD) should be loaded from the API, when setting up the form and it's validators. Secondly, the asynchronous validator could be ignored as the instance will travel back to the server for validation eventually.

The keen reader will have spotted the challenge with asynchronous backend validation. Do we bring the validation and the API endpoint to call to the frontend or do we verify the model as a whole before submitting it? It's not always a possibility to do verification and saving changes in one step, as further properties can be dependent on the property to be validated. It's entirely dependent on the situation.
A nice way would be to write a class/property decorator which contains the API route and method to call in order for asynchronous frontend validation to occur. Food for thought!