# Dashboard Jobs Monitoring

* **Jobs:**
* a job is a schedule of a task to be performed periodically or on a fixed point in time
* a job has the following properties:
* endpoint to called when the task is executed
* parameters to pass along with the endpoint call
* fixed date or cronjob
* tags (optional)

* **Instances:**
* an instance of a job represents the execution of a job
* every execution of the same job will create a new instance
* an instance is considered active as long as the task is running
* when a task ends, the instance remains available as logging information
* an instance can contain the following information:
* created, update times
* job information
* running-status (in-progress, failed or succeeded)
* error info in case of failure
* addition status (value, progress)
* sub-instances status

## Jobs
### Locations
**`\glxy-apollo-be-cms\glxy-apollo-be-cms-mlproj\src\cms\lib\jobs.xqy`**
This is the primary module

**`\glxy-apollo-be-cms\glxy-apollo-be-cms-mlproj\src\cms\lib\stg-jobs.xqy`**
This module exposes the same functions as the primary module jobs.xqy, but invokes them for the Staging-database context.

**`\glxy-apollo-be-cms\glxy-apollo-be-cms-mlproj\src\cms\main\jobs\{function-name}.xqy`**
These main-modules exposes the functions from the primary modules, so they are available from nodejs through the app-server. These main-modules need to be called with the Staging-database context.

### Functions
**createJob(...)**
Creates a new job item with following properties:
* endpoint to be called when scheduleData passes
* parameters to be passed with the endpoint call
* scheduleDate or cronjob
* ...

This function is called from the dashboard UI to create a new job

**createInstance(*jobId*)**
Creates a new instance for the given job and returns it's id
This function is called from the scheduler service, wich passes the instance-id to the endpoint

**setStatus(*value*, *progress*, *instanceId*)**
Sets the status values for the instance
This function is called from within the endpoind code, wich will be from nodejs or marklogic

**setSucceeded(*instanceId*[, *response*])**
Set the instance-status to succeeded with optional response information
This function is also called from within the endpoint

**setFailed**(*instanceId*, *errormsg*, *errorcode*)
Set the instance-status to failed with error information
This function is also called from within the endpoint

**initSubs**(*instanceId*, *count*, *options*)
Initializes the instance for sub-instances
This is only necessary when starting asynchronious processes in the endpoint action

@count: the total of asynchronious processes that will be started
@options:
  "auto-progress": boolean defaults to true
  "auto-finish": boolean defaults to true
  "mode": control the amount of information stored per sub-instance
    "full": created- and updated date, description, state, status and response 
    "small": description and final status
    "minimal": only as counts 

**createSubInstance**(*instanceId*, *description*)
Creates a sub-instance. The form of this sub-instance in the instance-document depends on the "mode" in the subs-options.
When mode=minimal, only the _started counter will be incremented
This function is called from within an asynchronious action.
*@returns* a subInstanceId

**setStatus**(*value*, *progress*, *instanceId*, *subInstanceId*)
Sets the status values for the sub-instance
This function is called from within an asynchronious action.

**setSubSucceeded**(*instanceId*, *subInstanceId*[, *response*])
Sets the resulting status of a sub-instance
When mode=minimal, only the _succeeded counter will be incremented

**setSubFailed**(*instanceId*, *subInstanceId*, *errormsg*, *errorcode*)
Sets the resulting status of a sub-instance
When mode=minimal, only the _failed counter will be incremented

## Life-cycle of a synchronous job
* scheduler gets triggered because of scheduledDate
* scheduler calls `jobs.createInstance(jobId)` => instanceId
* scheduler calls endpoint with params + instanceId
* endpoint does some stuff
* endpoint updates instance progress by calling `jobs.setStatus('I did some stuff', 50, instanceId)`
* endpoint does some more stuff
* ...
* endpoint finishes and calls `jobs.setSucceeded(instanceId, 'I did a lot of stuff!')`

## Todo

**Modifications to the endpoints**
To be able to update the state of an instance we need to pass the id to the endpoint
As long as no jobs:set-succeeded() or jobs:set-failed() is called, the instance stays in progress. *TODO: add timeout property*
Each endpoint that can be run as a scheduled job *(to discuss)* must call jobs:set-succeeded() or jobs:set-failed() when the process finishes
Alternatively you can use the sub-instances functionallity with *auto-finish* option

## Service actions that start asynchronious sub-actions (ex: remodel, updateIncremental)
* **sub-instances**
Some actions just starts a set of other asynchronious actions
To be able to monitor this kind of actions, we created sub-instances (subs). They reside in the owner instance document
These sub-instances can have their own status and even progress depending on the options set when initializing the subs (full, small or minimal)
* **minimal-mode**
In minimal-mode there only wil be a count of the started, succeeded and failed sub-instance actions
The option *auto-progress* will set the progress of the master action based on these counts
The options *auto-finish* will set the master instance to succeeded based on the counts *(failed sub-instances also result in succeeded at the moment!)*

## Example usage

### updateIncremental
> This action is mostly running inside nodejs
> So for the logging of job-state we created a js module that
> calls the ML-jobs module directly through the marklogic-nodejs api

### remodel
> This action runs mostly inside ML, so all state updates will be done directly calling the jobs-module

## Demo

### Incremental update
* create an instance (query-console)
* call the getIncremental endpoint and increase the days parameter unit you get some documents to be updated
* call the endpoint with swagger

{
  "days": 1, // fill in the days from previous step
  "instanceId": "76acd18c-aa62-4794-84f2-3aa061f81734" // fill in the correct instance-id
}

* monitor the progress in query-console by running:

(: rerun to update :)
xquery version "1.0-ml";
(for $instance in fn:collection("/jobs/instances")
  order by $instance/modified
  return $instance)[last()]

### Remodel-uris
* create an instance (query-console)
* call the remodel-uris endpoint with swagger

{
  "uris": [
    "/wkbe/pr/brons/docs/law/100786",
    "/wkbe/pr/brons/docs/law/110783",
    "/wkbe/pr/brons/docs/law/111829",
    "/wkbe/pr/brons/docs/law/112082",
    "/wkbe/pr/brons/docs/law/115452",
    "/wkbe/pr/brons/docs/law/115834",
    "/wkbe/pr/brons/docs/law/115855",
    "/wkbe/pr/brons/docs/law/116127",
    "/wkbe/pr/brons/docs/law/117835",
    "/wkbe/pr/brons/docs/law/120197",
    "/wkbe/pr/brons/docs/law/123117",
    "/wkbe/pr/brons/docs/law/123517",
    "/wkbe/pr/brons/docs/law/125975",
    "/wkbe/pr/brons/docs/law/1271",
    "/wkbe/pr/brons/docs/law/13161",
    "/wkbe/pr/brons/docs/law/13162",
    "/wkbe/pr/brons/docs/law/13179",
    "/wkbe/pr/brons/docs/law/149787",
    "/wkbe/pr/brons/docs/law/150002",
    "/wkbe/pr/brons/docs/law/150443",
    "/wkbe/pr/brons/docs/law/152490"
  ],
  "sync": true,
  "instanceId": "76acd18c-aa62-4794-84f2-3aa061f81734" // fill in the correct instance-id
}

* monitor the progress in query-console by running:

(: rerun to update :)
xquery version "1.0-ml";
(for $instance in fn:collection("/jobs/instances")
  order by $instance/modified
  return $instance)[last()]