# Form Builder Model and Use-cases

[TOC]

## Database and Storage

> We will use `MS-SQL Server` for persistent storages and `Redis` for caching.

> ⚠️ We need to think how we will handle some form complex properties like: `conditions`, `Thanks-Screen`

> ⚠️ I Ignored the need to order pages in this model, this need to be handled

> ⚠️ we need to consider data loading impact for new user and history sync

> ⚠️ we need to consider load important props only, in this case the source of truth about old props is the frontend, no data source validation

> ⚠️the source for changes is history as cache has no state stored

> ⚠️ when publish we need to read cached version not only persisted (trigger persistnce sync)

### Persistent Storages

#### Form Information `* Added 19-09-2021`

Table Contains a row for each form holding its main information without any details about its scheme.
This table should work as the aggregate root.

- `FormId`
- Some auditing information
- Some form properties
- `Last Published Version Id` ?

#### Drafted Form

Table Contains a row for each form holding its current state.

- `FormId`
- `Schema`
- Some auditing information
- Some form properties

#### Published Form

Table Contains a row for each form version (same form has multiple rows) holding its current state.

- `VersionId`:`Int vs Guid?`
- `FormId`
- `Schema`
- `IsUsed`
- `IsLatestVersion`: for better indexing?
- Some auditing information
- Some form properties

#### Form History (Events)

Table Contains a row for each event/command used to change the state of the form.
This table has a rule to keep only a specific number of events for each form `{History-Limit}`

- `EventId`
- `EventType`
- `EventData` (contains the previous state as well) `old, new in two props?`
- `FormId`
- `TimeStamp`
- Some other auditing information

### Cache (Redis)

#### Form Properties

We will have single `Hash` for each form to represent its the form global properties.

- Key: `Form:{FormId}`
- Value Type: `Hash<Property, Value>`

#### Form Controls

We will have a `Hash` for each control, so we could get and set its properties directly.

- Key: `FormControl:{FormId}/{ControlId}`
- Value Type: `Hash<Property, Value>`

#### Form History (Events)

We will have a `List` for each form, this list has a limited size `{History-Limit}`.

- Key: `FormEvents:{FormId}`
- Value Type: `List<EventDetails>`

#### Control Groups (Section, Page)

We will have an `Ordered Set` for each form group (Section, Page).
This sets will be sued to optimize `Reorder` and `Reparent` Operations

- Key: `ControlGroup:{FormId}/{ParentId}`
- Value Type: `OrderedSet<ChildId, Order>` | Value: `ChildId` | Rank: `Order`

#### Form Conditions

We will have single `string` for each condition to represent its the condition details as JSON string.

- Key: `FormCondition:{FormId}/{ConditionId}`
- Value Type: `string`

> We may need to have `Redis` collection to represent the condition/control dependency relation, for easy search

## Problems and challenges

### Dependency Inconsistent validation

#### Description

There will be inconsistency in the data stored in `Redis` cache if we apply some changes in the data depending on the validation occurs on the `Redis Client` (server) on some other data read before, as these three steps can't be represented as an atomic transaction (read - validate - manipulate). the data could be modified between the validation and the manipulation.

#### Example

- [00:00:00]
- Client 1: send request to reparent label to section
- Client 2: send request to remove this section
- [00:00:01]
- Thread1: read label, section info from `Redis`
- Thread2: read section info from `Redis`
- [00:00:02]
- Thread1: validate order (require some time to decode and compare order)
- Thread2: remove the section
- [00:00:03]
- Thread1: update parent, order properties in the label
- Thread2: send response publish the remove
- [00:00:04]
- Thread1: publish reparent command
- [Final result]
- Label reference none exiting parent

#### Solutions

- **Solution 1**: Making a second (read - validation) after the manipulation in order to trigger some sort of error or undo the whole transaction depending on the context [**Recommended**].
- **Solution 2**: Using optimistic update and retry on with the new correct version of the data [**Not Recommended** (High overhead on CPU and the database/ cache)]

#### Use-cases

- [Add New Control](#Add-New-Control)
- [Change Control Parent](#Change-Control-Parent)

### Inconsistence History (Event Order)

#### Description

This problem could occurs if two events (Command) executed the same time, the second command may be completed (Manipulation, Append Event) before the first command start to append its details to the history.

#### Solution

- **Solution 1**: Using `Redis` **Transaction** and perform the history append operation atomic with the actual manipulation.

## Lifecycles and use-cases

### Cache Life Cycle

> Loading and Clearing operations will be delayed with a specific period of time to prevent quick load/clear cycle.

- The form details will be loaded into the cache after the first form builder web-socket connection is opened.

- The form details will be cleared from the cache after all form builder web-socket connection are closed and the persistence sync is completed.

### Persistence Sync
> Send history size on persisit to be read and cleared.

The Persistence Sync could be triggered with

1. The size of `Form History` in cache reach a specific size `{sync-Threshold}`, this check is done after appending any new event details.
2. When all form builder web-socket connection are closed, after this completed the cache clear operation should start.

> How to differentiate between history original size and the new commands size for `{sync-Threshold}` check?
> we may need a **counter** for each form and use increment with each command added and reset after persistence sync operation.

### Form Builder Use-cases

> Redis Manipulations: (Set property, add key, remove key, append to history) is performed in a single transaction in each of the following use-cases

> Consider Using `SETNX, HSETNX, LPUSH` When adding any new `Redis` key to catch the duplication if exists.

All the following use-cases share the same generic flow with some specific rules and steps in the middle

1. Client Send Command with `SignalR`
2. Server start executing the command
3. Read any required information from `Redis`
4. `...use-case specific operations`
5. Append the event details in the cache `Form History`
- Check if the history size reach the`{sync-Threshold}`
- Trigger the Persistence Sync if reach the threshold
6. `...perform the second validtion if it was required` [[Dependency Inconsistent validation](#Dependency-Inconsistent-validation)]
7. Server response to the caller with the execution result (Done/Error)
8. Server broadcast the changes to other clients

#### Add New Control

1. Validate (`FieldId`, `ParentId`, `Order`) --> Error / Continue
2. Fix `Order` if there was an order conflict
3. Add the new `FormControl` to `Redis`
4. Add new `ControlGroup` if control was `Section` or `Page`
5. Add `FieldId` ranked with its `Order` to Its parent `ControlGroup` if control is not `Page`
6. `... Complete appending the event details as part of the redis transaction`
- ⚠️ we need to be able to remove this event if second validation fails.
- ⚠️ moving this step out of the transaction may result in the inconsistent events order [[Inconsistence History (Event Order) Problem](#Inconsistence-History-Event-Order)].
7. Validate (`ParentId`, `Order`)
- Fix `Order` again if there was an order conflict
- If the parent was deleted before the the adding event, then remove the field, the event and return parent not found error.
- If the parent was deleted after the the adding event, then remove the field, keep the event and return warning.

#### Remove Control

1. Validate (`FieldId`) --> Error / Continue
2. If the field is `Section` or `Page`
- Remove All children Hashes
- Remove the `ControlGroup` identified with this control as parent
3. Remove Field Hash
4. Remove `FieldId` from its parent `ControlGroup` if control is not `Page`
5. Remove any `Conditions` depend/affect this filed ❗ [Obsolete Conditions](#Obsolete-Conditions-⚡)

#### Change Control Property

1. Validate (`FieldId`) --> Error / Continue
2. Update the control property in `Redis`

#### Change Control Order

1. Validate (`FieldId`, `Order`) --> Error / Continue
2. Fix `Order` if there was an order conflict
3. Update `Order` property in the control Hash

#### Change Control Parent

1. Validate (`FieldId`, `Order`, `ParentId`) --> Error / Continue
2. Fix `Order` if there was an order conflict
3. Update `Order` and `ParentdId` properties in the control Hash
4. Remove `FieldId` from its old parent `ControlGroup`
5. Add `FieldId` to its new parent `ControlGroup`
6. `... Complete appending the event details as part of the redis transaction`
- ⚠️ we need to be able to remove this event if second validation fails.
- ⚠️ moving this step out of the transaction may result in the inconsistent events order [[Inconsistence History (Event Order) Problem](#Inconsistence-History-Event-Order)].
7. Validate (`ParentId`, `Order`)
- Fix `Order` again if there was an order conflict
- If the parent was deleted before the the reparent event, then undo the reparent operation, remove the event and return parent not found error.
- If the parent was deleted after the the reparent event, then remove the field, keep the event and return warning.

#### Revert from History

> **Not Sure**: we may be in need to lock the form and block the manipulations until the revert is completed to prevent any unpredictable behaviors.

> ⚠️ **Warning**: we need to consider the revert impact on the conditions [[Obsolete Conditions](#Obsolete-Conditions-⚡)]

This feature could be implemented in 3 different ways:

1. Append `RevertEvent` with the last `EventId` (❌Could be used only if we keep the entire form history without deletion)
2. Remove all events after the selected one to be reverted to
- Require to keep tracking of the deleted besides the undeleted events
- User can't revisit any of the deleted events to redo some changes
- Frontend `Redo` to cancel the `History revert` operation may not be applicable.
3. Append `RevertEvent` with difference list of the revert impact (state difference after and before applying the revert operation).

##### Revert with Difference List

1. Read all events to be reverted
2. Merge the events backward to remove duplicated manipulation on the same property
3. Perform the merged events in order
4. Add the operations list as part of the `EventDetails` appended to the history

**Example:**

- [E5]: Change `label 1` color from blue to red 🟥
- [E4]: Change `label 1` color from yellow to blue
- [E3]: change `label 1` text from "Label" to "Device Information"
- [E2]: Add `Section 1` to `page 1` with order "0.3"
- [E1]: Change `label 1` color from black ⬛ to yellow
- [E0]: Add `label 1` to `page 1` with order "0.5"
- Revert to `[E1]`:
1. [Reversed-E3]: Change `label 1` text from "Device Information" to "Label"
2. [Reversed-E2]: Remove `Section 1`
3. [Reversed-(E5,E4,E1)]: Change `label 1` color from red 🟥 to black ⬛

> **Note**: `[Reversed-(E5,E4,E1)]` merges all changes on the label color property, to be represented as change from the latest state to the oldest state.

### Form Conditions

The form conditions manipulations will be performed using simple `HTTP Rest APIs`, ~~which will be performed on `Redis` and will be priested with the history and the controls~~.

- Add Condition
- Edit Condition
- Delete Condition

#### Obsolete Conditions ⚡

The obsolete conditions are the conditions depending on/affecting non-existing fields.

- [Option 1]: Delete any obsolete condition after deleting one of its field s result of [control remove](#Remove-Control) or [History Revert](#Revert-from-History)
- We will be in need to have a second validation after applying Add/Edit Condition operations [[Dependency Inconsistent validation](#Dependency-Inconsistent-validation)]
- **Not-Simple Reverting**: We will be in need to store the deleted conditions in the history, to be able to revert and add the condition again if the user revert the delete operation.
- **Simple Reverting**: We could warn the user the deleted conditions will not come back after revert the field remove operation, and keep no info about the conditions in the history.
- [Option 2]: Keep all obsolete condition, don't delete the condition after deleting any field
- This solution will keep the conditions and the history separated from each other
- The user will have the option to manually (button click) delete all obsolete conditions
- A warning could be presented to the user to represent the obsolete conditions that needed to be modified or deleted by the user
- Obsolete condition will be added to the `DraftedForm` Scheme
- Obsolete condition will not be added to the `PublishedForm` Scheme
- Inactive Flag could be switched on read with the controls existence check (computed /not persisted)

### Publish Form

- Short discerption (Will be expanded in details), copy form draft, add if the last published is used, remove the old unused published form before add new version.

1. Client Sends Publish command with `HTTP POST`
2. Read form information from `Redis`
- Load information from `DraftedForm` if the publish triggered outside the builder and the form is not in the cache (⚠️ This option may be not available outside the builder).
3. Check the last published version of this form
- If the last version was unused, delete its record
4. Create new record on the `PublishedForm` with the new version

## Domain Model

### Entities

> The property `FormConrols` is a Collection Mapped in memory from `Schema`.
> The property `FormId` is not a `foreign key` , we don't need to reference entities from each other.

- Abstract `FormBase` (`FormId`, `Schema`, `FormConrols`, ...Some other form properties )
- Abstract `FormControl` (`ControlId`, `Type`, `Order`, `ParentId`, ...)
- `DraftedForm: FormBase` (`Ispublished`, `VersionId` )
- `PublishedForm: FormBase` (`PublishId`, `PublishTime` )
- `FormHistory` (`EventId`, `FormId`, `EventType`, `EventData`)

### Value Objects

- `EventData` : we need to have a specific type for each `EventType` driven from abstract `EventData` class

### Aggregates and roots

- `DraftedForm`
- `PublishedForm`
- `FormHistory`