# Dentally Sync Improvement

The initial release of the Dentally integration went down like a lead balloon. Users were unhappy that we only sync from the week previous to the integration. Users would like an integration that provides more history.

The reason why we only sync from the week previous is because Dentally sets a rate limit of 3600 requests per hour (1 per second). The rate limit is then reset on the hour after the first request (rather than replenished).

We are limited to making requests of 100 objects per page. For a large client, such as Dental Suite, their patient list approaches 50,000 patients meaning that a full sync of patients would take 500 requests, which isn't a problem.

The biggest problem is pulling dependent objects, where a separate request per object is required.

Guiding principles:

1. If we design the solution to work for the largest accounts, it will work for the smallest accounts

Possible solutions include:

1. Auditing what we actually sync and removing anything we are unlikely to use
2. Syncing ALL patients as a priority above anything else
4. Syncing all a patient's data just in time when linking
6. Reworking syncing so that it works over a longer period of time

Possible helpful features/tools:

1. https://laravel.com/docs/9.x/queues#delayed-dispatching - this could allow us to calculate how much we need to sync across objects, chunk jobs and schedule them

## Data point audit

- We can probably get rid of `payments` since these are satisfied via treatment plans and invoices - this will save pulling 9938 objects on time-based syncs
- We haven't used `accounts` and I'm not sure if we will at this stage
- I'm not sure if we need `invoice_items` - this will save 18285 on dependency syncs
- It doesn't look like we need to `fees`

## Syncing

Using Dental Suite as an example

### Timings

#### All time
| Object | Records | Requests |
| --------------- | ------- | ----- |
| Patients | 49333 | 494 |
| Invoices | 88233 | 883 |
| Payments | 133300 | 1333 |
| Treatment Plans | 182379 | 1824 |
| Appointments | 461545 | 4616 |
| **Total** | 914790 | 9150 |

This would take around 3 hours to sync.

If we remove `payments`, if deemed unnecessary, we could see:

| Object | Records | Requests |
| --------------- | ------- | -------- |
| Patients | 49333 | 494 |
| Invoices | 88233 | 883 |
| Treatment Plans | 182379 | 1824 |
| Appointments | 461545 | 4616 |
| **Total** | 781490 | 7817 |

This would take 2.1hours to sync.

##### Dependencies

| Object | Requests |
| ---------------------- | -------- |
| Accounts | 49333 |
| Invoice Items | 88233 |
| Treatment Appointments | 49333 |
| Treatment Plan Items | 182379 |
| **Total** | 369278 |

This would take around 103 hours. This is 4 and half days.

If we optimistically remove accounts and invoice items:

| Object | Requests |
| ---------------------- | -------- |
| Treatment Appointments | 49333 |
| Treatment Plan Items | 182379 |
| **Total** | 231712 |

This would take 65 hours to sync. Another option could be to use the patient ID to sync treatment plan items:

| Object | Requests |
| ---------------------- | -------- |
| Treatment Appointments | 49333 |
| Treatment Plan Items | 49333 |
| **Total** | 98666 |

This would take 27 hours to sync.

#### 6 months
| Object | Records | Pages |
| --------------- | ------- | ----- |
| Patients | 5358 | 54 |
| Invoices | 8243 | 83 |
| Payments | 133308 | 1334 |
| Treatment Plans | 15360 | 154 |
| Appointments | 33602 | 337 |
| **Total** | 180511 | 1962 |

This could be done in one sitting. But without payments:

| Object | Records | Pages |
| --------------- | ------- | ----- |
| Patients | 5358 | 54 |
| Invoices | 8243 | 83 |
| Treatment Plans | 15360 | 154 |
| Appointments | 33602 | 337 |
| **Total** | 62563 | 628 |

This would be even more comfortable and would provide us 2972 additional requests in the hour.

##### Dependencies

If we use patient ID and remove unnecessary dependencies:

| Object | Requests |
| ---------------------- | -------- |
| Treatment Appointments | 5358 |
| Treatment Plan Items | 5358 |
| **Total** | 10716 |

This would take 3 hours to sync.

## Implementation

There are two possible implementations:

1. Sync all time-based objects at once and then split up the dependencies into batches, scheduled once an hour
2. Universally sync one page/request at a time, at most once per second

The former would be simpler and would require minimal changes to the syncing interfaces but the latter would be a requirement for universal, reliable syncing.

Doing the former does not prevent us from working on the latter and would provide more utility out of the integration for existing customers.

Another strategy could be to do the former but sync all patients at once so that we have everybody who can be linked. If no records exist for a patient, we can pull the data in JIT on linking.

### Changes

To extend syncing to 6 months:

1. Remove `payments` from time-based extractor
2. Remove `invoice_items` from dependency extractor
3. Remove `accounts` from dependency extractor
4. Change `SyncReceiver` to sync from 6 months ago
5. Define `SyncDependencies` command
6. Implement `SyncDependenciesReceiver`
7. Change `SyncReceiver` to create batches of `SyncDependenciesReceiver` delayed by x amount of seconds
8. Change `LegacyJob` and `Command` to accept a delay in seconds