# Dentally Data Sync

We would like to build tools to link patients in Leadflo with Dentally so that, to start, we can synchronise financial data.

Staff use practice management software (PMS) to help manage the day-to-day of the practice. As a CRM, Leadflo are of overlap areas but is distinct. This leads to a large amount of "double work" where staff need to keep both Leadflo and PMS in sync. One such piece of software is Dentally, which has the advantage of providing an open, modern REST API for consumption by integrations.

This problem is worth solving because:

1. Our main competitor, DenGro, boasts a Dentally and SoE integration
2. Lack of a Dentally integration has had an negative impact on sales
3. A PMS integration will help us build a better experience for our users
4. Building a Dentally integration informs how we build other PMS integrations

This design focuses entirely on syncing data from Dentally to Leadflo ahead of transforming it into a state we can use to build new features. Anything beyond that is out of scope.

## Goals

Right now, we do not know what potential there exists between linking Dentally and Leadflo data. Before we can proceed with an ideal set of integration features, we need data from Dentally in a form we can explore via SQL.

This will then inform us on the potential for:

1. Automatic linking between patients in both systems
2. What we will need to build to facilitate manual matching
3. How we can improve on our automatic / semi-automatic linking mechanisms
4. What patients cannot be linked at all and how we identify them

This opens up a number of opportunities that help us move our product strategy forward. Immediately, this work will allow us to report on financial objectives (a distinct feature on the roadmap) both patient-by-patient and in aggregate _without_ manually inputting that data (which is something DenGro does not do). In future this will allow us to:

1. Sync location data so we can build catchment reports which could inform business growth via opening of new locatiions _without_ user input
2. Sync clinicians and which patients they are seeing so we can segment reports by clinician _without_ input from the user (another feature on the roadmap)
3. Potentially sync treatment progress to eliminate a large swathe of actions due that must be processed manually
4. Sync treatments and typical financial values so we can accurately estimate the value of each treatment type
5. Better use these data features (in data science sense) to explore and determine a lead scoring algorithm (another distinct feature on the roadmap)

In addition, building this portion of the data sync will inform our technical stategy by:

1. Battle-testing our existing integration abstractions such as authentication, management workflows and health checking
2. Showing the impact of an integration on performance and scalability
3. Testing the viability of Extract-Load-Transform vs Extract-Transform-Load for complex integrations
4. Highlighting potential seams when scaling the API to individual services

## Scope & requirements

The scope of this work is restricted to the Extract-Load parts of the Extract-Load-Transform cycle.

### Product requirements

As a user, I want to:

* Enable the Dentally integration so that I can link data between Leadflo and Dentally
* Re-sync data from Dentally into Leadflo so that I can keep the data in Leadflo up to date
* Re-connect Dentally to Leadflo when the connection fails so that I can keep the data in Leadflo up to date
* Disconnect Dentally from Leadflo so that I can stop syncing the data when I don't need it anymore

As a developer, I want to:

* Explore the congruence of data between Dentally and Leadflo using SQL so that I can validate opportunities for new features
* Encourage the link between Dentally and Leadflo for all users so that we can synchronise as much data as possible

### Technical / non-functional requirements

* **Performance** - the work should not raise P95 response times across the board by 50ms to maintain existing user experience
* **Scalability** - the data from Dentally must fit within our DB instance for potentially all of our clients
* **Testability** - all logic in the Dentally integration must be in some way testable without depending on the availability of the API
* **Availability** - the availability Leadflo must not depend on the availability of the Dentally API (satisified with existing integration abstractions)
* **Extensibility** - we expect to build new features on top of this integration and it must be easy to extend
* **Privacy** - we must ensure that the data we sync from Dentally is reflecting in our privacy policy and that our clients reflect this in their privacy policies

Questions:

* How should these non-functional requirements be prioritised?

### In scope

Transforming the data into a shape that Leadflo can use is out of scope for this piece of work. But it **will** be within the scope of work immediately following this piece of work so it must be considered as part of the design.

To satisfy upcoming requirements, we must sync:

* Patients
* Accounts
* Payments
* Invoices
* Invoice Items
* Treatments
* Fees
* Payment Plans
* Treatment Plans
* Treatment Plan Items

It will also be useful to explore the data from these sources:

* Appointments
* Treatment Appointments
* Session Appointments
* Practitioners

### Required Resources

For the solution to work as described, we will need:

* A Dentally sandbox account
* Access to a client's Dentally account
* Access to their [development documentation](https://developer.dentally.co/#overview)

## Solution

### Current solution


We use a Extract-Transform-Load (ETL) process to implement integrations.

This has a number of advantages:

1. We do not store data redundantly
2. We do not require multiple levels of persistence
3. Integrations perform well on-demand with little lag between a change in integration data and it being reflected in Leadflo

On the other hand, this means:

* We have little opportunity to explore the data in form we receive it
* As such, we have to use reasoning and documentation to hypothesise how integration data links with Leadflo data
* Re-syncing involves reading from the API again for data we have already read

For simple integrations where we know exactly how the data fits in with Leadflo, this works fine but we can forsee this breaking down with an integration where there are many unknown-unknowns involved with linking the data to Leadflo.

### Proposed solution


Instead, we will use Extract-Load-Transform (ELT) where we:

1. Read the data we want to integrate from the Dentally API
2. Load the data as-is into persistence
3. Transform the data into a form where Leadflo can use it

As with ETL, we will need to read from the Dentally API. Specifically, we will need to read the following objects from Dentally:

* https://developer.dentally.co/#patients - like Leadflo, this will be the "root" object which we must have before we sync other objects
* https://developer.dentally.co/#the-payment-object - these objects represent when a payment is made by a patient, allowing us to record financials over time
* https://developer.dentally.co/#the-account-object - these objects represent an aggregate of a patient's financials, including current balance and opening balance
* https://developer.dentally.co/#the-invoice-object - these objects tell us about individual charges to a patient, along with amount and amount outstanding
* https://developer.dentally.co/#the-invoice-item-object - these objects break down invoices so we can tell where value is coming from
* https://developer.dentally.co/#the-treatment-object - these give us a set of treatments at the practice so we can tie payments to treatments
* https://developer.dentally.co/#fees - these attach fees to individual treatments (akin to a fees table)
* https://developer.dentally.co/#the-payment-plan-object - these represent the ways in which patients can pay for their treatment
* https://developer.dentally.co/#the-treatment-plan-object - these represent a set of treatments over time - a treatment plan is the core product sold by a dental practice
* https://developer.dentally.co/#the-treatment-plan-item-object - these represent individual treatments on a treatment plan and is tied to a treatment, a treatment plan, a patient and so on

Since we're designing a solution to sync all of these objects, it might also be useful to sync other objects that we can use to explore the data for further valuable opportunities for new features:

* https://developer.dentally.co/#the-appointment-object - these objects will be useful to explore how we can automate certain actions due in Leadflo
* https://developer.dentally.co/#session-appointments - as above
* https://developer.dentally.co/#treatment-appointments - as above
* https://developer.dentally.co/#practitioners - this could allow us to automate the linking of patients to clinicians which would satisfy tracking by clinician without requiring user input beyond linking the patient

In addition, we will need to listen for signals these events have changed so we will need to use [webhooks API](https://developer.dentally.co/#webhooks) to register those changes. Specifically, we will need to track these events:

* `patient.created`
* `patient.updated`
* `payment.created`
* `payment.updated`
* `appointment.created`
* `appointment.deleted`

Note: although apppointments are out of scope, it would be helpful to include it since it's the only remaining webhook event. Otherwise, we would have to consider changing webhooks for the sake of adding it in later.

The lack of documented webhook events for other objects, like `invoice.created`, is concerning. We should experiment with the API and determine the answer to these questions:

* Does, for example, creating an invoice trigger any of the above webhook events?
* Are there more, undocumented webhook events? e.g `invoice.created`
* Is there a viable way to sync invoices using a purely pull model, without tripping the rate limit?

Instead of trying to keep track of what we need to sync, we should treat webhook events as signals and proceed to sync from a last synced date. Dentally's API allows us to list objects by date updated/created after a certain date.

#### Data Model

##### Dentally Objects

There are four requirements we must meet with the data we wish to sync:

1. Reliably pair up patients between the two systems
2. If not, reliably suggest which patients to pair with
3. Pull in amount paid by each patient
4. Pull in the unpaid treatment plan value by each patient

These fields on the patient object may help us identify patients in Leadflo:

* `family_id` - this field may help us identify related patients who may share the same phone/email address.
* `preferred_phone_number` - Dentally stores 3 different phone numbers for the patient. A matching preferred phone number could be a stronger indicator of a match than one of the numbers matching
* `preferred_name` - the name in Leadflo may match the preferred name over the legal name given in Dentally

These objects should allow us pair data up with Leadflo and report on financial information.

##### Schema changes

We will need tables that match the schemas above to persist the data from Dentally. These must match Dentally exactly so we need to copy these into a schema in this order:

1. Patients
2. Accounts
3. Payment Plans
4. Payments
5. Invoices
6. Invoice Items
7. Treatments
8. Fees
9. Treatment Plans
10. Treatment Plan Items

Invoice items refers to treatment plan and treatment plan items. Treatment plan items, in turn, refer to an invoice ID so it's not clear where the foreign key relationship lies (chicken and egg).

Perhaps we should eschew foreign key constraints entirely and depend on indexes. In general, we will need some time to understand the correct indexing strategy as we won't know how these tables will be consumed.

I'm not sure if we should migrate all of what appears in the treatment plan objects because there appears to be some incredibly sensitive information in there. We should not sync which teeth and surfaces are being treated.

In addition, we will need `client_dentally_settings` to track the state of syncs per client. This will need a field representing last sync from Dentally and another for the last sync to Leadflo.

| Field | Type | Constraints |
| -------------- | --------- | ----------- |
| client_id | int | PK |
| synced_at | timestamp | |
| transformed_at | timestamp | |

##### Data additions

We will need to add `dentally` to the `integrations` table so that it can be referenced as a foreign key in `client_integrations`. The `visible` attribute should be set to `false` until we are ready to roll this out.

##### Persistence

In congruence with ELT, persistence of extracted objects will be managed by "Loaders".

The sole responsibility of a `Loader` is to accept an object with a given schema and load it into one or more database tables.

Each `Loader` should have an in-memory counterpart that can be used when unit testing.

One interface for a `Loader` could be:

interface Loader
{
    /**
     * @param array{ patients: Generator<int, PatientObject>, ... }
     */
    public function load(array $objects): void;
}

Note: this is an example interface. There should be a separate interface for each type of object.

An implementation of a loader would accept a stream of objects and persist them to the correct database table(s). I suggest we implement a handful of `Loader`s manually and seek to generalise since it'll be incredibly repetitive. It is currently difficult to see how these can be generalised without first knowing what is different and what stays the same.

Since there is a high level of congruence between the shape of the objects and the shape of the schema, we could use a more dynamic approach that eschews static representation of the data (e.g an array of dicts over a DTO for Patient, Account, etc).

We will need, along with implementations:

* `PatientsLoader`
* `AccountsLoader`
* `PaymentsLoader`
* `InvoicesLoader`
* `InvoiceItemsLoader`
* `TreatmentsLoader`
* `FeesLoader`
* `PaymentPlansLoader`
* `TreatmentPlansLoader`
* `TreatmentPlanItemsLoader`
* `AppointmentsLoader`
* `TreatmentAppointmentsLoader`
* `SessionAppointmentsLoader`
* `PractitionersLoader`

Could we merge some of these into a single loader? E.g invoices and invoice items are closely related, as are treatment plans and treatment plan items. This could, however, complicate matters.

Or could we compress the whole set into a single Loader which takes a set of streams, consumes them and populates the DB? This would be a lot simpler to use and would present with an incredibly powerful interface.

#### API changes

There are a few changes we will need to make to the API service:

* An `IntegrationServiceProvider`
* Integration interface implementations (e.g `AuthGateway`)
* Webhook endpoints
* A `Sync` command and `SyncReceiver`
* Integration-specific `Extractor`s (both real and faked)
* An `Integrator`
* Change events e.g `PatientChanged` or `DentallyChanged`
* Listeners that respond to change events
* An API client with OAuth authentication

We will also need to consider HTTP failures when interfacing with Dentally. We have done this before with Gmail, Outlook and Infusionsoft and this works well so we should do something similar. We should, however, avoid doing what Outlook does and push the logic into a place that can be tested with a much simpler `Extractor` implementation.

##### Interfaces to implement

We will need an `IntegrationServiceProvider` with `dentally` as the name (which binds any integration-required implementations). As such, we will need a handful of dependencies first.

We will need an API client, which depends on an OAuth2 provider:

* `DentallyProvider`
* `DentallyResourceOwner`
* `DentallyClient`

We will also need an implementation of `DentallyAuthGateway` which uses the above to authenticate against the API. Dentally doesn't provide a token refresh mechanism. It is entirely based on whether the token is used. This method should simply return the token for compatibility.

We will also need a `Watch` command, with `WatchReceiver` stubbed out and bound:

final class Watch extends IntegrationWatch
{
}

We will also need a `Sync` command, with `SyncReceiver` stubbed out and bound:

final class Sync extends Command implements IntegrationSync
{
    public function __construct(
        public IntegrationSyncID $id
    ) {}
}

We will need an `WatchGateway` interface along with API and memory implementations:

interface WatchGateway
{
    public function watch(Subscription $subscription): void;

    public function delete(Subscription $subscription): void;
}

Where `Subscription` is a simple value object that includes the endpoint, name and secret.

final class Subscription
{
    public function __construct(
        public readonly string $endpoint,
        public readonly string $name,
        public readonly string $secret
    ) {}
}

##### Interfaces to introduce

In addition, we will need a number of new interfaces and implementations to develop and test the Dentally integration.

To track the state of previous sync per client, we will need:

* `SettingsGateway` interface
* `DBSettingsGateway` impl
* `MemorySettingsGateway` impl

This `SettingsGateway` is different in that it will accept two history tokens (as per the schema). This is so we can track both the position of syncs from Dentally and syncs to Leadflo in future.

interface SettingsGateway
{
    public function get(): Settings;

    public function persist(Settings $settings): void;
}

Where `Settings` is a simple value object that represents a client's settings:

final class Settings
{
    public function __construct(
        public readonly DateTimeImmutable $syncedAt,
        public readonly DateTimeImmutable $transformedAt
    ) {}
}

And to encapsulate calling the Dentally API, we will need a set of `Extractors` for all of the objects we need to sync. These are responsible for calling the API and fetching the data in a form that can be used by a `Loader`. Little to no transformation should occur to make this happen, except perhaps to exclude fields that we do not wish to touch.

All `Extractors` will have `API` and `Memory` counterparts, the latter for use during unit testing.

An `Extractor` could have the following interface:

interface Extractor 
{
    /**
     * @return array{ patients: Generator<int, PatientObject>, ...}
     */
    public function since(DateTimeImmutable $at): array;
}

As with `Loader`s, it's difficult to see how we can generalise extraction. But this may be possible. Using a single, powerful interface would keep extraction in one place and allow us to see, via proximity, great opportunities for generalisation.

This goes too for rate limiting and pagination handling, which would work exactly the same for all objects. In addition, these concerns could be handled via traits which can be attached to anonymous classes and tested in isolation.

It should be noted that these interfaces will need to support either a full sync or a sync from the last sync date (retrieved via `SettingsGateway`). The `Integrator` should be responsible for combining `SettingsGateway` and the API gateways and should encapsulate the logic in retrieving a stream of patient objects and payment objects. A "full sync" will not sync _all_ data - it will instead sync from a certain amount of time ago. We will go with 6 months but we should make this configurable in case we need more data given patients in PMS have multi-year lifetimes.

They will also be rate-limited and paginated. We are limited to 3,600 requests per hour per user. We are also limited to a maximum of 100 items per page.

Rate-limiting information is provided in response headers, which we can use to control extraction so it avoids tripping the rate limit.

* If we exceed the limit, do we have to wait for an hour or does it work on a per-second credit system given 3600 requests per hour?

##### Commands & receivers

We have already introduced `Watch` and `Sync` commands. We are required to implement the receivers for these commands.

###### `WatchReceiver`

The `WatchReceiver` is responsible for idempotently setting up webhook listeners for change events on relevant objects from Dentally. This will use the `WatchGateway` to achieve this. In addition, it will also require a route:

* `POST /integrations/{clientId}/dentally/changed`

These routes will need to be bound to `PostDentallyChanged` controller that extends from `IntegrationController` according to the rules of the Dentally webhooks system. From the documentation, it appears these operate much like Outlook webhooks.

We will also need to validate the signature of each webhook message. This can be achieved using a HTTP middleware. This is stored in the header `X-Dentally-Signature`. We can validate the signature by translating and adapting the following Ruby code provided by Dentally:

# request_signature - the signature from the X-Dentally-Signature header
#      request_body - the JSON body of the webhook request
#            secret - the secret for the webhook

require "openssl"

digest = OpenSSL::Digest.new("sha256")
calculated_signature = OpenSSL::HMAC.hexdigest(digest, secret, request_body)

if calculated_signature == request_signature
  # Signature matches, everything is good
else
  # Ignore the webhook as the signature is invalid
end

###### `SyncReceiver`

The `SyncReceiver` is responsible for using the `DentallyIntegrator` to pull a stream of objects from `Dentally` and use the repositories to persist these objects in the DB.

In the near future, we will need some kind of listener bound to `Heartbeat` that takes unsynchronised objects out of the Dentally tables, transforms them and passes them to Leadflo to integrate.

To do this, we could store two tokens in the settings table. One is for the last Dentally sync and the other is for the last sync to Leadflo.

This means we will need to track the created and updated datetimes in our own tables so we can query against them.

##### Listeners

We will only need an `AuthenticateClient` listener that looks up the client's integration token and binds it to the Dentally API client. This could be a candidate for generalisation for similar APIs but it is only required on Outlook right now.

#### Presentation

Users will need the ability to manage the integration once available in the same way that Gmail and Outlook can be managed.

Much of the work for this is already done but we will need a few additions:

* Adding the Dentally logo SVG
* Composing `DentallySetup`, `DentallyManage` and `DentallyRepair` flows
* Integrating the above into `ConnectedDentally` component

It is not clear what setup instructions we need to implement. We might need to make the setup video optional if it isn't applicable to a certain integration.

#### Other considerations

We will need to monitor DB capacity. This will inform us of whether we need to increase DB storage ahead of a full roll out. We can expect similar pressure on capacity from an SoE integration - though if we get Dentally right, we could use the traditional ETL process since we would have, by then, developed all of the rules and mechanisms for automatically and semi-automatically matching patients to patients.

Since PMS is used heavily, we can expect it to bear a higher load on our API tasks. We should test this and be prepared to increase the number of API tasks.

This will give us enough information to assess how this will impact us when we scale to 100 clients.

To extend this solution to cope with future requirements, we will need to add new Dentally tables, by design. There should be zero need to modify core tables.

### Testing

The requirements at this stage are to connect to Dentally and successfully populate the dentally tables with 6 months of data.

As such, we could cover the endpoint with a wide-coverage integration test using memory implementations of the gateways and verifying the contents of the database.

We should minimise the complexity of the gateways so that the logic remains unit testable. This was, I feel, a mistake on my part when building the Outlook gateways and a number of defects have been discovered in that code since release.

That said, much of the integration abstractions are already well tested and production-hardened so we do not need to test these things explicitly.

After release, we should:

1. Connect the master account with our Dentally sandbox account in production and verify synchronisation of data
2. Connect Changing Faces LF account with their Dentally account and again verify synchronisation of data

### Monitoring

Integrations would benefit from a number of metrics in general:

* Total client integrations gauge
* Healthy integrations gauge
* Unhealthy integrations gauge
* Disconnected integrations gauge
* Total syncs counter
* Syncing gauge
* Completed syncs counter
* Failed syncs counter

We may need to upgrade our Grafana instance to paid to acommodate this.

We should also gauge the capacity of the database:

* Total capacity gauge
* Amount used gauge

We should alert on approaching the DB capacity. In the event of this, we should just increase the storage capacity of the DB instance. I suggest this be a conservative 75% of capacity and another at 80% and that we optimistically double the size of storage similar in principle to expotential backoff. This can, however, be a point of discussion.

We can estimate the impact on DB capacity by:

1. Taking the current size of the local DB (or from scratch)
2. Syncing from Dentally locally
3. Measure the change in DB capacity
4. Extrapolate into an estimate to both 30 and 100 clients
5. Compare against current DB capacity

### Roll-out plan

We will need a `dentally` feature flag. This will be used to hide the integration either in the API or UI.

We should trial the solution first with:

1. Changing Faces
2. Dental Suite

We will need to craft a release email for canaries but we do not need to crat full release materials at this stage since it will remain a canary release for some time.

## Success

High-level success will mean both Changing Faces and Dental-Suite are hooked up to Dentally and syncing data without unrecoverable failure. In addition, the Dentally data will be queryable on-demand against LF's data set, which we can use to explore further opportunities for new features.

This translates into these metrics:

* 2 healthy Dentally client integrations
* Regular 99.99% successful Dentally syncs

We should mark the release of Dentally in Grafana and ensure the P95 response time is impacted by no more than 50ms. Anything more _should_ be anamolous because the endpoints should merely push onto the queue. With a high number of requests, however, this could degrade performance across the board and given how busy Dentally generally is (it's used all day every day), we can expect a high number of requests.

Otherwise, this work will be completely isolated from the rest of Leadflo so should not have any impact on other components.

## Work

### Tasks

Pre-requisites:

- [ ] Sign up for a Dentally sandbox/developer account
- [ ] Implement integration metrics
- [ ] Implement DB capacity metrics
- [ ] Define `dentally` feature flag
- [ ] Prevent displaying dentally in integrations list if disabled

Experimentation:

- [ ] Find out if `patient.updated` webhook events are triggered by invoices
- [ ] Find out if there exists undocumented webhooks
- [ ] Find out the viability of using a pull-based mechanism for objects we do not have webhook events for

Database:

- [ ] Define `dentally_patients` schema
- [ ] Define `dentally_payments` schema
- [ ] Define `dentally_accounts` schema
- [ ] Define `dentally_invoices` schema
- [ ] Define `dentally_invoice_items` schema
- [ ] Define `dentally_treatments` schema
- [ ] Define `dentally_fees` schema
- [ ] Define `dentally_payment_plans` schema
- [ ] Define `dentally_treatment_plans` schema
- [ ] Define `dentally_treatment_plan_items` schema
- [ ] Define `dentally_practioners` schema
- [ ] Define `dentally_appointments` schema
- [ ] Define `dentally_treatment_appointments` schema
- [ ] Define `dentally_session_appointments` schema
- [ ] Define `client_dentally_settings` schema
- [ ] Create migration to insert `dentally` integration

UI:

- [ ] Add Dentally SVG to UI resources
- [ ] Make setup video optional
- [ ] Implement `DentallySetup` flow
- [ ] Implement `DentallyManage` flow
- [ ] Implement `DentallyRepair` flow
- [ ] Compose together into `ConnectedDentally` component

Integration:

- [ ] Implement Dentally OAuth2 provider
- [ ] Implement `DentallyClient`
- [ ] Implement `AuthenticateClient` listener
- [ ] Define `Watch` command
- [ ] Stub out `WatchReceiver`
- [ ] Define `Sync` command
- [ ] Stub out `SyncReceiver`
- [ ] Implement `DentallyAuthGateway`
- [ ] Setup `DentallyServiceProvider`

Watching:

- [ ] Implement `VerifySignature` middleware
- [ ] Define `POST /integrations/{clientId}/dentally/changed` route
- [ ] Implement `PostDentallyChange` integration controller
- [ ] Define `Subscription` value
- [ ] Implement `WatchGateway` interface and implementations
- [ ] Implement `WatchReceiver`

Settings:

- [ ] Define `Settings` value
- [ ] Implement `SettingsGateway` interface and implementation

Pulling data:

- [ ] Define `Extractor` interface
- [ ] Implement `APIExtractor`
- [ ] Implement `MemoryExtractor`

Persistence:

- [ ] Define `Loader` interface
- [ ] Implement `DBLoader`
- [ ] Implement `MemoryLoader`

Syncing:

- [ ] Implement `SyncReceiver`

Release:

- [ ] Do an end-to-end manual test with sandbox account
- [ ] Enable the feature for CF
- [ ] Do an end-to-end test with Changing Faces
- [ ] Enable the feature for Dental Suite
- [ ] Craft & send a release notice to selected canaries

### Milestones

> Insert a list of time-bound milestones here
> *TBD in discussion*

### Future work

> Insert a list of items that will need to be completed in future
> *TBD in discussion*

## Deliberation

* When is the best time to discuss this document with team?

### Discussion

To be answered after discussion:

* Are there any points that team members disagree with?
* Are there any additions that team members suggested?

### Open questions

* Is it worth syncing more than we need to avoid complexity of partial objects?
* How many months of data should we sync on first/full sync?

To be answered after discussion:

* Are there any questions the team does not know the answer to?

## Notes

We need to sync these objects:

* Patients
* Payments
* Accounts
* Invoices
* Invoice Items
* Treatments
* Fees
* Payment plans
* Treatment plans
* Treatment plan items
* Practitioners
* Users
* Appointments
* Treatment appointments

Of these, we can query these by date:

* Patients
* Payments
* Invoices
* Treatment plans
* Appointments

We can consider these to be "root" objects that we can use to sync other objects:

* Accounts can be queried by patient ID
* Invoice items can be queried by invoice ID
* Treatment appointments can be queried by patient ID or treatment plan ID
* Treatment plan items - we *may* sync these by date but if we have support for syncing by `treatment_plan_id`, we may as well do that so we don't have a bunch of orphaned treatment plan items

Of the above, these are added less often, have a lower footprint and can be synced say once per day:

* Treatments
* Fees
* Payment plans
* Users
* Practitioners - we *may* sync these by date but better of syncing all at once on some interval

We have 3 groupings of extractor:

1. **Time-based extractor** which extracts objects that have changed since a given date
2. **Dependency extractor** which extracts objects based on a set of other object IDs
3. **All extractor** which extracts all objects at once every time

This means that we can't have an all encompassing loader. Sets of objects will be loaded at very different times. Though, it could accept a partial set of object streams.

Each extractor should return a dictionary that is compatible with the shape the `Loader` expects, that is a mapping from keys (e.g `"patients"`) to a stream of objects.

This means we can adapt the current `Extractor`/`APIExtractor` to an `TimeBasedExtractor`/`APITimeBasedExtractor`.

We can define a `DependencyExtractor` that accepts a dict of keys to streams of IDs and returns a dict from keys to streams of objects.

The `AllExtractor` would accept nothing and return a dict from keys to streams of objects.