# FHIR Async for CRUDS Interactions & Operations

### Proposal: Expand FHIR Async to support generic CRUDS interactions and operations, in addition to Bulk Data Export. This can be done by defining a Bundle-based output, in addition to the Bulk Data Manifest output.

Today FHIR's http://hl7.org/fhir/async.html defines a request pattern designed for interactions that:

* Take a long time to perform
* Produce a large volume of data

The general interaction pattern is:

1. Kick off a request
2. Receive a "status" endpoint and
3. Poll the status endpoint until the request has completed
4. On success, receive a Bulk Data Manifest with links to FHIR NDJSON files (* these are suitable for large, unordered piles of data)

#### Why doesn't this work for CRUDS interactions and many operations? Bulk Data Manifests are designed to support a bulk `$export` scenario, or other interactions that produce a big pile of FHIR resources that clients need to copy wholesale. But other use cases for async aren't a great fit for a Bulk Data Manifest. We want to independently track response status codes, location headers, ETags, resource content, and errors associated with a CRUDS interaction or operation.

#### Use Cases

* Consumer devices submit sensor data as `Observation`s, asynchronously. Allows the server to process incoming results asynchronoulsy, in a queue.
* *Why not a Bulk Manifest?* Because we want to explicitly track a small amount of data, conveying details about the *result* including a status code, location headers, ETags, and an individual OperationOutcome or server-modified Observation (e.g., with elements added/stripped by the server's business logic)
* Trading parters submit data in transaction Bundles, asynchronously. Allows the server to manage expensive (and often network intensive) validation requests in the background.
* *Why not a bulk manifest?* Same reasons as above -- for each Bundle entry we want to convey a set of status data that FHIR Bundles model very nicely, and
* Long-running operations. For example, re-indexing a server based on a new search parameter, or executing a risk assessment algorithm against data where significant processing time is required. The result may be as simple as a RiskAssessment or Parameters resource; FHIR Operations describe the inputs and outputs effectively; but while FHIR'S Operations framework [points to](http://build.fhir.org/operations.html#asynchronous) the async page, there is no current approach to convey a single output.
* *Why not a bulk data manifest?* We only need a single `Parameters` resource as the operation output (or a single `RiskAssessment` resource, etc).
* (etc, add use cases here!)

#### Async handling for all FHIR CRUDS interactions and operations
We want a way to take *any* FHIR CRUDS interaction or operation and apply asynchronous handling. We already have most of the building blocks we need, in the current FHIR Async spec.

In any async interaction, requests for the *status endpoint* are a way of monitoring progress; an error (e.g., 5xx error) in response to the *polling request* should not indicate that the underlying CRUDS interaction has failed. It just indicates that polling has failed. We maintain a bright line here: the only way for a client to learn the result of the CRUDS interaction is to evaluate the *eventual response* once polling has completed.

FHIR's existing async pattern gives us almost everything we need; the only tweak we need to the existing async pattern is, instead of defaulting to a Bulk Data Manifest at the end of the process, we can instead use a `batch-response` Bundle containing one single entry; this allows us to represent status codes, locations, etgags, and resource content associated with the CRUDS interaction result. The client looks at `Bundle.entry[0].response` to understand the outcome. (Note that for searches, this does eventually produce a `search` type Bundle inside the `batch-response` Bundle's `.entry[0].response`. We don't love Bundles-in-Bundles, but sometimes they happen ;-))

 sequenceDiagram
    Client->>Server: POST /Observation<br/>Prefer: respond-async
    Server->>Client: 202 Accepted <br/>Content-Location: /path/to/status/123
    Client->>Server: GET /path/to/status/123
    Server->>Client: 202 Accepted <br/>Retry-After: 30
    Note over Client: Waits patiently for thirty seconds
    Note over Server: Server finishes the 'create'<br/>Prepares 'batch-response' Bundle with one entry
    Client->>Server: GET /path/to/status/123
    Server->>Client: 200 OK <br/>{"resourceType": "Bundle", "type": "batch-response", ...}

For example, a successful create may eventually lead to:

GET /path/to/status/123
Accept: application/fhir+json

200 OK 
Content-type application/fhir+json

{
  "resourceType": "Bundle",
  "type": "batch-response",
  "entry": [{
    "response": {
       // could say "403"
       // if the client is unauthorized, etc, etc,
       "status": "200 OK", 
       "location": "Observation/123"
       // additional fields if needed
       //  * etag
       //  * lastModified
       //  * outcome
    },
    "resource": {
      // populated whenever "normal" (synchronous) CRUDS
          // interaction would have included a resource in the
      // response body -- e.g. if client initially specified
      // `Prefer: return=representation`
      "resourceType": "Observation",
      "id": "123", 
      // ... snipped for brevity
    }
  }]
}

#### What about `$export`, and other operations and searches that return large piles of data?

This proposal would not replace the Bulk Data Manifest format, because it's useful in many cases where a client really does want an unordered pile of resources as the output -- and because we don't want to break backwards compatibility with existing `$export` operations. For `$export`, the Bulk Data Manifest would continue to be the default output format; and we could document a common request parameter like `_asyncOutputFormat=BulkDataManifest` for searches or custom operations where a client needs to explicitly opt into the Bulk Data Manifest output format on a case-by-case basis.