FHIR Async for CRUDS Interactions & Operations

Proposal: Expand FHIR Async to support generic CRUDS interactions and operations, in addition to Bulk Data Export. This can be done by defining a Bundle-based output, in addition to the Bulk Data Manifest output.

Today FHIR's http://hl7.org/fhir/async.html defines a request pattern designed for interactions that:

• Take a long time to perform
• Produce a large volume of data

The general interaction pattern is:

1. Kick off a request
2. Receive a "status" endpoint and
3. Poll the status endpoint until the request has completed
4. On success, receive a Bulk Data Manifest with links to FHIR NDJSON files (* these are suitable for large, unordered piles of data)

Why doesn't this work for CRUDS interactions and many operations? Bulk Data Manifests are designed to support a bulk $export scenario, or other interactions that produce a big pile of FHIR resources that clients need to copy wholesale. But other use cases for async aren't a great fit for a Bulk Data Manifest. We want to independently track response status codes, location headers, ETags, resource content, and errors associated with a CRUDS interaction or operation.

Use Cases

• Consumer devices submit sensor data as Observations, asynchronously. Allows the server to process incoming results asynchronoulsy, in a queue.
  • Why not a Bulk Manifest? Because we want to explicitly track a small amount of data, conveying details about the result including a status code, location headers, ETags, and an individual OperationOutcome or server-modified Observation (e.g., with elements added/stripped by the server's business logic)
• Trading parters submit data in transaction Bundles, asynchronously. Allows the server to manage expensive (and often network intensive) validation requests in the background.
  • Why not a bulk manifest? Same reasons as above – for each Bundle entry we want to convey a set of status data that FHIR Bundles model very nicely, and
• Long-running operations. For example, re-indexing a server based on a new search parameter, or executing a risk assessment algorithm against data where significant processing time is required. The result may be as simple as a RiskAssessment or Parameters resource; FHIR Operations describe the inputs and outputs effectively; but while FHIR'S Operations framework points to the async page, there is no current approach to convey a single output.
  • Why not a bulk data manifest? We only need a single Parameters resource as the operation output (or a single RiskAssessment resource, etc).
• (etc, add use cases here!)

Async handling for all FHIR CRUDS interactions and operations

We want a way to take any FHIR CRUDS interaction or operation and apply asynchronous handling. We already have most of the building blocks we need, in the current FHIR Async spec.

In any async interaction, requests for the status endpoint are a way of monitoring progress; an error (e.g., 5xx error) in response to the polling request should not indicate that the underlying CRUDS interaction has failed. It just indicates that polling has failed. We maintain a bright line here: the only way for a client to learn the result of the CRUDS interaction is to evaluate the eventual response once polling has completed.

FHIR's existing async pattern gives us almost everything we need; the only tweak we need to the existing async pattern is, instead of defaulting to a Bulk Data Manifest at the end of the process, we can instead use a batch-response Bundle containing one single entry; this allows us to represent status codes, locations, etgags, and resource content associated with the CRUDS interaction result. The client looks at Bundle.entry[0].response to understand the outcome. (Note that for searches, this does eventually produce a search type Bundle inside the batch-response Bundle's .entry[0].response. We don't love Bundles-in-Bundles, but sometimes they happen ;-))

For example, a successful create may eventually lead to:

GET /path/to/status/123
Accept: application/fhir+json

200 OK 
Content-type application/fhir+json

{
  "resourceType": "Bundle",
  "type": "batch-response",
  "entry": [{
    "response": {
       // could say "403"
       // if the client is unauthorized, etc, etc,
       "status": "200 OK", 
       "location": "Observation/123"
       // additional fields if needed
       //  * etag
       //  * lastModified
       //  * outcome
    },
    "resource": {
      // populated whenever "normal" (synchronous) CRUDS
          // interaction would have included a resource in the
      // response body -- e.g. if client initially specified
      // `Prefer: return=representation`
      "resourceType": "Observation",
      "id": "123", 
      // ... snipped for brevity
    }
  }]
}

What about $export, and other operations and searches that return large piles of data?

This proposal would not replace the Bulk Data Manifest format, because it's useful in many cases where a client really does want an unordered pile of resources as the output – and because we don't want to break backwards compatibility with existing $export operations. For $export, the Bulk Data Manifest would continue to be the default output format; and we could document a common request parameter like _asyncOutputFormat=BulkDataManifest for searches or custom operations where a client needs to explicitly opt into the Bulk Data Manifest output format on a case-by-case basis.