Aries Cloudagent Python: simplifying the endorser and revocation workflows

Background

Aries Cloudagent Python recently introduced the ability for non-privileged agents (from here on: author) to request endorsement of transactions that need to be written to the ledger by an agent (from here on: endorser) with ledger write privileges.

A common use case for this functionality is for an organization running an agency of authors (that can be created/destroyed without needing any write interaction with the ledger) and a single instance of an endorser agent that acts as a gateway for all write needs:

• registering the author's DID on the ledger
• endorse writing a new schema to the ledger
• endorse writing a new credential definition to the ledger
• endorse writing a new revocation registry definition to the ledger
  • this could be an ongoing task as revocation registries get full
• endorse writing revocation registry entries to the ledger
  • there are two types of actions that potentially need supporting: contextual write (when the entry is created) and deferred write (e.g.: bulk publishing of all revocation entries once daily)

While there may be scenarios that require the controller(s) to orchestrate the endorsing protocol on both the author and endorser end, it is much simpler (and therefore more likely to be adopted by users) to allow authors and endorser to be setup to complete the endorsement protocol steps transparently, just by setting up appropriate configuration when starting the agents.

An example of how complex/involved endorsing can become can be seen in this "credential revocation with endorsement" BDD integration test for ACA-Py.

Proposed Solution

Global Settings

Similarly to other settings in ACA-Py, there should be a few new global settings/startup parameters.

For the endorser protocol:

• --auto-endorse-transactions (defaults to false): this setting would be used on the endorser agent, and would force every endorsement request to be automatically endorsed, without emitting a webhook and having to manage state on a controller.
• --endorser-public-did or --endorser-alias: these settings would be mutually exclusive, and would be used by the agent to identify which connection to use when requesting endorsement for transactions. While alias is more human-readable, if the connection with the endorser was established using its public did specifying its public did as startup parameter would allow the agent to NOT be restarted to apply the changes.
• --endorser-protocol-role: either endorser or author, it would define the role of the agent when requesting endorsement or endorsing transactions .

When using the above settings, the REST APIs exposing endorsement functionality should be turned to read-only mode to avoid unintentionally altering the globally defined settings. In case of catastrophic failures, the agent(s) would need to be manually restarted, in which case the startup parameters could be temporarily altered to allow for manual intervention - if necessary.

For revocation:

• --explicit-revocation-registry-handling (defaults to false): similarly to what described above, handling of revocation should be transparent to admin API consumers unless explicitly requested.

Similarly to endorsement, the purpose of this flag is to prevent unintentional modification of globally defined settings, and could be used to turn those endpoints to "write" mode in scenarios where manual intervention is required.

Functional Changes: Endorsement

The endorser protocol requires some additional metadata on the connection record used by the endorser and the author:

• Agent role: needs to be set on both agents (endorser and author) for each connection record. It will be inferred by the --endorser-protocol-role startup parameter and it will be applied to all connections (this makes the reasonable assumption that an agent will always be an author OR an endorser).
• Connection to the endorser (on the author agent): rather than setting this metadata item manually, it will be inferred by searching the connection record using --endorser-public-did or --endorser-alias. Note: since a connection will NOT exist the first time the author agent starts up, the connection lookup should be performed every time a request that requires endorsing is initiated.
• Auto endorsing requests: as described above, on the endorser agent based on the value of --auto-endorse-transactions
• Request endorsement: authors should automatically handle requesting endorsement for any object that requires a ledger write. The API consumers will - as an example - POST a schema to the ledger and the agent would take care to perform all of the transaction preparation steps, waiting for endorsement and writing the transaction before returning a result.

Functional Changes: Revocation

Unless specified by setting the --explicit-revocation-registry-handling to true, handling of the revocation registry should be transparent to the admin API consumers. When using an unprivileged author agent, this means that every transaction requiring a ledger write will need to go through an endorser. For the next points the assumption that the changes to the endorser protocol described above have been implemented is being made.

This means that:

• when a new credential definition is created, the agent will:
  • automatically handle the endorser protocol execution for the credential definition
  • automatically handle the endorser protocol execution for the revocation registry definition
  • automatically create a tails file and set its state to active
  • automatically upload the tails file to the specified tails server URL
  • automatically handle the endorser protocol execution for publishing the initial revocation registry entry
• automatically handle the endorser protocol transaction for publishing revocation registry entries once a credential has been revoked
  • to be defined whether with endorsement enabled publishing of a revocation registry entry should be a one-step by specifying publish: true in the payload, or it will be enforced to call the publish endpoint every time instead
• the agent will take care of rotating revocation registries as they fill-up:
  • a second, inactive revocation registry should be available at any time (creating a revocation registry is an expensive task)
  • when the registry in use is full, the agent will de-activate it and activate the next available registry
  • the previous tasks follow the same pattern as when a revocation registry is initially created, and rely on the appropriate transactions to be endorsed and executed

Implementation Guidelines

Changes are required in several areas of ACA-Py and the current underlying implementation patterns may be different from case to case.

Rather than focusing on implementation details, the following common sense recommendations should be considered:

• avoid code duplication: extract any duplicated code into reusable utilities as much as possible
• avoid embedding logic in the route handler, if possible: route handlers should only be responsible for preparing the received payload, waiting for a response and submitting it. Dedicated handler classes should be used where possible

Implementation Details

Implementation will use the aca-py event bus.

When a transaction is created, it will contain meta-data about what to do after the transaction is endorsed and completed (i.e. written to the ledger).

For example:

{
  "state": "transaction_created",
  "messages_attach": [ ... ],
  "formats": [ ... ],
  "thread_id": "f601f40f-c7e6-4a17-8888-643c1718427a",
  "transaction_id": "a721ddc1-0f42-45ac-9899-abea30f587d7",
  "meta_data": {
    "params": { ... },
    "events": [
      {
        "topic": " ... ",
        "payload": ...
      }
    ]
  }
}

This meta-data will:

• Provide a mechanism to pass parameters to functions that will get executed in the future (for example passing the revocation_registry_size, specified when creating a cred def, to the revocation registry code).
• Specify what kind of event will get emitted once the transaction gets endorsed/completed (reducing the need fot the existing custom code).

Once the transaction is completed, the TransactionManager will use the transaction meta-data to emit any events if necessary. The respective managers (SchemaManager, CredDefManager, RevocationManager, etc) must subscribe to the appropriate events on startup, and provide functions to handle each type of event.

For example, for creating a new Schema the flow would look as follows:

The "non-endorser" scenario would be re-factored as follows to use the events: