# ACA-Py scalable mediation

## Assumptions

- For the given deployment, there are multiple ACA-Py pods constituting the *service agents* being dynamically scaled according to demand
- There is one *mediator agent*, which may also be constituted by multiple pods scaling according to demand
- The mediator and service agents may or may not share a message bus (redis, nats, etc.)

## Mediator setup

- The mediator agent would have its own private key, the routing key. It would not need to share a database with the other agents
- It would also have a set of connections to service agents as well as a set of public key -> connection mappings used to forward incoming messages (for DIDComm v2 the forwarding might be DID-based)
- The mediator would necessarily have a public endpoint
- It may expose a DID document using did:web for ease of service agent configuration

## Service agent setup

- Service agents would use `coordinate-mediation` to communicate with the mediator agent and establish forwarding routes. When creating DID documents for peer or public DIDs, they would insert the mediator agent's routing key and use its service endpoint(s)
- Service agents would not necessarily have a public address outside of the cluster. They may communicate with the mediator agent using a local transport (message bus)

## Message transport

The mediator agent could handle both inbound and outbound communication for the cluster. This provides for guaranteed delivery of inbound messages by queuing them for delivery to the service agents, being tolerant of pod restarts and upgrades.

Outbound messages from the service agents to external agents could be placed into a queue in a shared message bus and delivered by the mediator agent (with appropriate retry semantics, and ideally with feedback on delivery failure). If there are any externally-dispatched webhooks then these could be handled in the same way. This shortens the time required to shut down a service agent pod, and avoids dropping pending outbound messages that are still in the queue for delivery.

Bidirectional communication (sending responses on the same socket) could become difficult. The return-route transport decorator does not account for message forwarding. This means that outside agents without public endpoints would likely not be able to connect to the service agents without using their own mediator agent. DIDComm v2 in particular does not assume the presence of a bidirectional channel.

# ACA-Py Persistent Queues -- non-mediation

Here is a proposed implementation for a non-mediation scenario. In this, the Queue (Redis, currently) holds messages for processing.

Inbound requests come from external sources to the ACA-Py service, either the Controller for the service or other agents on the Internet. Those are received by the Inbound Request processor, queued, and later pulled from the queue by ACA-Py instances and processed.

Outbound requests are queued by the ACA-Py instances and are intended for delivery to the same two "external" entities -- the controller and other agents on the internet. The Outbound Request processor pulls from the queue and delivers the message to the intended recipient.

package "ACA-Py Framework\nService" as afs {
    node "ACA-Py #1" as ap1
    node "ACA-Py #2" as ap2
    node "ACA-Py #N" as apN
    node "ACA-Py KMS\nSecure Storage" as kms
}

node "Redis\nQueue Implementation" as red

package "Queue Processors" as qp {
    node "Inbound\nRequests" as qir #lightgreen
    node "Outbound\nRequests" as qor #lightgreen
}

node "Controller" as con

cloud "Internet\n" as in {
    node "Other\nAgent" as oa1
}


kms -down-> ap1
kms -down-> ap2
kms -down-> apN
ap1 <--> red
ap2 <--> red
apN <--> red
con -down-> qir
red --> qor
qor --> con
oa1 -left-> qir
qor --> oa1
qir --> red