# BPA - Thoughts on Multi Tenancy

This document tries to explain what the BPA is and what multi tenancy concepts could be applied.

Terms used:

**DID** - Public identity, not to be confused with ephemeral did's like peer did's.
**Wallet** - Stores public/private key pairs that belong to the DID, received credentials and optionally references to credential and presentation exchanges and is maintained by aca-py. Currently aca-py only supports **one public did per wallet!** This could change with the did:indy method and potentially with askar, but it is out of scope here.
**BPA** - (Business Partner Agent) Acronym that means a piece of software that wraps other software (aca-py, controller, UI, database) under one umbrella.
**Tenant** - Logical grouping of identities, users and their data.

*(Important! Multi Tenancy should not be confused with scalability, and scalability concepts like horizontal scaling should still be supported by this work.)*

# BPA

A single BPA instance consists of four components:

1. Controller UI – Vue.js
2. Controller Backend - Micronaut/Java
3. aca-py – Python
4. Database - Postgresql

Other components like an (indy) ledger are needed to use anoncreds with aca-py, but are shared between instances.


# Multi Tenancy Scenarios

## Option 1: Handled by an outside component (Agency)

The BPA stays as is and spinning up another instance is handled by an K8s controller (Agency).


### Handling of DID's

One Tenant - One Wallet. If another DID is needed another instance needs to be set up.

left to right direction
actor Tenant
node BPA
node Wallet
Tenant -- BPA : 1:1
BPA -- Wallet : 1:1

### Pros and Cons

**Pro:**
- No refactoring needed
- Can be implememted rather quickly
- K8s features can be used by the Agency
- Easy to allocate resources (billing)
- Instances are fully separated
- Settings are fully customizable

**Con:**
- Eats more resources (also see motivators below)
- Dependency on K8s or similar technologies
- Too much overhead if clients need to share the same configuration anyway

## Option 2 A: Make the BPA an agency

Here another controller backend is added to the BPA that handles aca-py and backend orchestration.


### Handling of DID's

One Tenant - One Wallet. If another DID is needed another instance needs to be set up.

left to right direction
actor Tenant
node BPA
node Wallet
Tenant -- BPA : 1:1
BPA -- Wallet : 1:1

### Pros and Cons

**Pro:**
- More efficent handling of cluster resources
- Multi DID support light, one per tenant, but at least on the same instance
- Works in scenarios where all tenants need to share the same configuration

**Con:**
- Needs a lot of refactoring (see below)
- Makes option 1 obsolete as a agency would deploy another agency
- All sub wallets share the same configuration, no customization
- Makes it hard to commit more resources to a single tenant, or throttle a tenant that behaves badly

## Option 2 B: One tenant can handle multiple wallets

Here the logic to set up sub wallets is integrated directly into the existing code base.


### Handling of DID's

One tenant multiple wallets. DID creation happens directly in the existing frontend and not by another component/service.

left to right direction
actor Tennant
node BPA
node Wallet1
node WalletN
Tennant -- BPA : 1:1
BPA -- Wallet1 : 1:n
BPA .. WalletN : 1:n

### Pros and Cons

**Pro:**
- Multiple did's and their wallets are available under one UI/API
- Integrates well with Option 1

**Con:**
- Combining this with Option 2A will be a nightmare

## Logging into a tenant

From the user’s perspective everything stays as it is now, and this still means that DID’s and their wallets are completely separated and require separate logins. Selecting the tenant during login can happen for example through:

- subdomain per tenant
- some information that is stored in a JWT token
- id that is stored locally in the BPA's user table.
- some other gateway that adds additional information to the request e.g., HTTP header, cookie etc.

## Motivators for option 2A

### Money

The number of deployment units is not a huge issue when a BPA operator runs a Kubernetes cluster. But the most expensive and limiting factor might be memory/storage consumption if instances are scaled to the thousands. If we assume that an average BPA instance requires 1GB of memory, then running 1000 instance would require 1TB of memory. On AWS this could equal a single **x2gd.16xlarge** worker node that is somewhere in the 2500 USD/month area. If you add gateway, storage, and backup costs to this we can roughly estimate 5 to 6$ per instance. If this is cheap or expensive depends on the pricing model. If the BPA should be offered for free these are extensive costs.

### Complexity

This is a two-edged sword. On the one hand having a more traditional setup gives BPA Operators more deployment options. For example, one could also run a single instance on a single server and then add features like load balancing and clustering later. But on the other hand, this creates new issues when a single tenant uses more resources than another, or a single service component runs out of resources. So, in the end I would argue that complexity is not a good motivator to start this work as it only shifts problems.

To summarize. The main motivator for starting this work might be to reduce deployment units and, hence reducing costs.

# If we do option 2A

The rest of the document only relates to option 2. Option 1 would work very differently and is TBD.

## Refactoring tasks by component

As the BPA is an amalgam of multiple individual services, multi tenancy concepts need to be considered on the component level first and then in a second step one needs to think about how the concepts can work together.

### Controller

List of possible controller changes:

- aca-py authentication method is changed, Java wrapper already supports this
- Authenticating user needs to be mapped to an instance. UserDetailsMapper and/or AuthenticationProvider
- Decide on a database sharding mechanism, see next section
- API: User Authentication object part of all API method signatures
- Decide if this should be completely standalone or if parts of the instance initialization can be handled by an outside component. As aca-py already has a provisioning step setting up base and sub wallets could be handled by the component that handles wallet provisioning.
- aca-py Java client Bean needs to be in session or request scope because the Bearer Token is different for each user
- Webhook handler needs to use the x-wallet-id header to map events to instance id.
- Instance provisioning API needed either in the controller or someplace else as we need to save some info like instanceId, x-wallet-id, BearerToken
- Different startup tasks needed for this mode
- Check all caches if they need to be partitioned by instance id
- Disable bootstrap un/pw, needs to go through registration step
- What about the web mode?

### User Interface

- separate tenant management page needed
- own user and role

### Postgres

Generally, there are three concepts that come to mind on how to shard a database:

1. Own database per instance. Pro: This is transparent to the backend. Con: Creation must be handled by an outside component and databases cannot be switched on the fly per default and requires an extra driver layer to handle this. When done improperly this can turn ugly fast and database maintenance becomes complex as well as flyway is pretty much disabled.
2. Own schema per instance. Pros and cons are the same as above.
3. Shard or instance Id per instance (composite key). Pro: database maintenance mechanisms stay the same. Con: This is not transparent to the backend as the database requires the id to be present in all tables. Also changes in a lot of method signatures become necessary.

### aca-py

Aca-py is already multi-tenant ready

https://github.com/hyperledger/aries-cloudagent-python/blob/main/Multitenancy.md

#### Changes in aca-py

Wallet types:

*Base Wallet*

Can do nothing but create other wallets. Authenticate with --admin-api-key

*Sub Wallet*

Same as before, but different authentication with Bearer Token

#### aca-py flags for multi tenancy

--multitenant
--jwt-secret <jwt-secret>
--multitenant-admin

#### Create Sub Wallet

##### In base wallet

POST /multitenancy/wallet

{
  "key_management_mode": "managed",
  "label": "Alice",
  "wallet_dispatch_type": "default",
  "wallet_key": "MySecretKey123",
  "wallet_name": "MyNewWallet",
  "wallet_type": "indy",
  "wallet_webhook_urls": [
    "http://localhost:8080/log"
  ]
}

Returns bearer token for sub wallet.

##### In sub wallet

Starts completely empty, hence the following steps need to be orchestrated:

- change authentication method to Bearer token with admin key
- create private did
- make private did public
- register endpoints, either self by attaining endorser role, or via another endorser

## Where to put orchestration?

Right now, the whole BPA stack is self-contained, pre-configured, most of the configuration cannot be changed during runtime and software is started in dependency order, meaning database - aca-py - controller. With multi tenancy there needs to be a way to trigger sub wallet creation for a new tenant and some mapping between tenant id and wallet id. This logic can either be put into another orchestration service that in a second step can inject this information into the controller, or directly into the controller.

### Orchestration steps in the controller

#### Dev or single tenant mode - basically what we have now

- aca-py is started empty
- aca-py client is running in admin mode
- The controller automatically creates the first sub wallet (via the provisioning api) and, the default user is mapped to tenant id 1

#### Multi tenant mode

- aca-py is started empty
- aca-py client is running in admin mode
- The BPA is started without any running start up tasks, and no default user is created, it is not possible to login to a tenant
- provision rest api is called either by another controller or by a admin using the tenant management UI. Provisiong a sub wallet triggers:
- create sub wallet with aca-py
- stores/creates mappings, returns instance id or provisions with provided ids
- triggers other startup tasks if needed e.g. did creation and ledger endpoint registration. Tenant controller can do API calls on behalf of the sub wallet as the Bearer token is known.
- optional: create database user in BPA DB e.g. if Keycloak is not configured
- (The BPA could also be configured with a set of instance ids and the backend would create sub wallets (default users) for all those ids)

### Tenant Provisioning

POST /api/tenant-admin/provision

{
  "description": "Optional description: My BPA",
  "seed": "Optional seed: 0000000000000000",
  "isEndorser": true
}

If isEndorser is set to true two things can happen, first if testnet the DID is registered via the configured ledger explorer (same as now), or second the DID already needs to be registered on the ledger. If set to false we either add the functionality that all ledger writes need to go through another endorser, or we disable functionality like schema and credential definition creation.