# Warg Federation Design Proposal

## Motivations
- **Privacy** of network activity;
- **Availability** through mirroring, caching and replication;
- **Data locality** by having the registry API and content local within a datacenter or network;
- **Access control** policies for private registries;

A team wants to use components from the registry in their deployed service, but does not want to depend on availability from any upstream registry in their deploy process. They want a registry that implements a "read-through cache", attempting to sync from upstream on each fetch but serving the latest cached log if that fails. To enable this use case, the registry protocol needs to support mirroring individual package logs from an upstream registry into a local registry.

## Intended Use Cases

As there will not be a single, central registry, these potential patterns may emerge from federation capabilties.

#### Personal Registry
Developers can have a personal registry (with access controls) for publishing their own packages and mirrored dependencies.

#### Company and Team Registries
A team or company may have their own registry and access controls. The registry controls which packages (and versions) are imported from other public and private registries. The registry content may be mirrored and distributed on the company networks or datacenters.

#### Home Registry
Developers use the CLI to connect to their default "home" registry. The home registry may import (and optionally mirror) packages from other public and private registries. This registry may be shared or personal to the developer.

#### Platform Deployment Registries
Developers publish to their home registry (or company or team registry) the packages for deployment. The platform registry authenticates and imports the developer packages into the platform registry for deployment.

The platform registry may control and audit the packages (and versions) available as package import dependencies.

#### Low-Throughput Registries
A registry can be the source of truth for its packages but not handle high traffic load. Highly scalable mirrors can distribute its packages without sacrificing verifiability.

#### Local Mirrors
To reduce network costs and improve performance, registries can be a local mirror for a datacenter or network.

#### Registry Monitors
Both registries that import packages from other registries as well as specialized monitors of other registries can audit and verify other registries' state.

## Federation Implementation

## Proposed API Changes
If the `X-Proxy-Registry` HTTP header is provided in the request, then the registry will interpret that header value as the registry domain name and respond as if it proxies the request to that federated registry.

#### `POST /v1/namespaces`
Client requests `{"namespaces": [string]}` the registry domain name that is the source for each namespace. The server responds with an error for unknown namespaces. If all namespaces are known by the registry, then the server responds with:
`{"{namespace}": null | {"namespace": "{originNamespace}", "registry": "{registryDomain}", "registryContext": null | "{registryDomain}", "proxy": boolean}}`

`null` for the namespace indicates that it is not federated and is sourced from the current registry.

#### `GET /v1/package/{logId}/record/{recordId}`
This endpoint is for polling the record status of an attempt to publish a package record.

`published` object is simplified and does not return the content sources download URLs. That functionality is replaced by the next endpoint.

#### `GET /v1/content/{digest}` *NEW*
For a given content digest, returns content sources download URLs. Similar to `published` state on the previous Get Package Record endpoint.

#### `GET /v1/fetch/ledger/{startingIndex}` *NEW*
A fetch endpoint for [[`registryIndex`, `logId`, `recordId`]].

#### `POST /v1/verify/checkpoint` *NEW*
Request body is a signed timestamped checkpoint envelope. Response is success status code or error status code with JSON body. The HTTP header for the registry domain is the context on what registry the checkpoint is for.

## Alternative Proposed API Changes
#### `POST /v1/fetch/logs`
**Request:** No changes. Client does not need to know whether packages are federated or not to request package logs.

**Response:**
No changes, if no federated packages are returned.

If federated packages are returned, each federated package envelope adds 2 fields:
- `federated`: `string` federated registry ID;
- `federatedLogId`: `string(AnyHash)` `logId` of the federated registry (which may be different than how it is mapped in this registry);

Also, a `federatedCheckpoints` field is included as an `object` with `string` keys of the federated registry ID and the values `TimestampedCheckpoint` envelopes signed with the operator key of this registry (not the federated registry). These federated checkpoints correspond to the checkpoint `logLength` of the parent registry.

#### `POST /v1/proof/consistency`
For federated packages, the request adds a `federated` field of an `object` with `string` keys of the federated registry ID and values `{"from": integer, "to": integer}` for the log length `from` and `to` between federated checkpoints. The response will include the `federated` proof bundles.

#### `POST /v1/proof/inclusion`
For federated packages, the request adds a `federated` field of an `object` with `string` keys of the federated registry ID and values `{"logLength": integer, "leafs": [integer]}` for the log length and leaf indexes of each federated package. The response will include the `federated` proof bundles.

#### `GET /v1/package/{logId}/record/{recordId}`
This endpoint is intended only for non-federated package records, polling for status of an attempt to publish a package record.

`published` object is simplified and does not return the content sources download URLs. That functionality is replaced by the next endpoint.

#### `GET /v1/content/{digest}` *NEW*
For a given content digest, returns content sources download URLs. Similar to `published` state on the previous Get Package Record endpoint.

The content `digest` may be for federated or non-federated package content.

#### `GET /v1/fetch/ledger/{startingIndex}` *NEW*
A fetch endpoint for [[`registryIndex`, `logId`, `recordId`]].