# where is that CAR parked?

## TLDR; by @alanshaw

Problems:

1. We're missing a mapping of CAR CID -> bucket location.
3. We don't have bucket events in Cloudflare, so need the client to tell us when it has uploaded something.
4. We don't want clients to create location claims for internal bucket URLs because we might change the location in the future (reputation hit for client).
5. We want freeway code to be usable in Saturn nodes, so ideally it uses only content claims to discover locations.

Proposed solution:

* Simulate bucket event by having client submit `store/confirm` invocation.
* Capability name TBC!
* Confirms successful transfer of data.
* Effect linked by `store/add`.
* Create CAR CID -> bucket mapping table (or repurpose old elastic ipfs `blocks-cars-position`).
* `store/confirm` handler writes to the table.
* Materialize location claims on demand for CAR CID from data in table.
* Short lived so we can change location and not have to revoke 1,000's of UCANs.

> location claims claim that a CID is available at a URL.

w3s read interfaces currently rely on hard assumption that data sits on a given bucket. Freeway and Roundabout assume data is available at R2 bucket `carpark-prod-0`, while Hoverboard relies on a table (from E-IPFS) with CARs locations in S3 to attempt to try to find a car in R2 bucket `carpark-prod-0`.

Location claims look like the perfect fit for storing this information. However, we rapidly found a few challenges assuming location claims issued by the client could encode this information:
- client does not know the internals
- we could go into deriving on the client based on the write URL, or even write the write URL (that would not be readable)
- client has no control over the target and if data is migrated to a different target, potentially resulting in reputation damage
- claims could be to private buckets that are not accessible from anyone

Therefore, clients should only write claims that are clear to them like, a location claim for `w3link`.

But `w3link` and other read interfaces would not know where a CAR is parked in a multiple bucket CAR... This way, building an index of CAR CIDs to URLs actually looks more like a problem of w3s than the user. w3s read interfaces need a "private location claims" service where they can find where a CAR is stored.

Being a responsability of w3s would put bucket events as the clear path to run some code to write this mapping. However, this path is not compatible to write to R2 today, but also we would not link it to the `store/add` that triggered this.

We still need to perform HTTP Request(s) to finalize storing some content. Adding an extra invocation is actually not problematic as we could just batch them all in one CAR. Consequently, we could have a `store/added` capability that could replace the bucket event. This MAY be an effect from `store/add` that SHOULD run after file is written into the bucket. This invocation could then be executed together with `filecoin/offer`, `ipni/publish`, etc.

When moving into Saturn being able to know where CARs are parked, we could open this service for certain actors who will serve content from these buckets. Or get materialized claims, where we could even replace URLs by roundabout to decrease operations costs.

TLDR we need a **CARWHERE** service where we keep internal mappings of CAR CIDs to URLs. This will enable our read interfaces to be compatible with multiple buckets by not requiring clients to write location claims for internal buckets, but requiring w3s to create this mapping.

---
(old archive - may still be relevant for formats)

# Location claims URIs former proposal

## Goal

Formalize location claims URIs for content hosted in web3.storage owned buckets to optimize readability and costs within our infrastructure for the potential use cases.

Note that it is not in scope of this document to define how a client will get to know the locations to claim. This has also its own challenges, like should we want to infer it from presigned URL? should we return on `store/add` receipt the locations where it will be written to?

## Background

[Location claims](https://hackmd.io/IiKMDqoaSM61TjybSxwHog?view#Location-Claims) claim that a given CID can be retrieved from a given location URI. Currently, web3.storage operates a few buckets on several providers/locations. In addition, users could also produce claims with third party buckets not provisioned and controlled by web3.storage.

Multiple claims can be made for the same content. Therefore, it is worth mentioning that a claim may not be readable from all actors, as some may be behind a given set of permissions/capabilities. Different locations/URIs may provide have readability limitations, different rate limits, or optimized egress costs for certain scenarios.

## Read Use cases

A large number of use cases may exist, here we will be focusing on just a subset to start.

### CF Edge Gateway

CF Edge Gateway (aka w3link) is a CF worker running within CF datacenters that acts as a IPFS Gateway. Under the hood, it relies on Freeway. When a given root CID is requested from w3link, w3link SHOULD have preference to read CAR files directly from locations within CF datacenters (i.e. R2 backed locations). This will not only be faster, but will also not incur into additional costs reading the data at rest (compared to if that data needs to be read from a S3 bucket for example).

#### Requirements

- be able to identify a claim as backed by R2
- not highly rate limited
- have bucket name in the claim to rely on workers client to read data
- have bucket object key in the claim

### Roundabout

Roundabout is a serverless function that serves CAR files (only) by their CAR CID (or PieceCID). Main motivation for Roundabout is egress cost optimizations for data at rest on buckets owned by web3.storage. It serves CAR files also without being heavily rate limited like the public R2 Object URLs.
Aiming to decrease overall system egress costs, this endpoint creates a presigned URL for the requested content and redirects to it.

#### Requirements

- be able to identify a claim as backed by R2
- have bucket name in the claim to be able to create presigned URL
- have bucket object key in the claim
- have bucket secrets to be able to create presigned URL (in Roundabout env)

Note that Roundabout is currently used for internal pipelines, like for Filecoin SPs to ingest data at rest to be aggregated and commited. If at some point web3.storage does not own buckets where data is written, it will be deprecated given that is its sole purpose. In the meantime, can also be an opportunity to use within other use cases like Saturn Nodes reads and Station.

### Saturn Nodes

Saturn Nodes may have data at rest on their own network. When they would rely on content claims to know where to find requested CARs, they SHOULD prioritize reading from their network. However, if they do not have the data, they will need to have a fallback. Either fetch it from somewhere else and proxy the data, or redirect to a different location. Redirecting to a different location may be trickier, given that location may not be accessible. Saturn nodes could also rely on Roundabout to get the requested CARs

#### Requirements

- be able to identify a claim backed by a Saturn node
- not highly rate limited
- (if fallback as Roundabout - perhaps with a custom claims reader client)
- be able identify that requested content is in a bucket owned by web3.storage?

### Station Nodes

At some point we MAY want to rely on Station Nodes to do computing. For instance, when we want to validate user offered Filecoin Piece really corresponds to the content, having multiple Station Nodes to download the content and compute a PieceCID would require Station nodes to fetch the data. Relying on content claims to get to know claimed locations for the data will be critical.

Performing reads for computations may have different requirements, but ideally they should have minimal egress costs for the data at rest. Being able to rely on roundabout would make it cheaper and more decentralized, compared to make it mandatory to run code in the provider where data is at rest.

## Bucket objects Location URIs

Defining how the clients will write claims for these target locations is critical to have a mapping of these locations to the buckets to fulfill all requirements above.

Typically, objects in S3 buckets can be located via following URIs:
- S3 URI (e.g. `s3://<BUCKET_NAME>/<CID>/<CID>.car`)
- Object URL (e.g. `https://<BUCKET_NAME>.s3.<AWS_REGION>.amazonaws.com/<CID>/<CID>.car`)
- can be used to fetch the bytes by any HTTP client if bucket is public

However, R2 object locations have different patterns, instead of following S3 pattern. They can be:
- Public Object URL for Dev (e.g. `https://pub-<INTERNAL_R2_BUCKET_IDENTIFIER>.r2.dev/<CID>/<CID>.car`)
- can be used to fetch the bytes by any HTTP client, if bucket is public and heavily rate limited
- [R2 docs](https://developers.cloudflare.com/r2/buckets/public-buckets/#enable-managed-public-access) state that such URLs should only be used for dev!
- Custom domain object URL (e.g. `https://<CUSTOM_DOMAIN>.web3.storage/<CID>/<CID>.car`)
- can be used to fetch the bytes by any HTTP client, if custom domain is configured in R2
- can be rate limited by operator configuration
- account will need to pay for the egress of reading from the bucket
- Presigned URL (e.g. `https://<ACCOUNT_ID>.r2.cloudflarestorage.com/<BUCKET_NAME>/<CID>/<CID>.car?Signature=...&Expires=...`)
- can be used to fetch the bytes by any HTTP client, if has the signature query parameter and is not expired
- no heavy rate limits in place, together with no egress costs to read data at rest

Note that a claim may not be readable from all actors, as some may be behind a given set of permissions/capabilities.

### URIs comparison

The main pattern that we can identify is to have URLs that can be accessed by any HTTP client. Except for S3 URIs and given the correct setup/keys is available, all other URLs are fetch'able. Therefore, we can assume as an advantage that claim is directly fetchable without any pre-knowledge.

Having a claim that cannot be used by every potential client (i.e. needs some extra permissions) is also a disadvantage that may represent penalties in a reputation system. Moreover, rate limits can have a negative impact on reputation as well.

URIs that include all necessary information to enable derivation for smart clients that could rely on Worker R2 Bindings or to generate Presigned URLs are critical for several use cases. URIs that minimize Egress costs can also be preferred by smart clients.

Per the above, considering S3, it looks like we should rely on S3 Object URL (e.g. `https://<BUCKET_NAME>.s3.<AWS_REGION>.amazonaws.com/<CID>/<CID>.car`).

But with R2, there is no perfect fit. The only good option would be the format used by presigned URLs, but it should only be created on request given their expiration. The custom domain offers better retrievability than the Public Object URL for Dev, but has not enough information encoded for smart clients (i.e. no way to know bucket name). For R2, we will likely need to add 2 location URIs:
- Custom domain object URL (e.g. `https://<CUSTOM_DOMAIN>.web3.storage/<CID>/<CID>.car`)
- can be used out of the box to fetch content
- Presigned URL like URI without query params (e.g. `https://<ACCOUNT_ID>.r2.cloudflarestorage.com/<BUCKET_NAME>/<CID>/<CID>`)
- won't really work, but smart clients can see if it is available and rely on its encoded info to use CF Worker Bindings / create presigned URLs, etc.

Alternatively, we can require `<CUSTOM_DOMAIN>` to become bucket name, in order to make it work as a single location URI. Main disadvantage besides the extra requirement, is that there is no url mention of being a R2 bucket, which would mean hard coded assumptions. We could also consider to mimic S3 URL for R2 here as well.

The Public Object URL for Dev should not be adopted, as it is heavily rate limited, we do not know what CF may do with it in the future, and also does not even have information about the Bucket name.

## Client location claims proposal

Nothing prevents us from claiming multiple location URIs for a given content, however we may need to also be careful on having multiple claims for the same location as if it is not fully available it MAY be ranked badly in whatever reputation system we may create. However, some smart clients MAY benefit of cost savings or faster retrievals if they have extra information encoded in the URI.

In conclusion, this document proposes that w3up clients, once they successfully perform an upload, they create location claims within following formats:
- S3
- `https://<BUCKET_NAME>.s3.<AWS_REGION>.amazonaws.com/<CID>/<CID>.car`
- R2
- `https://<CUSTOM_DOMAIN>.web3.storage/<CID>/<CID>.car`
- `https://<ACCOUNT_ID>.r2.cloudflarestorage.com/<BUCKET_NAME>/<CID>/<CID>`

Note that it is not in scope of this document to define how a client will get to know the locations to claim. This has also its own challenges, like should we want to infer it from presigned URL? should we return on `store/add` receipt the locations where it will be written to?