# Data anywhere

> In a world where `web3.storage` fully relies on [content claims](https://github.com/web3-storage/specs/pull/86), user agents SHOULD compute and claim indexes for the data they upload during the "upload" phase, while the multiple read interfaces provided by web3.storage SHOULD also rely on this claims to serve content.

## Why

With this project we aim to:
- not compute and claim indexes on behalf of users
- not be dependent on SPECIFIC write targets that trigger events on write to perform additional work
- not be tied to specific bucket names
- writing directly to R2, removing the duplicate copy in S3
- improve performance of first read

## What

We are running a [content claims service](https://github.com/web3-storage/content-claims) already, but we did not fully embrace it in [w3up](https://github.com/web3-storage/w3up) implementation yet, nor in the multiple read interfaces of web3.storage.

On the client side, [w3up client](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client) SHOULD be responsible for issuing the necessary claims, so that the data can be retrieved according to the user needs, as well as invocations to offer Pieces for Filecoin storage. Today these happen mostly via handlers called from bucket events. The service MAY perform computations to also issue claims for given indexes, as well as Filecoin piece computation for validation.

Moving to a paradigm where content can be writen anywhere obviously requires read interfaces to have additional logic to know where this content lives, instead of how it operates today, where it either assumes a well known bucket name, or relies on the old E-IPFS dynamo table (and manually swaps to CF based endpoint first!). For this, we will be relying on an approach with [Content Claims]((https://github.com/web3-storage/specs/pull/86)) + [carwhere](https://hackmd.io/2dmtDIwvTAivXgtHIH7K5w) (probably rename to datawherehouse).

[w3up](https://github.com/web3-storage/w3up) `store/add` handler provides a presigned URL for a bucket in S3 (`carpark-prod-0`). For cost reasons, web3.storage replicates all content into a R2 bucket (`carpark-prod-0`) to benefit from cheaper egress. However, we end up also paying for double storage, as well as for 1x egress of all data received. Writing to S3 first was a result of lack of features in R2 at the time (like bucket events), together with its initial stage in life. This replication also degraded the first read times, given content could still not be replicated before the first read happens.

## Who

- @vasco-santos
- @alanshaw ?
- @gammazero ?
- @bengo ?

## Plan

### Blockers/uncertainty
* w3up client refactor work in progress
* [IPNI UCAN service](https://github.com/web3-storage/w3up/issues/1210) current state unknown

### Writes pipeline
- [ ] @vasco-santos Promote `storage/deliver` and `storage/confirm` [RFC](https://github.com/web3-storage/RFC/pull/10) to spec
- [ ] @vasco-santos Add `storage/deliver` and `storage/confirm` handlers to `upload-api` https://github.com/web3-storage/w3up/pull/1310
- [ ] @vasco-santos Add `storage/deliver` and `storage/confirm` capabilities in client
- [ ] @vasco-santos Add [datawherehouse claims](https://hackmd.io/2dmtDIwvTAivXgtHIH7K5w) (probably rename to datawherehouse) on the `upload-api` new `storage/*` handlers
- [ ] @vasco-santos Add handler function to validate Pieces computed offered by client
- [ ] @vasco-santos Invoke `filecoin/offer` in Client sunsetting the Bucket event
- [ ] @vasco-santos Invoke `assert/equals` in Client with PieceCID equivalent to CarCID instead of Bucket event
- [ ] Invoke `assert/partition` in Client to finalize upload flow instead of being part of `upload/add` handler
- [ ] Land `ipni/*` spec https://github.com/web3-storage/specs/pull/85
- [ ] Implement `ipni/*` service
- [ ] [Invoke `ipni/offer` in client](https://github.com/web3-storage/w3up/issues/1210) / `assert/inclusion` may be encoded in it (should this be opt in?)
- [ ] Swap presigned URL from S3 to R2 on the `store/add` handler
- [ ] @vasco-santos Setup S3 lifecycle rule to expire content from S3 `carpark-prod-0` bucket
- [ ] P2 - datawherehouse historical backfill OR simply fallback to old bucket name for now

### Reads pipeline
- [ ] materialized claims
- [ ] @vasco-santos roundabout relies on content claims + datawherehouse to find where to redirect to https://github.com/web3-storage/w3infra/pull/313
- [ ] @bengo hoverboard relies on content claims to read data https://github.com/web3-storage/hoverboard/pull/20
- [ ] needs to integrate with datawherehouse?
- [ ] freeway relies on datawherehouse
- [ ] P2 - [Produce content claims from read pipeline](https://github.com/web3-storage/w3up/issues/1206)

## Old documents

- [decouple w3filecoin aggregation pipeline trigger](https://hackmd.io/@vasco-santos/BkXqWGNY6)
- [where is that CAR parked?](https://hackmd.io/@vasco-santos/BkKLtOHtp/edit)
- [carpark upgrade plan](https://hackmd.io/4eccTSSgS6Gosp8OAoZ1Vw)

# Syncs

## 7th March

### Goals

- [align on location claims vs datawherehouse](https://github.com/web3-storage/RFC/pull/13)
- [align invocations for delivery/confirm](https://github.com/web3-storage/RFC/pull/10)