DAG House – w3up Phase 3

# DAG House – w3up Phase 3 **5 weeks, ending Mar 10** 🔴 P0 🟡 P1 🟢 P2 ## 🔴 w3access + w3provider + w3session - [ ] Finish implementing [w3provider](https://github.com/web3-storage/specs/blob/main/w3-provider.md) (MVP free tier provider) - [ ] [w3protocol#347](https://github.com/web3-storage/w3protocol/issues/347) Email template improvements - @travis, @alanshaw - [ ] [w3protocol#348](https://github.com/web3-storage/w3protocol/issues/348) GET -> POST on email validation links - @travis, @alanshaw - [ ] [w3protocol#341](https://github.com/web3-storage/w3protocol/issues/341) Postmark daghouse domain move to web3.storage - @travis, @alanshaw - [ ] Finish implementing [w3access](https://github.com/web3-storage/specs/blob/main/w3-access.md) - (Sharing delegations across agents/devices) - [ ] [w3protocol#393](https://github.com/web3-storage/w3protocol/issues/393) `access/delegate` - @bengo, @alanshaw - [ ] [w3protocol#394](https://github.com/web3-storage/w3protocol/issues/394) `access/claim` - @bengo, @alanshaw - [ ] [w3protocol#395](https://github.com/web3-storage/w3protocol/issues/395) (w3access & w3session in client) - @gozala, @alanshaw - [ ] Finish implementing [w3session](https://github.com/web3-storage/specs/blob/main/w3-session.md) @gozala - (Verification via email and did:mailto usage) - [ ] [w3protocol#293](https://github.com/web3-storage/w3protocol/issues/293) Fix the delegation over websocket slowness - @travis, @alanshaw - [ ] [w3protocol#384](https://github.com/web3-storage/w3protocol/issues/384) Finish single URL for web3.storage Ucanto - @bengo ## 🔴 Turn off clusters - [ ] [backup#1](https://github.com/web3-storage/backup/issues/1) Migrate data - [x] make (new) list of CIDs to migrate for web3.storage – @alanshaw - [ ] complete web3.storage cluster migration - @francisco - [ ] complete nft.storage cluster migration - @francisco - [ ] [pickup#68](https://github.com/web3-storage/pickup/issues/68) Rollout pickup - [ ] migrate web3.storage pinning services to pickup (current: 50% traffic) - @nearform - [ ] migrate nft.storage pinning services to pickup (current: 0% traffic) - @nearform - [ ] update nft.storage to use pickup - @olizilla - [ ] Inform netops when safe to decommission ### Notes Aiming for end of Feb completion. ## 🟡 w3s.link + nftstorage.link gateways - [ ] remove upstream CF gateway (save $) - @olizilla + @vasco-santos - [ ] remove upstream pinata gateway (save $) - @olizilla + @vasco-santos - [ ] redirect back to dweb.link for content not available in freeway^ - @olizilla + @vasco-santos - [ ] Q: Can we turn on gzip and save some bandwidth? - @olizilla + @vasco-santos ### Notes This would reduce proxied traffic significantly... CF is charging us for egress of content we're not storing. ## 🟡 Elastic IPFS costs - [ ] Verify blake2b fix is cause of increased egress - @alanshaw - [ ] Research solutions for reducing costs - @vasco-santos ### Notes Seeing sustained increase since jan - egress costs - dyanmo costs from indexing per block ## 🟡 w3up Metrics - [ ] UCAN LOG Stream Kinesis gateway for other services (access-api, ...) - @vasco-santos - [ ] integrate Redshift as Kinesis consumer - @vasco-santos - [ ] Kinesis consumers for tracking - @vasco-santos: - [ ] store/add count - [ ] store/add size - [ ] space count - [ ] individual space store/add count - [ ] individual space store/add size - [ ] Integrate Prometheus like API + Grafana - @vasco-santos - [ ] decide what Grafana instance to use - [ ] usage over time?? - [ ] Figure out how to extract NFT.Storage specific metrics ### Notes Docs: * https://www.notion.so/DS9-Q1-metrics-goals-41b617591d8e4f7fb5bae94f655d6c82 * [2023-02-02 meeting notes](https://www.notion.so/2023-02-02-metrics-MVP-sync-3b939213b45046368980a9d8dd7cb7bc) * [Draft of product requirements](https://www.notion.so/UCAN-metrics-Requirements-Document-c4abe1951076459284af3928de4a6800) Critical Metrics: - Total uploads - Upload data volume - Data in deals in filecoin ## 🟡 Specifications - [ ] Tweaks on specification for [store/upload protocol](https://github.com/web3-storage/specs/blob/main/w3-store.md) @gozala - Reads like blog/how-to - needs more formal - [ ] Specify expected invocation responses ## 🟢 w3ui - [ ] [w3ui#142](https://github.com/web3-storage/w3ui/issues/142) w3console design - @munns @jbrawny - [ ] [w3ui#378](https://github.com/web3-storage/w3ui/issues/378) upload progress reporting (fix upload-client, w3up-client, headless components) - @travis ## 🟢 w3filecoin - [ ] [w3protocol#353](https://github.com/web3-storage/w3protocol/issues/353) CommP in client - @gozala @vasco-santos - [ ] Consumer stack - add CAR CIDs to w3filecoin pipe after store/add [w3filecoin#12](https://github.com/web3-storage/w3filecoin/issues/12) - @vasco-santos - [ ] Wire dead letter queues [w3filecoin#10](https://github.com/web3-storage/w3filecoin/issues/10) - @vasco-santos ### Notes Filecoin integration is a big missing piece! These tasks are **Blocked on spade**: - spade write integration - spade deal success check / retry? - (not) spade read integration - find deals my CID is in. - metrics - filecoin aggregates count - filecoin aggregates size ## 🟢 Checkup - [ ] [checkup#3](https://github.com/nftstorage/checkup/issues/3) Update checkup to consider data in new S3 buckets - @alanshaw @vasco-santos --- # A future phase! ## Receipts (deprioritized) We are invested in this. It's WIP. We shold land it. it's: - on critical path for billing. - on critical for metrics (currently, we could calculate in other way) _what is the minimal receipts flow?_ we need to tightly define what success of this endevour looks like - finish off work to land it ucanto - do work work to integrate it into store/add and s3 ObjectCreated ## w3provider (deprioritized) - [ ] [w3protocol#349](https://github.com/web3-storage/w3protocol/issues/349) `provider/get` `consumer/add` - [ ] [w3protocol#350](https://github.com/web3-storage/w3protocol/issues/350) `provider/add` ## Others - Publicly verifiable timestamps for UCAN invocations [web3-storage/specs#27](https://github.com/web3-storage/specs/pull/27) - Add UnixFS directory sharding [w3up-client#54](https://github.com/web3-storage/w3up-client/issues/54) - Add support for HAMT directories [ipld/js-unixfs#37](https://github.com/ipld/js-unixfs/issues/37) - CAR retrieval by CAR CID [w3protocol#274](https://github.com/web3-storage/w3protocol/issues/274) ## Notes ^gateway: total cost of serving old content may be higher if we bounce the user to dweb.link and it then has to fetch it from e-ipfs, but we should just go for it and monitor. if freeway doesn't have it then currently we pay twice as ipfs.io serves it to w3s.link. migration off old api: ??? ### Tech debt - validate CIDs for blocks in CARs - previously cluster was our validator. we don't have a replacement yet. ### Questions what's costing us the most right now? - CDN data transfer in Cloudflare (not clear, but everything that leaves CF's network.)