> Improving the reads pipeline. Designing, implementing and deploying new services. Several re-architectures to enable cost savings across all our systems, and work to enable the integration of w3up with Filecoin, this semester had it all

# Systems shipped / in progress

## w3infra

- designed and architected w3infra with Oli relying on proposed architecture with AWS Event Bridge to act upon CAR write events
- implemented `upload-api`, Replicator Stack, Satnav Stack and DB Stack
- integration tests pattern for SST projects

## UCAN Log Stream

- proposal, research, architecture design and implementation of the central piece for introspection and async tasks of w3up platform
- collaboration with Irakli to get Receipts in Ucanto and integrated through our APIs
- important piece for upcoming UCAN Invocation spec work
- as we decentralize more and more web3.storage, this will become the central piece and brain of our platform
- already integrated with `upload-api` and `access-api`, as well as on the process of being integrated with `aggregate-api`

## w3up Metrics

- proposal, architecture design and implementation of a scalable w3up cumulative metrics system
- leveraged learnings from our old metrics system that was over and over again a problem due to its heavy queries
- leveraged some ideas on how prometheus was designed
- Dashboard is available in grafana
- easily pluggable with any other consumers to get metrics from any system
- PoC to extend this metrics system to have real time metrics relying on Kinesis Firehose, Amazon Athena, S3

## w3filecoin + Spade integration

- wrote UCAN based aggregation spec for integrating web3.storage with spade
- immplemented aggregation-api and aggregation-client
- worked together with Irakli to land support in Ucanto for effects and attach inline blocks
- designed the w3filecoin architecture
- redesigned based on previous wrong assumptions on how to calculate aggregate size. Previous data stack architecture based on filling up Aggregate ferries over time while keeping track of their size in DynamoDB would not work when we need to compute commP of commPs every time to see if CARs fit in aggregate
- started implementation
- designed the spade-proxy architecture
- started implementation already having aggregate-api in place
- in progress to integrate with spade

# Cost savings

## Reads pipeline rearchitecture

- re-arranged from a Multi-tier racing gateways into a Multi-tier resolution gateway that only leveraged Cloudflare Cache, Cloudflare R2 (via Freeway), and would redirect to PL public infrastructure if content would not be available in the previous resolution tiers
- Multi-tier racing gateways design aimed for the best performance, resilience and Cache Hit ratio possible, while new design aims to get to the lowest cost possible to operate, even if it means less resilience and performance
- Drastically reduced data transfer costs in CF (bigger cost factor)
- Decomissioned dedicated IPFS Gateways also saving us 10k$ a month

## E-IPFS

- Identified and Droped a redundant DynamoDB write per block indexed (at our scale this was a massive cost). For the January 2023 usage, this change would save us ~4k$ a month. However, for some spikes that we saw in February, huge spikes could get the cost of this redundant write to 20-50k$/month
- Implemented bitswap cancel message handling. Handling cancel messages was implemented in E-IPFS resulting in ~10% of data transfer saved at the time of implementation (~250GB Egress per day).
- DynamoDB disabled point in time recovery backups that was costing us 3.2k$/month

## Way to single Cloud provider for storage

- Migrated well over 50% of the CAR files only in S3 to R2 as part of our direction to only have one copy of the data (instead of S3 + R2). This was over 400TB guaranteeing full data integrity and that all needed indexes were created to enable Freeway to serve this content.

## Decomission Superhot API in progress

- While it is still not decomissioned (team decided to wait), we already cut over 50% of our infra spending for it.

## Roundabout

- we can get Free Egress out of Cloudflare with pre-signed URLs. Initial users will be Riba’s Dagcargo and the w3filecoin pipeline. The impact here is that previously all the uploaded content was transferred once out to put it into Filecoin Storage Providers.