# Elastic Provider Events

We need visibility into the progress/status of items as they are ingested into EP and provided to the indexer nodes.

There's a few applications for this:

* Metrics (monitoring and alerting).
* Insights - need to know when a complete DAG has been received.
* Pinning Service API - need to know when something is "Pinned".
* End users?

EP architecture: https://www.notion.so/AWS-Elastic-Provider-Architecture-a722abda1ad4447da4886abe934454f5

Good idea: https://www.notion.so/CAR-Events-Service-4df71335f0354e36afa260eb3e9fc917?v=258030573c3d45d3beea722ff44d1dca

## MVP

At minimum, in order to get EP into production we _should_ be able to monitor and alert on the time it takes for an uploaded CAR to become available on the Gateways. This is the time it takes for an uploaded CAR to be read, indexed and recorded in the DynamoDB.

* 2 events
1. CAR file received: `S3 URL, size, timestamp`
3. CAR file fully indexed into EP: `S3 URL, timestamp`
* Metrics collector
* Listens to events
* Publishes a Prometheus scraping endpoint
* Histogram total CARs, total time by buckets 0MB -> 100MB

### Design Proposal

(WIP) - @bengo

Metrics Collector
* new logical component
* receives events from ipfs-elastic-provider (et al?) by configuring the Metrics Collector as an ipfs-elastic-provider eventTarget (see 'Indexer Lambda' section below)
* forwards events to any other event monitoring things
* e.g. prometheus pushgateway?
* keeps track of stats about CARs/CIDs using Cloudflare durable objects
* question: Does this logically live within ipfs-elastic-provider, web3.storage, nft.storage, or is it a dag.house shared service that is 'outside' all three of those?
* proposal: it seems like its either a dag.house shared service or a part of web3.storage? Because it doesn't seem like a core part of ipfs-elastic-provider if all of that is in AWS but the metrics collector is in cloudflare
* decision: What repository does the code live in?
* candidates
* new repo
* https://github.com/ipfs-elastic-provider
* https://github.com/web3-storage
* decision:
* new repo: https://github.com/ipfs-elastic-provider/metrics-collector
* decision: What deployment environment runs the code?
* candidates
* cloudflare workers
* aws lambda
* aws ec2 or similar
* proposal: It's a new cloudflare worker
* @mikeal had this in mind when @bengo asked on 2022-06-16
* questions
* Can a cloudflare worker accomplish what @alanshaw wrote of 'Publishes a Prometheus scraping endpoint'?
* Would it be ok if this service records to prometheus via pushgateway?
* decision: what libs/frameworks to use for this service?
* proposal
* @bengo
* ~~ucanto~~ (not for POC)
* candidate #5
* candidates
1. https://github.com/web3-storage/ucanto/
* @mikeal encouraged @bengo to consider this on 2022-06-16
2. cloudflare worker js
* cons: dont want to be too tightly coupled to one deployment env. Better to have small adapter code that reads cloudflare invocation and turns into ucanto invocation (or equiv)
3. node http RequestListener
* cons: tightly coupled to node.js and not web platform
4. koa/express
* cons: tightly coupled to node.js and not web platform
5. Code against service worker Request API
* pros: cloudflare encourages this already
* cons: a little overhead when running in node.js
* decision: How will it write to Durable Objects?
* e.g. what will durable object API be. Will there be a single Durable Object? One durable object per CAR?

Metrics Prometheus Scrape Worker

* handles periodic requests from prometheus
* reads data from durable object written to by 'Metrics Collector'
* exports in prometheus format
* open questions
* what data will it read from which durable object
* What will the prometheus metric structure be?
* requirements 'Histogram total CARs, total time by buckets 0MB -> 100MB'
* Histogram total CARs by buckets 0MB -> 100MB
* car_size_bytes
* histogram
* buckets - 0-100Mb stepping by 10Mb
* Total time by buckets 0MB -> 100MB
* uploaded_to_indexed_duration_seconds
* gauge
* labels
* byteLength - car file size in bytes, buckets 0-100MB by 10MB increments

Indexer Lambda
* new configuration option:
* eventTargets - locations that indexing events will be sent to for monitoring
* web3.storage ipfs-elastic-provider will be configured such that this is set to the Metrics Collector
* events will be sent via HTTPS eventually
* maybe ucanto-over-https
* future: events could be published to an SNS eventTarget
* future: configure allowlist of which events to deliver to which eventTarget. may not be needed for MVP or ever
* new function in scope during main job: `dispatchEvent`
* in practice this method will fanout events to the configured `eventTargets` via HTTP post
* `dispatchEvent(new CarFileReceivedEvent)`
* `dispatchEvent(new CarFileIndexedEvent)`
* decision: Should async event delivery block the rest of the indexing thread?
* Is it a big deal if an event cant be delivered to eventTarget?
* decision: Should event publishing go directly to Metrics Collector, or should it publish to SNS, and then a listener on that topic would forward to metrics collector?
* proposal (accepted): Just push straight to metrics collector
* based on @bengo talking with @mikeal 2022-06-16
* no need to pay for SNS
* assuming metrics collector is a cloudflare worker
* lambda -> cloudflare unlikely to get rate limited because lambda has lots of IPs

CAR File Received Event

* decision: When to dispatch this event?
* proposal:
* [~~here?~~](https://github.com/ipfs-elastic-provider/ipfs-elastic-provider-indexer-lambda/blob/dev/src/index.js#L137)
* [here](https://github.com/ipfs-elastic-provider/ipfs-elastic-provider-bucket-to-indexer/blob/main/src/index.js#L22)

CAR File Indexed Event

* decision: When to dispatch this event?
* proposal (accepted): [here](https://github.com/ipfs-elastic-provider/ipfs-elastic-provider-indexer-lambda/blob/dev/src/index.js#L202)

## Ideas for Events (Post MVP)

* CAR upload complete
* A single CAR has been uploaded to the system
* For split CARs this doesn't mean the entire DAG has been completely uploaded.
* CARv2 index written: `S3 URL, timestamp`
* We have a [lambda that does this](https://github.com/ipfs-elastic-provider/side-indexer-lambda)
* Complete DAG received: `Root CID, timestamp`
* A complete DAG may be split over multiple CARs.
* Knowing when a _complete_ DAG has been received is important!
* Need a lambda to walk the DAG to determine this - triggered by "CARv2 index written" event perhaps (or S3 PutObject)?
* Block events (might be too much!?):
* Block received: `multihash, size, S3 CAR URL, timestamp`
* Block indexed: `multihash, timestamp`
* Indexed into the EP DynamoDB
* Block advertised: `multihash, timestamp`
* CID written to advertisement in S3 bucket
* Indexer node advertisement written
* Indexer node read advertisement
* Indexer node has indexed CAR!??

---

# Other thoughts

## Upload status

Maybe with this system we can get to a stage where we can report upload status like this:

1. 🟣 Uploading.
* Upload is currently in progress.
1. 🟠 Received.
* A complete DAG has been received and is awaiting indexing into EP.
1. 🟡 Available.
* The DAG has been indexed by EP and is available via Gateways and direct connections to the EP bitswap peer.
* The indexer node advertisement has been created and is waiting for indexer node(s) to read the advertisement.
1. 🟢 Available➕Discoverable.
* One or more indexer nodes have read the advertisement for the upload.
* I have no idea how to do this right now!

---

---

---

---