# IPNI Spring 2025 Emergency Operational Stability Work

## How did we get here?
IPNI has been in a 'keep the lights on' mode for the last 2.5 years. In the final year of bedrock, the team was steered away from work on the service. In the last 1.5 years, IPNI received an operational blue fund grant from PLFIF to cover capital costs of migrating from the cloud to cheaper servers, and for the ongoing operational cost of that operation - it did not include funding for time to develop the software stack - the agreement by IPNI was to "keep the service running *as is*."

While this strategy has been sufficient for 2024, we have noticed significant degredation over the last months. This is due to a combination of several factors. We also are in a position with limited operational actions that can be taken to restore service quickly.

* In March, a degraded disk was identified on the Berlin instance. The disk was replaced, but the decision was made at that point to flush the ZFS pool and re-ingest on the node. The ZFS healing of the data on the failing disk was not guaranteed, and the index on that machine had become bloated. One backlog software item is to support more effective cleanup of data from removed providers, but in the interim the operational strategy is to periodically task one instance with re-indexing.

* In the same period, the access pattern has changed significantly. This has demanded constant tunning of the existing KV store which was originally designed for a different traffic pattern. Specifically:
* with the growth of Spark suage, IPNI now has an access pattern with regular bursts of requests for un-cached CIDs. The configuration of caching and the index has not had time to attempt to optimize for this usage pattern.

* Evolution of IPFS delegated routing and the retirement of large gateway caches has increased the IPFS load on IPNI.

* An increase in providers who want low-latency IPFS-style publishing (many small advertisements) versus batched filecoin updates (fewer advertisements each representing ~32gb of data), has resulted in re-ingestion taking much longer. Following advertisement chains are fetched one-by-one, so when each advertisment has only a few CIDs but are very deep we're seeing this process able to stretch on for days or weeks to synchronize one of these providers.

## What do we do about it?

### Hardware refinement
In the next two weeks, we will:
1. upgrade the cpu of our "2.5th / backup instance" into a full 3rd instance.
2. upgrade the memory in the 2nd instance to relieve pressure observed bottlenecking current database compaction.

### Provider Ingest Prioritization

In the period from April 11th to April 19th, the secondary Indexer was stuck synchronizing 50 providers (the number of worker threads). Once these threads are allocated to providers with low network bandwidth or very long chains, the ingest process becomes rate limited by that network ingest speed. We see that the the server spent much of the past week at <20% utilization, waiting on receiving lists of CIDs from these providers.

We can get the replica up to full-performance much faster by prioritizing providers with high bandwidth to better approximate a 'fair quality of service' between providers.
We propose to:
1. update indexStar (the routing / translation server) to count which providers are being returned, so that the ordered list can be used for prioritization
2. Babysitting the Berlin instance re-ingestion to prioritize used providers to be able to switch to it once we can estimate it is at least 75% caught up.

### Investigation of mirroring failure
One technique IPNI uses with an aim to speed up the process of re-indexing is to save advertisements seen by providers locally. When re-ingesting, these already seen chains can be read from disk, rather than over the network. As flagged above, this is not fully working. We see on the current re-ingest that this local store of advertisements has filled up 32TB of allocated space. While we have sufficient cache space between our instance, the operational work currently has each node use it's local pool for this cache, rather than having the instances share a pool. We need to next research if
1. there is a path to efficiently segment or otherwise share the pool between instances
2. we can expand the size of the pool in at least one of the instances to be large enough for the full cache and have all instances use that cache
3. there are ways to use the pool more efficiently.

We believe that changes here will not be able to meaningfully improve the speed for resolving our current degredation, but is one of the factors leading to the current extended outage, and so a path to mitigation should be identified.

### 'Double Cursor Ingest'

The most accessible additional capability we wish we had is for more flexibility in ingest.

Ideally, new advertisements can be followed from startup, and separate from backlog ingest of historic advertisements.

We see many queries focusing on 'recent' content. We believe that by having a new instance prioritizing content it sees published since startup, it can be used for offloading of queries, and thus reduce load on other instances within hours to days of startup.

We estimate this at 2-3 weeks of engineering work.

## What do we need?

### Structure that supports prioritization of IPNI

We cannot easily pay ourselves for our time, or prioritize needed work in the current structure. Unless things are visibly broken as they are now, our team members are dis-incentivized from spending time on IPNI instead of their day jobs.

Breaking this down:
1. IPNI Does not have a budget to contract dedicated time for handling community requests, keeping up with changing workloads, software engineering, or much else
2. There is some (~$30k) of money remaining in the IPNI entity that we should have used for emergency contracting of time to mitigate these issues when they started last month. This has been hindered by:
a. We were told that any use of these remaining funds would put the OIF bank account below it's minimum and we needed to figure out how to dissolve the entity, or wait for more money after requesting funding. this was the first time we learned of this additional 15% overhead on the allocated funds.
b. PLFIF has proposed using the current ProPGF structure, but that will take at least a couple months to get a funding refresh to allow us to negotiate a changed structure.