---
# System prepended metadata

title: Trieve Infra

---

# Trieve Infra

## Order of Operations

1. ~~Qdrant in EKS~~
2. Operating procedure for scaling Qdrant
> [name=tomf-trieveai] Vertical autoscaling is not available yet
3. Operating procedure for recovering Qdrant
> [name=tomf-trieveai] Recorvery for non-sharded instances have a few methods with tradeoffs, rebuild from pg manually - worst, snapshotting api & snapshot loading from volume with entrypoint override - not much better, Listener -> disk is useful for backups where we lose all healthy nodes, can still recover from Listener node volume - currently using this pattern. Of course in typical clusterws mode nodes are able to discover each and bootstrap as long as there is at least one healthy node. 
4. ~~Prometheus connected to Qdrant scraper endpoint~~
> [name=tomf-trieveai] this is confirmed working using the serviceMonitor annotation in helm values
5. ~~Prometheus connected to Qdrant scraper endpoint5. Grafana to~~ visualize Qdrant prom 
> [name=tomf-trieveai] need to write dashboard and bring it into repo
6. ~~PR to add sentry query analytics to ingestion worker's pg queries~~ (Not query analtitics, just taken from spans) (https://sentry1.internal.trieve.ai/organizations/trieve/performance/summary/spans/calling_insert_chunk_metadata_query:5df97e477bd9adec/?project=2&query=&statsPeriod=1h&transaction=upload_chunk)
7. ~~PR to add sentry query analytics to ingestion workers Qdrant queries~~ (https://sentry1.internal.trieve.ai/organizations/trieve/performance/summary/spans/calling_create_qdrant_point:4dde64e0373325be/?project=2&query=&statsPeriod=1h&transaction=upload_chunk)
8. PR to add sentry query analytics for ingestion workers redis queries
9. Pump HN and figure out chunks/s bottleneck in worker
10. PRs to fix worker bottleneck (TBD)

-----------------------------

?. Start getting remaining services into EKS
? + 1. Plan what's needed for multi region 

## Goal 1: Get HN Demo Pumping

### Qdrant

- Established operating procedure to scale from 16gb to 256gb mem
- Place into Kube cluster 

### Monitoring

- Prometheus for Qdrant
- Prometheus for Postgres
- Grafana for visualizing Prom

### Code

- Sentry PR to add query timings for pg and Redis

## Goal 2: Get everything into Kube Service

- ingestion worker with ability to scale horizontally
- API server
- Redis
- Postgres
- TEI server (Jina, bge, nomic) 

## Goal 3: Get ingestion worker to 1k chunks/s

- Identify current bottleneck 
