# VectorETL usage

config.yml

source:
    type: S3
    aws_secret: ...
    ...

destination:
    type: Qdrant
    host: ...
    ...

embeddings:
    ...

fetch_mode: full | incremental
insert_mode: upsert | append | replace

python main.py

Running above will cause:
0. Load config
1. Create SQLite db if not already exists.
1.1. Connect to Source and fetch files
2. Check SQLite db for previous runs.
4. DELETE FROM SQLite, return ids
5. INSERT INTO SQLite, return ids
6. Chunk files.
7. Generate embeddings.
8. DELETE FROM VectorDB using ids
9. UPSERT above to VectorDB using ids.
10. Update metadata to SQLite db.

## SQLite schema

Table documents {
  id integer [primary key]
  document_path string
  file_updated_at timestamp
  file_metadata json
  created_at timestamp [default: `now()`]
  updated_at timestamp
}

Table document_chunks {
  id integer [primary key]
  document_id integer
  created_at timestamp [default: `now()`]
  updated_at timestamp
}

Ref: document_chunks.document_id > documents.id