The discussion of iceberg rust development.

People

  • Jan Kaul
  • Renjie Liu
  • Xuanwo
  • Fokko Driesprong

Time

2023-07-19 16:00 ~ 17:00 (UTC+8)

Unresolved issues

  • Project location
    • A new repo (like apache/iceberg-rust)
    • A new folder (like rust under apache/iceberg)
    • Wait for disscussion on email list. (both looks good to us)
  • Some rust releated setup
    • rust edition (2021?)
    • rust version (stable?)
  • The roadmap of v0.1 release?
    • specs
    • arrow interop
    • compilation to wasm/wasi (optional)
  • Relation between projects
    • icelake will be the user of iceberg-rust
  • Catalog support:
    • Nessie?
    • databend catalog?
    • risingwave catalog?
    • Maybe we should provide a public trait instead?
  • Object storage
    • OpenDAL
      • OpenDAL can have try on WASM.
  • IceLake design
    • Split in_memory structs and on_disk formats
      • We have internal structs that close to the on_disk formats which is eaiser to maintain.
      • Users will use helper functions like {parse/format}_table_metadata to parse/format in_memory structs into Vec<u8> (a.k.a Bytes).
      • Users have TryFrom/TryInfo impl for our in_memory structs from/to arrow and avro.
    • in_memory structs will not have dynamic values like serde_json::Value, so users don't need to handle them by hand.
  • Roadmap at iceberg repo
Select a repo