The discussion of iceberg rust development.

People

• Jan Kaul
• Renjie Liu
• Xuanwo
• Fokko Driesprong

Time

2023-07-19 16:00 ~ 17:00 (UTC+8)

Unresolved issues

• Project location
  • A new repo (like apache/iceberg-rust)
  • A new folder (like rust under apache/iceberg)
  • Wait for disscussion on email list. (both looks good to us)
• Some rust releated setup
  • rust edition (2021?)
  • rust version (stable?)
• The roadmap of v0.1 release?
  • specs
  • arrow interop
  • compilation to wasm/wasi (optional)
• Relation between projects
  • icelake will be the user of iceberg-rust
• Catalog support:
  • Nessie?
  • databend catalog?
  • risingwave catalog?
  • Maybe we should provide a public trait instead?
• Object storage
  • OpenDAL
    • OpenDAL can have try on WASM.
• IceLake design
  • Split in_memory structs and on_disk formats
    • We have internal structs that close to the on_disk formats which is eaiser to maintain.
    • Users will use helper functions like {parse/format}_table_metadata to parse/format in_memory structs into Vec<u8> (a.k.a Bytes).
    • Users have TryFrom/TryInfo impl for our in_memory structs from/to arrow and avro.
  • in_memory structs will not have dynamic values like serde_json::Value, so users don't need to handle them by hand.
• Roadmap at iceberg repo