The discussion of iceberg rust development. ## People - Jan Kaul - Renjie Liu - Xuanwo - Fokko Driesprong ## Time 2023-07-19 16:00 ~ 17:00 (UTC+8) ## Unresolved issues - Project location - A new repo (like `apache/iceberg-rust`) - A new folder (like `rust` under `apache/iceberg`) - Wait for disscussion on email list. (both looks good to us) - Some rust releated setup - rust edition (2021?) - rust version (stable?) - The roadmap of v0.1 release? - specs - arrow interop - compilation to wasm/wasi (optional) - Relation between projects - `icelake` will be the user of `iceberg-rust` - Catalog support: - Nessie? - databend catalog? - risingwave catalog? - Maybe we should provide a public trait instead? - Object storage - [OpenDAL](https://github.com/apache/incubator-opendal) - OpenDAL can have try on WASM. - IceLake design - Split `in_memory` structs and `on_disk` formats - We have internal structs that close to the `on_disk` formats which is eaiser to maintain. - Users will use helper functions like `{parse/format}_table_metadata` to parse/format `in_memory` structs into `Vec<u8>` (a.k.a Bytes). - Users have `TryFrom/TryInfo` impl for our `in_memory` structs from/to arrow and avro. - `in_memory` structs will not have dynamic values like `serde_json::Value`, so users don't need to handle them by hand. - Roadmap at iceberg repo -