• This doc has notes from previous OpenDAL discussions and agendas for upcoming syncs.
• Content here will be concluded to the maillist after each sync.
• Please add agenda items that you’d like to discuss.

Useful Links

• Google Meet Links: https://meet.google.com/xvv-vcny-jxn
• OpenDAL Community: https://opendal.apache.org/community/

---

Upcoming Sync

---

Previous Sync Notes

2024-05-31

Releases

v0.47 is about to come.

2024-05-10

Releases

v0.46 has been released.

2024-04-19

Highlights

• OpenDAL oio Refactor
• Upcoming v0.46.0 release

Releases

No release yet.

Discussion

No discussion yet.

2024-03-29

Highlights

• New oio API
• ofs progress
• GSoC Event
• OSPP Event

Releases

No release yet.

Discussion

2024-03-08

Highlights

• Happy new year!

Releases

v0.45.1

The first release for all packages.

Discussion

• All in async trait

from

trait opendal::raw::oio::Read {
  fn poll_read(&mut self, cx: Context<'_>,bs: &mut [u8]) -> Poll<Result<usize>>;
}

into

trait opendal::raw::oio::Read {
  async read(&mut self, bs: &mut [u8]) -> Result<usize>;
}

trait opendal::raw::oio::ReadDyn {
  fn read_dyn(&mut self, bs: &mut [u8]) -> Pin<Box<dyn Future<Output=Result<usize>>>>
}

2024-01-26

Highlights

• New Committer, Wenkang Xu @WenyXu.
• Apache OpenDAL Graduated
• New PMC Member, Liuqing Yue @dqhl76
• New Committer, Yang Shuai @hoslo

Releases

v0.44.2

• New Cloud Service Support: Added support for pCloud, Yandex Disk, Koofr, and Apple iCloud Drive, enhancing the range of cloud storage options available.
• Concurrent Writing Enhancements: Implemented various features for concurrent writing and associated behavior tests, significantly improving concurrent data processing capabilities.
• HDFS and WebHDFS Improvements: Introduced atomic write features and multi-write via CONCAT for HDFS, increasing reliability and efficiency in HDFS operations.
• GDrive and S3 Updates: Implemented path caching for GDrive and introduced a trash mechanism for file deletion. Improved error handling in S3 when credentials are empty, enhancing user experience and error management.
• Bug Fixes and Refactoring: Addressed various bugs and refactored code across several services, including fixing issues in RangeWriter and BlockWrite, and reorganizing test layouts.
• Documentation and CI Updates: Improved documentation wording, updated copyright and trademark information, and made adjustments in CI tests and dependency updates, ensuring better clarity and smoother development workflow.

Discussion

• Looking for GSoC 2024 Project Ideas
  • ofs (Need an RFC)
  • SDPK blobstore

2024-01-05

Highlights

• New Committer: Congyi
• Graduation Started!
• New sub-project: ofs
• New Services: pcloud (Thanks @hoslo)
• Happy New Year!
• New RFC: Concurrent Write

Releases

v0.44.0

• New Services: seafile (Thanks @hoslo)
• New Services: upyun (Thanks @hoslo)
• New RFC: Buffered Reader
• OpenDAL is available on WASM32
  • s3
  • azfile
  • gdrive
  • gcs

v0.44.1

• ASF branding changes

Discussion

• New RFC: Deleter API

2023-12-15

Highlights

• New PPMC Member Xiangdong (@G-XD)
• Huggingface File System Support
• ConcurrentFutures to remove the depends on tokio
• Almost all graduation preparations have been completed.
  • Dependencies Analysis (thanks @Xuanwo)
  • Maturity Model (thanks @suyanhanx)
• Versioned Documents with Nightlies

Releases

• V0.43
  • Upgrade to v0.43
    • Public API
      • List Recursive
      • List Prefix -> changed the behavior of stat a dir path
    • Raw API
      • Lister Align
  • New Features
    • Enhancements in language bindings
      • retry layer, capability support and read/write stream for nodejs (thanks @suyanhanx)
    • Extended support for new services and storage solutions
      • alluxio and b2 support (thanks @hoslo)
  • Changes
    • Refactoring in testing and workflow planning
    • Refactor the testing workflow for 7 services.
  • Documentation
    • Add basic usage for java binding

Discussion

• [DISCUSS] How about changing OpenDAL Community Sync time?
• [DISCUSS] Graduate OpenDAL as TLP?
• [RFC] Buffered Reader

2023-11-24

Highlights

• OpenStack Swift support (Thanks @morristai)
• Alluxio rest API support (Thanks @hoslo)
• B2 support (Thanks @hoslo)
• Add File support for python binding (Thanks @xuanwo)
• Service Config
• New User!
  • questdb: An open source time-series database for fast ingest and SQL queries
• RFC: Concurrent Stat in List (Thanks @morristai)
• Change List API into poll based and return one entry instead (Thanks @xuanwo)
• Benchmark vs s3 & vs fs
• Capability and layer support for nodejs binding (Thanks @suyan)

Releases

• v0.42
  • New Features
    • Enhancements in language bindings
      • including new behavior testing, async operations support, and renaming capabilities.
      • Examples include feat(binding/python): Support AsyncOperator tests and feat(binding/java): add rename support.
    • Extended support for new services and storage solutions
      • feat(service/libsql): support libsql
      • feat(service/azfile): add azure file service support.
    • Introduction of new monitoring and metrics features for Prometheus, like
      • feat(prometheus): add bytes metrics as counter
      • feat(layer/prometheus): Support custom metric bucket for Histogram.
  • Changes
    • Refactoring in testing and workflow planning
      • Introduce OpenDAL Workflow Planner
      • refactor(services/gdrive): Extract folder search logic.
    • Enhancements in code cleanliness and consistency, like
      • refactor(bindings/C): Implement error with error message
      • refactor(core): use list_with in Operator::list.
  • Fixes:
    • Bug fixes ensuring stability and reliability, like
      • fix: Fix read result not full
      • fix(services/s3): Accept List responses without ETag.
  • Documentation:
    • Addition of new documentation and updates for better understanding and usage of features, such as
      • docs: Add docs in website for sqlite/mysql/postgresql services and docs: update release doc.

Discussion

• List Prefix Progress (from @xuanwo)
  • For example, we can speed up the list(“aaa”) for this dir.
    • aaa1
    • aaa2
    • aaa3
    • bbb1
    • bbb2
• OpenDAL benchmark vs AWS SDK s3
  • Which language that AWS SDK s3 written in?
    • Rust

2023-11-03

Highlights

• New Committers @manjusaka!
• Milvus’s OpenDAL C binding integration has been merged (Thanks @jiaoew and @yah01)
  • https://github.com/milvus-io/milvus/pull/25642
• New Test Framework
  • Core’s New Test Framework
  • Java’s New Test Framework (Thanks @tison and @G-XD)
  • Python’s New Test Framework (Thanks @manjusaka)
  • Nodejs’s New Test Framework (Thanks @eryue0220)
• Core
  • Lazy Reader support (Thanks @Xuanwo)
• C binding
  • add reader support (Thanks @jiaoew)
  • add error message support (Thanks @jixinyou)
  • API refactor (remove the blokcing prefix) (Thanks @jixinyou)

Releases

• No release so far.
  • We have a release plan in next week

Discussion

• OpenDAL Incubator Report - November 2023
  • https://lists.apache.org/thread/6c4fd4bxh7bpokf8d8h3r7l6dhk0lcmv
• Prepare for graduation
  • OpenDAL RFC Policy
    • https://lists.apache.org/thread/60580mcwtzjxo8bcbd56j0kql0k3cm8t
  • OpenDAL Release Policy
    • https://lists.apache.org/thread/fyd2fz39w8ygc5rqkw1olhytw016scp5
• When will we make a formal C binding release? (from @jiaoew)
  • We need to address some issues first:
    • Add test framework for C binding
      • https://github.com/apache/incubator-opendal/issues/3472
    • Add API Docs and Guide for C binding

2023-10-13

Highlights

• New Committer G-XD!
• New Services!
  • mysql
  • sqlite
  • d1
  • libsql
• New Test Framework!
  • java
  • python
• New Workflow Planner! (@xuanwo)
• New Users!
  • pants
  • dalfs
  • manas
  • terraphim-ai

Releases

• V0.41
  • Rust Core
    • Add PrometheusClientLayer (thanks @flaneur2020)
    • Support set offset in ghac write and other polish (thanks @Huon Wilson)
    • Add mysql support (thanks @Zheaoli)
    • Add postgresql connection pool support (thanks @Zheaoli)
    • Add sqlite support (thanks @Zheaoli)
  • Java Binding
    • New Test Framework (thanks @G-XD)
  • oay
    • Extract dav-server-fs-opendal (thanks @Young-Flash)

Discussion

• 2023-09-22

Highlights

• OwO #1
• OpenDAL v0.40 has been released

Releases

• v0.40
  • Rust Core
    • API Cleanup
      • Merge append into write
      • Add lister and list to replace list
      • Merge range_read into read
      • oio::Write::copy_from removed
    • New Features
      • Buffered Write
  • Java Binding
    • Release setup
    • Blocking layer enabled
  • Cpp Binding
    • Release setup
  • Oay
    • Webdav supported

Discussion

• Are we ready to graduate from the incubator?
  • https://incubator.apache.org/guides/graduation.html
• Move root from services config to operator?
  • https://github.com/apache/incubator-opendal/issues/3151
• Expose services options so that users can working on struct?
  • https://github.com/apache/incubator-opendal/issues/3152

2023-09-01

Highlights

• The first testing sync happened!
• Gdrive test has been setup (Thanks, suyan!)
• Oay’s webdav test has been passed (Thanks, young-flash!)
• Java binding release workflow (Thanks, tison!)
• Cpp binding is started (Thanks, silver!)
• New Users
  • FileCodeBox (python binding)
  • Milvus (c binding)

Releases

Discussion

• OpenDAL new release workflow
  • https://opendal.apache.org/docs/contributing/release
• OpenDAL v0.40 Release Plan
• Some crazy ideas
  • OpenDAL FS
  • OpenDAL REST API
  • OpenDAL Cache