Linux 核心專題: 雲端檔案系統設計和實作

執行人: RealBigMickey

Github 連結：https://github.com/RealBigMickey/DisFS

TODO: FUSE 原理和近期核心效能改善

參照 s3fs，探討其原理
研讀〈To FUSE or Not to FUSE: Performance of User-Space File Systems〉
研讀〈Extension Framework for File Systems in User space，探討如何利用 eBPF 改善 FUSE 檔案系統效能，藉由快取 inode 與 dentry，以降低對 metadata 的存取延遲
分析 FUSE Hooks Up With IO_uring For Greater Performance Potential In Linux 6.14 揭露的效能改進手法

TODO: 設計雲端檔案系統

參照 rhizofs 一類的實作。請留意：為確保專案具有更高的應用彈性，開發者在使用者層級應避免採用以 GNU GPL 授權發布的程式碼與函式庫（GNU LGPL 授權則可接受使用）

Goal: Use FUSE (Filesystem in Userspace), discord bots, SQL, and C programming to create an online storage service that's both a Filesystem & website.

For a MVP (Minimum Viable Product), I plan to implement the following commands:
You said:

• getattr
  • Used for retrieving metadata
• readdir
  • Needed for listing directory contents
• open
  • Required for reading and writing
• read
  • Essential for viewing and copying
• mkdir
  • Allowing creation of new directories
• rmdir
  • Allowing removal of empty directories
• create
  • Needed to create new files
• unlink
  • Essential for deleting files
• write
  • Needed to actually write data to files
• rename
  • Needed for mv

FUSE maintains five queues for (1) interrupts, (2) forgets, (3) pending, (4) processing, and (5) background. The kernel module manages the complexity of request queues, prioritization, and fairness.

So developers using FUSE for user-space filesystems don't need to manually manage these queues. Able to focus on implementing each operation (e.g., read, write, mkdir, getattr) as separate callback functions in their FUSE daemon.

To benchmark the performance of this project, I'll be using

$stackfs$, from the paper —《To FUSE or Not to FUSE: Performance of User-Space File Systems》

Note: Discord Bot data limits

Future improvements

• The main cause of slowness in user-space file systems is the frequent kernel-user context switches that occur. The paper《Extension Framework for File Systems in User space》goes into detail on how to remedy this through registering "thin" extensions in the Linux kernel to handle certain simpler tasks, while the complex logic is kept on the user-space side.
• Add a web interface that's reminiscent of
  $Google$
  $drive$ or
  $Dropbox$, allowing for user friendly access on the fly, across devices.
• Stresstest the system to see it's limits
• Create more discord bots to work in parallel, negating PER-BOT API limits (There still exists PER-ROUTE limits)

Overview

Revision of my project from 2024 -> Discord Clouding v0.1 review

Fuse flow: