Distribyted

• Distribyted
  • Introduction
  • Key Features
    • Abstracted Filesystem Approach
    • Smart Compression Handling
  • How Distribyted Works
    • Practical Example: SQLite File Access
    • Extending the Concept
  • Comparison with IPFS
  • Distribyted Gate: An Experiment in Browser-Based Torrent Handling
  • Future Development Ideas
    • Karma System for Peer Prioritization
    • Consistent Hashing for Torrent Content Distribution
    • Exploration of Advanced Data Formats and Compression Algorithms
    • Implementation of BEP 44 for Mutable Torrents
    • Webseed Integration for Cost-Effective Data Storage
  • Demos:
  • Contact

Introduction

Distribyted offers a unique solution to handling large datasets as a torrent client. This platform is designed to revolutionize dataset consumption by overcoming traditional storage limitations, enabling data exploration and utilization regardless of its size.

Key Features

Abstracted Filesystem Approach

At the heart of Distribyted is its novel approach to treating torrent files as filesystems. This allows for a range of options, including FUSE, HTTP, and WebDAV. The design is inherently flexible, making the implementation of additional serving methods straightforward and enhancing the application's adaptability to various use cases.

Smart Compression Handling

Distribyted stands out from conventional torrent clients with its ability to directly mount certain types of compressed files. This feature is particularly advantageous as it enables users to download only the essential parts of a dataset. This efficiency minimizes bandwidth usage and expedites access to critical data, making it an ideal solution for handling large-scale datasets.

How Distribyted Works

Distribyted redefines the way torrent files are accessed and utilized. By interfacing with torrent files as if they were part of a filesystem, it unlocks unprecedented flexibility and efficiency in dataset handling.

Practical Example: SQLite File Access

Imagine needing to read a SQLite file. Typically, this would involve downloading the entire file before any data can be accessed. Distribyted, however, employs a more intelligent approach:

• It begins by requesting SQLite headers, which are essential for obtaining necessary metadata. This triggers a block request for the relevant parts of the torrent.
• Subsequently, the SQL client is loaded, allowing the execution of queries. In cases where filtered queries with a WHERE condition are used, Distribyted reads the B+Tree index and then requests only the specific bytes needed from the SQLite file.

This process demonstrates Distribyted's ability to provide efficient and targeted data access, a significant advancement over traditional methods.

Extending the Concept

This approach is not limited to SQLite files but extends to various file types within file systems, such as zip, rar, tar, etc.

For instance, when needing to read a file within a ZIP archive, Distribyted fetches only the ZIP file headers and downloads the specific compressed file you need, bypassing the retrieval of extra data.

The process is further detailed in this sequence diagram:

Comparison with IPFS

While IPFS uses UnixFS for file handling, Distribyted adopts a different approach for optimized performance.

In IPFS, navigating the file information scattered throughout the tree can be slow. Distribyted overcomes this by providing all essential file information upfront in the header, significantly speeding up the file tree generation process.

Notably, the initial exploration of IPFS integration in Distribyted shifted towards Torrent use due to its superior performance and efficiency. This decision is evidenced in our early implementation.

Distribyted Gate: An Experiment in Browser-Based Torrent Handling

Distribyted Gate is a testament to the versatility and power of handling torrent files as filesystems.

It utilizes Service Workers to create a virtual HTTP server within the browser itself, enabling the serving of webpages directly from a static webpage. This can be hosted on any static HTTP server or even run locally.

Unlike other solutions that require specific coding for torrent compatibility, Distribyted Gate allows for the generation of a torrent from any static HTML content, ensuring ease of use and broad applicability.

Disclaimer: The functionality is not super polished yet, you might have to refresh the browser a couple of times to avoid some errors on the Service Worker side. Some ad blockers might avoid connections to needed trackers.

Future Development Ideas

There are several exciting possibilities on the horizon that can further enhance its capabilities and utility. Below are some of the innovative ideas being considered for future development:

Karma System for Peer Prioritization

• Concept: Implementing a karma system to create a more efficient and reliable peer network.
• Functionality: This system would prioritize peer nodes based on their contributions to the network, such as the duration they spend serving content or the frequency with which they provide data that is in high demand.
• Benefits: Such a system would incentivize nodes to contribute more to the network, ensuring a healthier and more robust distribution system. It would reward those who consistently support the network, leading to faster and more reliable access to data.

Consistent Hashing for Torrent Content Distribution

• Concept: Utilizing consistent hashing to optimize the distribution of torrent content among peers.
• Functionality: This method would automatically spread torrent content across peer nodes, ensuring a more balanced and efficient distribution.
• Benefits: This approach can significantly enhance data retrieval speed and reduce bottlenecks, especially in scenarios where certain data segments are in higher demand. It ensures a more equitable load distribution among peers, improving overall network performance.

Exploration of Advanced Data Formats and Compression Algorithms

• Concept: Expanding support for other data formats and compression algorithms, particularly those that facilitate seeking, like Parquet, Snappy, and S2.
• Functionality: By integrating these formats and algorithms, Distribyted would be able to handle a wider range of data types more efficiently.
• Benefits: These advanced formats and compression algorithms are designed for high performance and can support efficient data seeking. Integrating them would greatly enhance Distribyted's capabilities in handling large-scale, complex datasets, making it an even more versatile tool for data management.

Implementation of BEP 44 for Mutable Torrents

• Concept: Integrate the BEP 44 (Bittorrent Enhancement Proposal 44) specification into Distribyted to enable mutable torrents. This feature would allow updates to data within a torrent without changing the torrent's identifier (infohash). Note: I helped to implement BEP 44 on the Go DHT implementation.
• Functionality: Utilizing the BEP 44 spec, Distribyted can support mutable data within a torrent. This means certain fields or files within the torrent can be updated or changed while keeping the torrent's identifier constant.
• Benefits:
  • Dynamic Content Management: This would transform Distribyted from a static data serving tool to a dynamic content management system. It is particularly useful for datasets that undergo frequent updates, like financial data, live feeds, or collaborative projects.
  • Enhanced Collaboration and Sharing: This feature would be invaluable for collaborative projects, where multiple users need to contribute or update shared datasets. It ensures everyone has access to the latest version of the data without the hassle of managing multiple torrent files.

Webseed Integration for Cost-Effective Data Storage

• Concept: Implement webseeds in Distribyted to utilize S3-compatible static storage systems for initial seeding of data, thereby reducing the reliance on and cost of traditional centralized storage solutions.
• Functionality:
  • Initial Seeding from S3-Compatible Storage: The primary data seed would be stored on an S3-compatible static storage system. This approach leverages the scalability and reliability of cloud storage for the initial seeding process.
  • Peer-to-Peer Offloading: Once the initial seeding is done from the main source, subsequent data distribution will primarily occur through peer-to-peer sharing. This significantly reduces the load on the main storage source.
  • Webseed Protocol Implementation: Distribyted would implement the webseed protocol from the BitTorrent specification, which allows torrents to retrieve data from HTTP(S) servers in addition to the traditional peer-to-peer network.
• Benefits:
  • Reduced Storage Costs: By offloading the majority of data distribution to a peer-to-peer network, the overall reliance on and cost of centralized storage solutions are significantly decreased.
  • Enhanced Scalability: This system allows Distribyted to handle large datasets and high user demand more efficiently, as the distribution load is spread across a network of peers rather than a single server.
  • Reliability and Availability: Even in cases where peer availability is low, the data remains accessible via the webseed, ensuring consistent availability.
• Challenges and Considerations:
  • Initial Seeding Strategy: Careful consideration must be given to the strategy for initial seeding, including the selection of storage providers and management of seeding costs.

Demos:

• This document as a static webpage torrent.
• Flickr dataset on a zip file, accesible from HTTP directly fetching requested content on demand.
• Sqlite seeking from the browser static webpage example. Open browser console to see what is going on. The same example served directly from a Distribyted node.

Contact

• Resume: https://ajnavarro.com/
• Github: https://github.com/ajnavarro