---
# System prepended metadata

title: Sharing large datasets

---

# Sharing large datasets

_Note, this is part of the [2021 IPFS project planning process ](https://github.com/ipfs/roadmap/blob/master/2021-IPFS-Project-Planning.md) - feel free to add other potential 2021 themes for the IPFS project by opening a [new issue](https://github.com/ipfs/roadmap/issues/new/choose) or discuss this proposed theme in the comments, especially other example workstreams that could fit under this theme for 2021. Please also review others’ proposed themes and leave feedback [here](https://github.com/ipfs/roadmap/labels/2021%20Theme%20Proposal)!_
## Theme description
Opimize IPFS for sharing large datasets. Examples would include movies, game assets, operating system distributions, docker images, scientific data sets etc.

## Hypothesis
Sharing large data sets in a p2p way is obviously beneficial. It is immediately obvious that sharing large data sets with a centralized approach comes with large cost and availability downsides.
Developers are frequently confronted with this problem.
Sharing large data sets is one of the first thing people usually try when evaluating ipfs. Currently it performs significantly worse than similar p2p technology such as bittorrent or dat/hypercore.

## Vision statement
IPFS should be the default technology that people reach for when sharing large public data sets. The speed of sharing large data sets should be within a factor of 2 of what is physically possible.

## Why focus this year
_Please discuss why 2021 is the right year for this theme._

## Example workstreams

- Offloading the DHT
  Currently, a large part of the cost of sharing large datasets is providing the individual blocks to the DHT. This is not really necessary for many kinds of large data sets. The amount of information stored in the DHT needs to be limited in such a way that it is no longer proportional to the size of the data set.

- Advanced block set reconciliation
  Sharing large data sets such as a copy of wikipedia published via an IPNS name involves answering the question which peers have which data.
  The current bitswap approach is very basic. It involves basically asking for individual hashes.
  This workstream would investigate higher performance approaches of set reconciliation such as the PinSketch algorithm.
  
- Low level optimizations
  This workstream would complement the two other workstreams with doing some low level optimizations to increase bandwidth of large data set replication.

## Other content
_Please include links to other relevant content, notes, etc._
