---
# System prepended metadata

title: Thesis

---

# Thesis

The "thin waist" of the "merkle forest" needs a stable, ubuquitous and nearly-universally agreed upon stream chunking algorithm that is sufficiently good to not require a switchover for several decades. The term "good" here covers performance/resource usage in terms of
 - computation phase space/time ( RAM/CPU )
 - requirements at rest (deduplication potential)
 - requirements at transport (overhead for subdag identification/communication)

**Provided** [such a beast exists](https://xkcd.com/2268/) (and I firmly believe it does, similar to Zooko's ["local maximum for hash functions" conjecture](https://twitter.com/zooko/status/835294257888002050)) it would apply simultaneously at the following spots **without** needing to work within the IPFS ecosystem at all, but rather parallel to it:
  - A boon for FLOSS mirrors: duplication in packaging archives is rampant
  - True "holy-grail" ETAG in the HTTP world
  - A block-deduplication-atom for ZFS/BTRFS/whoever-will-listen

## Part 1: how we got here

### 2016 Obective(s):
 - Ability to serve both 
     - https://kernel.org/pub/linux/kernel/v5.x/ and 
     - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/

   utilizing the same backing leaf nodes for everything
 
 - Derive a stable stream hashing function that converges with the above but at the same time is suitable to be included in something like [`rhash`](https://www.mankier.com/1/rhash#Hash_Sums_Options)

 - {{ Insert 30 second demo of the toolkit from that era }}

### Problems identified while doing above

TLDR: If one wants to interoperate with current and *future* web gateways, one is essentially at the mercy of PL. Moreover it is **really** difficult to advocate for any spec work ahead of time, as "the code is not written yet". Which basically means that you either:

 - dump stuff into the DHT and hope you got it right / that PL will bend to your worldview
 - wait ( what I've been doing, because the above alternative is a dick move )

Specific issues: [I wrote this list back in December, some stuff may have changed since](https://hackmd.io/EX5uh93eRfuivmfNbZYgsA)

## Part 2: go-ipfs-time
In late January real work on https://github.com/ipfs/specs/issues/227 started (followed by rapidly diverging expectations of what the end result should be)
  - mini-thesis:
     - The time for "experimental interfaces" has passed: islands of stability are **absolutely necessary** at the tooling boundaries to ooze confident "yes, you can use this for your data" on every turn
     - "systemd is an architectural mistake": building more and more intricate stateful multi-gigabyte-eating daemons does not help adoption. In other words: this *should not* be writen: https://github.com/ipfs/notes/issues/434 
         - Lean on simple "unix-style" streaming interfaces as much as possible, while allowing extensibility
         - "Do one thing and do it well" components
         - Keep an eye on runtime (RAM/CPU) performance envelopes

  - {{ 5 min demo of what :dagger_knife: does as it is right now }} 

## Part 3: where do we go from here

As it stands :dagger_knife: is the perfect tool to bridge the:
 - "we want it fast but don't know much about the scientific method" crowd with the
 - "we know information theory inside out, but can not write real code to save our lives" crowd, with the 
 - "I just want to mirrot my ubuntu archive quickly" crowd.

Sticking points:
  - If this were at my pre-PL time, I would know exactly how to market/run/extend this project
  - After joining PL everything became **really** "complicated"
  - {{ the remaining 45 mins go towards figuring what do we do }}