## Sprint Planning: data.gov (aka 300 TB Challenge)
Date: 2017-17-01

#### Lead: @flyingzumwalt
#### Notetaker: @flyingzumwalt

#### Participants

- @flyingzumwalt
- @jbenet
- @whyrusleeping
- @kubuxu
- @jonnycrunch

## Notes

## Useful Links & Issues
* ipfs/archives#87: **Main Sprint Issue** for data.gov Sprint
* ipfs/archives#104: **Main data.gov Epic**: Replicate 350 TB of Data Between 3 Peers (and then the World)
* [Epics](https://github.com/ipfs/archives/issues?q=is%3Aopen+is%3Aissue+label%3Aepic+milestone%3A%22Data.gov+%28aka+300+TB+Challenge%29%22) for the whole sprint
* [ipfs-pack Stories & Epics](https://github.com/ipfs/archives/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Data.gov+%28aka+300+TB+Challenge%29%22+label%3Aipfs-pack)
* [filestore Stories & Epics](https://github.com/ipfs/archives/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Data.gov+%28aka+300+TB+Challenge%29%22+label%3Afilestore)

## Big Optimizations

**TODO**: dig up diagram @whyrusleeping created

* **Adding** is still very slow
* adding large files is faster than adding lots of small files
* need a way to test these things See ipfs/archives#102
* @lgierth recently added ~3.2 TB for CCC. It took about a day to add. Performance dropped as the repo grew. Would have taken half a day if performance had stayed constant.
* @kubuxu ran some tests (see https://github.com/ipfs/archives/issues/105#issuecomment-273235394)
* Path forward: design good tests. See ipfs/archives#102
* **fetching from network** is very slow (won't be able to fill the pipes)
* tests should also address this
* **DHT with huge datasets** might get oversaturated
* DHT is not going to scale in time for this sprint -- means we need to find a way to do the routing See ipfs/archives#120
* {@whyrusleeping mentioned something i didn't hear..}
* **Garbage Collection** might not work with huge datasets
* **leaving GC out of scope for this sprint.**
* **Bitswap** hasn't really been tested yet
* @lgierth & @whyrusleeping ran some tests on this but didn't get clear info. It took over a week to ???
* See ipfs/archives#121
* Private Networks -- do we need it in order to do ipfs/archives#116 and ipfs/archives#120?

## Test Suite

See ipfs/archives#102

Currently not scaling well. Don't have good metrics, graphs or reports about performance -- where/when/how performance dipped under certain circumstances.

We need to know more than "Does it scale?". We need to know "how does it scale?" So we can identify the domain of problems, etc.

## Filestore

See [filestore Stories & Epics](https://github.com/ipfs/archives/issues?q=is%3Aopen+label%3Afilestore)

The current implementation mixes porcelain UX concerns with the underlying iplementation/plumbing. This makes the interfaces confusing & complicated. It also makes the underlying plumbing more complicated and less robust than it should be.
Best approach: take the pieces of the code that we need and package it as an *experimental* feature with simple, straightforward interfaces.

@jbenet & @whyrusleeping need to sit down and figure out how they want to proceed with this. @flyingzumwalt will try to capture that info in the [filestore Stories & Epics](https://github.com/ipfs/archives/issues?q=is%3Aopen+label%3Afilestore) Main things that need to be specified:
* How to do the internals/plumbing
* What the UX should look like

## ipfs-pack

* [ipfs-pack Stories & Epics](https://github.com/ipfs/archives/issues?q=is%3Aopen+is%3Aissue+label%3Aipfs-pack)

### The case for ipfs-pack

Currently the way people use go-ipfs is with `ipfs add` which creates a duplicate copy of the added data on the machine. With `filestore` we aim to build indexes of pointers to data/blocks in-place. This solves performance concerns, but creates a brittle situation -- if you move the file, ipfs won't be able to serve it any more. ipfs-pack aims to address this by building manifest files that hold the indexes that match ipfs hashes to the content. If you store those manifest files alongside the cotnent they point to, it becomes a portable dataset.

Extending that idea, if you create little .ipfs repositories next to the manifest files, it becomes possible to

* serve that dataset as its own little ipfs node
* register the contents of that dataset with another ipfs node, serving the content directly from wherever you've stored/mounted it

#### Why implement ipfs-pack now?

* Makes the UX much smoother for providers and their peers
* packs make the a lot of these concepts clear, straightforward & relatable

<!-- After each call, it is the responsibility of the notetaker to save the last
version of the notes in a file in ipfs/pm/meeting-notes, by opening a branch and
submitting a PR. -->