# Building Something Like BitTorrent Part 5: File Storage Application

The final part covers how I wrapped a simple file storage application around the Kademlia DHT described in the previous 4 parts. From the video in the original part the interface I implemented has these methods
- `bootstrap(vector<string> endpoints)` - restricted to founders: create a new DHT session available against each provided endpoint
- `join(string remote_endpoint, string local_endpoint)`: join an existing DHT session available against the remote endpoint, where the new node is available against the local endpoint
- `exit()`: leave the current DHT session
- `list()`: list all files available in the current DHT session
- `store(vector<string> in_filepaths)` - restricted to founders: upload each provided file in the current DHT session
- `load(vector<string> files, string out_filepath)`: download each provided file from the current DHT session and write all (concatenated) to the provided file path

In my implementation the application runs as a daemon (running the Kademlia DHT logic) or in an interactive loop with each method above handled the same.

## File Metadata
There's a fundamental incongruence between the file storage application and the underlying DHT - the file storage application needs to query file data based on the deterministic file name, while the DHT indexes sharded file data (Chunks) based on nondeterministic Keys. In order to fix this I added an exception to the above DHT rule - the DHT will expose the following interface for Key generation
- `Key key_from_data(Chunk c)`: generates a key from the chunk's data bytes (e.g. by taking the SHA-1 hash)
- `Key key_from_string(string name)` (new): generate a key from a constant name

When adding a file to the DHT there are now 2 operations:
- the file contents are sharded into Chunks -> each Chunk `c` is inserted into the DHT using the Key gotten from `key_from_data(c)`
- a file metadata Chunk is constructed - the contents of this metadata Chunk are the **sequentially ordered + null-terminated + concatenated Chunk keys from above** -> the metadata Chunk is inserted into the DHT using the Key gotten from `key_from_string(filename)`

Now it's possible to access the data chunks for a given filename by
- getting the metadata Chunk from the DHT using the deterministic Key
- listing the nondeterministic Keys in the metadata Chunk and getting the file contents in each corresponding data Chunk

Finally we want to be able to list all files stored in the DHT. I added a reserved "index" file with a single Chunk, the contents of which are the **null-terminated + concatenated** file names (here the Key for the the Chunk is gotten from `key_from_string("index")`).

---
With this extra metadata layer the implementation of the top-level file storage application becomes pretty trivial

### bootstrap
- create a new DHT session for each endpoint - use the successive endpoint in the vector as the corresponding remote endpoint in `session::join`
- note - this must be done concurrently (otherwise the very first session will block indefinitely since it'll just be waiting for the successive endpoint to start!)
- create the index file in the DHT session with no files

### join
- create a new DHT session from the provided local/remote endpoints using `session::join`

### exit
- exit the DHT session using `session::exit`

### list
- retrieve the contents of the index file from the DHT session using `session::get` (again using the deterministic index file Key)
- the contents of the index file are just the names of the file so these are directly returned

### store
- shard the file contents into Chunks and insert each data Chunk into the DHT (saving the Chunk Keys) using `session::set`
- concatenate the Chunk Keys and insert the resulting metadata Chunk into the DHT using `session::set` (as described above)
- add the file name to the index file (read the contents using `session::get` and update with the new file using `session::set`)

### load
- retrieve the metadata Chunk from the DHT using `session::get` (again using the deterministic Key gotten from the file name) and parse the contents into the data Chunk Keys
- retrieve the data Chunks from the DHT using `session::get` using the data Chunk Keys