---
tags: datdot
---

# Polkadot Treasury Proposal

---
### Applicant

Contact person: Nina Breznik for DatDot by Playproject.io
Company: Vision Baker Ltd.

### Polkadot address
Ftr1nVnd4vdePzboaDW64Hru3n47MS68zE51zJQWYtsibTo

### Date
December 2021

### Requested Budget in EUR
104k

### Short description

**[DatDot](https://datdot.org) team is building an autonomous hosting network for p2p data systems.** Think of it as a Filecoin but for Hypercore protocol, built with Substrate.

[Hypercore protocol (started in 2013)](https://dat.foundation/about/history/) is a distributed ledger technology and set of data syncronization primitives with focus on immutable history.

Due to their P2P nature it's hard to have guarantees about the availability of p2p data systems. If we want to keep data available and up to date we have to keep our computer running or we have to rent a server.

**Challenges:** What if you want other people to help you host your data?

- what incentives do they have?
- how to find them and trust them?
- how to verify they are 'seeding' your data?

**Solution:** A bridge between Hypercore Protocol and Substrate with a built-in incentive model which manages relationships between:

- hypercore creators/publishers and
- hosters (who keep data available/host the data)

---

## About DatDot team

Our team has been **active in the blockchain space for many years**. **Nina Breznik** and **Alexander Praetorius** have been contractors with the Ethereum Foundation for about 3 years and **Joshua Mir** has been working with Parity tech and is an active member of the web3 ecosystem, especially Kusama network.

We have also been active in Node.js and the Dat p2p data sharing ecosystem for the past 8 years.

We have therefore decided to build the bridge between the two ecosystems - Polkadot/Kusama and Dat ecosystem - by developing a pallet for managing data sharing on Substrate based chains with Hypercore protocol.

We've applied for the [Grant](https://github.com/w3f/General-Grants-Program/blob/master/grants/speculative/datdot.md) in the late 2019 and were successful in receiving Web3 foundation's support of 50.000 EUR for 3 milestones.

For the past 2 years we have been building the technology and it hasn't always been easy working with 2 bleeding edge technologies - Substrate and Hypercore protocol - which have during this time introduced multiple breaking changes.

We succesfully submitted our first milestone - a prototype of the DatDot system - a pallet running on a Substrate chain, taking to a service.js code, running a hypercore protocol.

After this milestone we realized that building a pallet won't be enough, we will have to also build a custom chain to support our economic model.

We have spent a few months specifying the system with the updated incentives for each role in our p2p hosting network to make our economic and security model really solid. After that we started working on the implemention of the system as a combination of chain and service.js logic while refining it even more,

Because implementation feedback cycles have been too slow while working with javascript and rust in parallel (while specing out the whole system), we have decided to build a custom node.js server to simulate the chain and all of its extrinsics and connecting to it by using polkadot.js. This is how we managed to work on the system design refinements and implementation in a way more effective way.

In the meanwhile hypercore protocol went through 2 major updates - first the Hypercore module itself and then also its networking layer, a so called hyperswarm DHT. Changes opened a path to make our system even more secure and stable, but took another couple of months of refactoring.

In the meanwhile we have also been working on the UX and UI of the DatDot p2p hosting network and have managed to implement a significant portion of this task.

While not being able to deliver the second milestone yet, due to above mentioned curcumstances, we have been 3 full time + 2 part time people working on the project, **financing it (for the past 24 months) only with the first milestone payment (15.000 eur) and the rest purely from our savings**.

---

## What problem exactly is DatDot trying to solve?

The future of decentralized data sharing is in protocols like Hypercore and IPFS. The difference between them is that Hypercore protocol focuses on mutable datasets, which means data lives on one known public address and the content is always updating - but every change to a so called hypercore is cryptographically signed. This of it as git with torrent like features, but for data. This gives the protocol some sort of super powers because every new change is like a signed log and all the data in the dataset can be therefore easily split into micro data parts and synced with readers through p2p network (i.e. DHT) extremely fast and efficient.
But there is a problem p2p data protocols have and it is that the datasets don't live on a public chain, but instead they live only with one or few peers. Therefore, data is only available as long as at least one of that peers is online.
There is therefore a need for a service that helps to keep the data always online for other readers and which incentivizes users to do so. And this is what DatDot is doing for hypercore protocol (similar to what Filecoin does for IPFS).

With DatDot hosting network hypercore protocol is finally be able to offer a reliable p2p storage solution for mutable data, which can be used by:
- blockchain users (using datasets public key as an address),
- by hypercore protocol ecosystem,
- as a database for decentralized apps,
- for sharing community or scientific data resources
- and because of its efficiency also for (live) streaming big datasets, such as games, VR or live shows.

---

## How does it work?

DatDot chain works as a marketplace with users offering storage on one side and with users looking for storage on the other side. Users can also be both at the same time.

Think of DatDot as a simple desktop app with a connected view in the browser. When user needs their data hosted, they can use a simple form to post a request. A dataset that needs hosting will then be split into micro parts (so called dats) and DatDot will auto-select multiple users with matching resources (storage, bandwidth etc.) to host some of the dats.

On the other side every user is able to offer their resources to the network and once the hosting request is made and their performance rating is good enough, their devices are auto selected to perform the task.

Once hosting task has started, DatDot chain will in regular intervals choose random peers to perform an attestation and report to the chain how hosters are performing in their tasks. Hosters will need to respond by providing the Proof of storage (proof they are storing the data) and the Proof of performance (actual data delivery to the readers). To achieve the required performance and quality of the service more than one hoster will be providing the service. Users, providing the service, will be paid for their service every time the service performance was verified and report has been sent to the chain.

If user doesn't manage to provide the required performance or if their computer goes offline, they will be removed from the task and their performance rating will be updated accordingly.

DatDot chain will be a Substrate chain, connected to Polkadot network, but we will also release pallet(s) which other substrate based chains can use for their networks.

## APP UX examples

DatDot app works as a 'marketplace', where user can:

**1. OFFER A SERVICE**


---

**2. REQUEST A SERVICE**


**USER TXS SCREEN**


## What needs to be done?

As mentioned, we have already built the first proof of concept, but without detailed economic and cryptographic logic. This economic logic and cryptography has since been implemented but in javascript chain simulation. There are some adjustments that need to be done but majority of the work is done. We will only need to update verification logic for verifying proofs from Hypercore v9 to Hypercore v10 that has been released very recently.

Next, we also have a few networking issues that need to be fixed. In our network peers are connecting to each other to perform multiple tasks in parallel - for some tasks they act as service providers, in others as service requestors or maybe they are chosen by the chain to perform data encoding or to verify the hosting (as attestors). For each of the tasks peers need to connect to other peers on DHT so we need to make sure their connections are not duplicated and that connections and data streams between peers are properly closed after tasks are finished.

Also, for the network to properly work, we need to make sure all the running tasks are properly stored so they can be restarted if user's node crashes (or if their computer shuts down etc.). This enables the processes and execution to continue once the node is restarted.

Next, we also need to take care of proper key management. Data exchange between users for their tasks in the whole network runs through hypercores and each hypercore needs a new keypair so if a user wants to use DatDot on a new device or if they need to restore their account, we can't expect users to backup hundreds of keys. Instead, we are implementing a system to backup only the root key and store key derivation information on an encrypted hypercore, so user can easily restore all needed keys from there.

And finally, after all that is done we need to implement the whole chain logic (based on the live spec/javascript simulation) in rust/substrate. And last but not least - rust and hyper nodes have to be at the end packaged to run inside the electron app, using the user interface that also still needs some work.

There is still a couple of months of work infront of us, therefore we need support, to be able to pay our team for their excellent work.

## Milestones

| Milestone | Deliverable | Link | Notes |
| --------- | ----------- | ---- | ----- |
| 01 | Implementation of final economic and cryptographic logic | [Process diagrams](https://hackmd.io/mKQoDioEQMq-2CfAM7-Aiw?view) | Final implementation in Javascript based chain simulation (living spec for rust) |
| 02 | Networking module | [Networker module](https://github.com/playproject-io/datdot-service/blob/wip/src/node_modules/_datdot-service-helpers/networker2.js) | Module for managing networking and data streams |
| 03 | Restarting tasks system | | System for storing & restarting active tasks |
| 04 | Vault | [Vault](https://github.com/playproject-io/datdot-service/tree/wip/src/node_modules/datdot-vault) | Key management system with key derivation and backup logic |
| 05 | Rust node | [JS node](https://github.com/playproject-io/datdot-service/blob/wip/src/node_modules/datdot-node-javascript/chain.js) | Implementation of Rust node based on Javascript simulation |
| 06 | UI implementation (view) | [UI](https://github.com/playproject-io/datdot-ui/issues) | UI for DatDot app - based on the wireframes |
| 07 | Desktop app | | Desktop app which runs chain node and hypercore deamon |
| 08 | Documentation | [Docs](https://datdot.org/#doc) | Updated documentation and installation guide |

## Budget

| Milestone | Deliverable | Budget (in EUR) | Expected delivery |
| --------- | ------ | ------ | --- |
| 01 | Implementation of final economic and cryptographic logic | 8k | April |
| 02 | Networking module | 8k | March |
| 03 | Restarting tasks system | 8k | March |
| 04 | Vault | 8k | February |
| 05 | Rust node | 32k | August |
| 06 | UI implementation (view) | 8k | August |
| 07 | Desktop app | 24k | September |
| 08 | Documentation | 8k | September |
| 01 ... 08 | **SUM** | **104k** |

This is the estimation of our budget needs, hope all or at least part could be covered through the Polkadot Treasury.

## Payment Conditions

Payment should be divided in 2 instalments:
- I. instalment (32k): milestones 01 - 04
- II. instalment (72k): 05 - 08

We will report back to the council once the deliverable is ready and share the links to the code. DOT rate is based on 30 day avg tool by Subscan.

## Why Polkadot

Our system is built on Substrate so it is natural for us to connect the chain to Polkadot network. There are many reasons for this. First is definitely the security of the network that Polkadot as a service can provide to our chain, second the opportunity to offer our service to the existing chains and lastly, we also think that Polkadot ecosystem has a great community of developers and end users who are tech savy enough to demand and use solutions that we are trying to offer.

We had been in early discussions with projects like Joystream, Subsocial, Subsquid and Kodadot and they are seriously considering implementing p2p storage solutions into their tech. But it's a chicken and egg problem, because projects are looking for the storage persistence/pinning layer (Think Filecoin, Crust, Permaweb) and using the availability of those services to decide on their storage layer (IPFS, Arweave).

We are a future parachain asking for treasury funds and one might think this gives us a certain advantage when competing for the parachain slot. But we hope it is clear that funds we are asking for can not buy the slot, the deposits are orders of magnitude higher. Furthermore, we are asking for support to cover very concrete programming and design tasks since we are an independent, not VC funded team.

We currently don't have a parachain slot strategy: as stated, this funding will not subsidize any crowdloan campaign or bid, if that were the path we decided to take - instead, in the short term we will readily encourage other live parachains to make use of our pallets after they are complete, within their own economic models.