# DATDOT REPORT 2023 🌿

## Menu
* [FUNDING](https://hackmd.io/LTc1X7kmTVGq-7Nt4To8XA?both)
* [Grants Timeline](https://hackmd.io/gDDWHpPeQaGEzv2T-dW_Kg?edit)
* [Grant Programs](https://hackmd.io/hy-ZmXE1QqCgKVRgzOVWFw?edit)

## Financial Situation
**DatDot received funds to implement an incentivized peer-to-peer hosting service, but was faced with adverse conditions leading to significant delays and the need for additional funding.**

**DatDot**, a substrate blockchain-based project, received funding from the Web3 Foundation and the Polkadot treasury at the end of 2019 and in 2022, respectively. Despite the received grants of (€50k + 2000 DOT from Web3Foundation and €100k from Polkadot Treasury), the effective funding over the last ~3 years has been totalling €50k (€30k from Web3Foundation and €20k from Polkadot Treasury) due to delayed and not yet achieved milestones and the crash of cryptocurrency markets in general, leaving us severely underfunded. The funds are meant to finance the implementation of an incentivized peer-to-peer hosting service and to augment the team with additional personnel to speed up the process. However, due to underestimating the effort needed and facing a lasting crypto winter, a period of economic downturn in the cryptocurrency market resulting in a significant devaluation of the funds received, the project faced significant challenges. As a result, the team exercised restraint in terms of personnel recruitment and instead focused on pushing the project forward with a limited team. Despite the passage of over a year, the cryptocurrency market has yet to fully recover and the value of the funds received stays greatly diminished. The project has not been moving as fast as expected and **DatDot** has been compelled to apply for an amendment and supplementary funding. The team is committed to pursuing the project, but would be grateful for more support.

## Recent Work

### 1. UI/UX research consolidation and finalizing wireframes
**We consolidated UI/UX wireframe research for DatDot and made significant progress on frontend modules to simplify implementation in a way compatible with the hyperocre technology.**

**DatDot** has been focused on improving the UI and UX of our platform through comprehensive research and consolidation of wireframes. Our designs, now in a single version-controlled Figma format, are mostly ready for implementation. We have created a detailed project plan in GitHub issues to guide development. Additionally, we have established conventions for component development and created tutorials for contractors. We have also developed a "protocol-maker" module for connecting apps built this way to our core P2P technology with ease. Prototyping and implementation of the first wireframed components has been completed, indicating progress towards our goals.

### 2. Upgrading DatDot service to holepunch hypercore v10 release
**We refactored our code to fix breaking changes introduced by the latest hypercore and it's new collaboration features**

The holepunch team, currently main maintainer of the core primitives around hypercore/dat, upgraded the main modules, to features to better support multiwriter data structures for collaboration uses cases. This meant significant breaking changes for our **DatDot** service and we have been tirelessly working on upgrading our modules to stay up to date with the latest developments. Our refactoring work is in it's final stages and we are now debugging the last minor glitches. We hope to release the latest hypercore 10 upgrade of the **DatDot** service soon.

### 3. Bridging the ecosystems (dat-ecosystem & dotsama)
**We network and maintain relations with dat-ecosystem projects and collaborate on harmonizing our APIs to be compatible with those of other projects, to ease and maximize later adoption and to actually bridge between the dat-ecosystem and dotsama community**

The "dat ecosystem" started in 2013 and is an ecosystem of projects and communities, creating 100% open peer to peer infrastructure to allow secure decentralized sharing and efficient replication of large data sets between peers, all built on top of the **dat protocol**, which was renamed to hypercore as a low level building block. It is noteworthy, that dat ecosystem has grown to an impressive 22 active projects, each with their own active community of developers and users, building apps, tools and libraries on top of the shared underlying peer to peer infrastructure primitives, which range from data sharing platforms, to collaborative research tools, to decentralized social networks, and more, all of which have the potential to benefit from our substrate based **DatDot** hosting service once we release it. This highlights the strategic importance of our engagement with this ecosystem and the potential for our solutions to be widely adopted and indirectly promote the dotsama ecosystem, to actually deliver the "bridge" between ecosystems we set out to build.

To maximize the potential for adoption of **DatDot**, we work hard to closely align our developer facing **DatDot** APIs with other dat ecosystem projects for compatibility reasons, so we have been actively building relationships with the most foundational of dat-ecosystem projects, most notably with SocketSupply, who are working on a peer to peer web runtime called "ssc" and keet-holepunch building another peer to peer web runtime. Both p2p runtimes enable cross operating system native peer to peer app development with (html/css/js) for all major desktop and mobile operating systems. Refactoring our **DatDot** bridge api to align closely with dat-ecosystem and those runtimes helps every p2p app developer to use our dotsama/substrate based **DatDot** p2p data hosting service.

### 4. DatDot bridge architecture consolidating research
**Most of our work went into the datdot bridge research consolidation, which turns out to be difficult to estimate and aims to create a "web kernel" that combines most of the fundamental core web mechanisms with to intergrate well with hypercore, a user data wallet, a standardized p2p web app developer UX and our datdot p2p hosting system**

Lastly, our **DatDot** research consolidation draws closer to an end too but this is the part that turned out much more difficult than expected, mainly because of the **DatDot** browser bridge. There have been half a dozen to a dozen attempts to create that bridge by other dat-ecosystem projects in the past, but mostly as researech spikes to prove it is possible, but without the funding or will to maintain and without the ability to interface with a users "data wallet". The bridge we are designing bridges between:
1. our substrate based **DatDot** hosting chain extrinsics API
2. the hyper-dht-relay deamon to enable access to peer to peer data sharing in the web
3. a standards based user data wallet which doubles as a data vault
4. and a nice developer facing API to build on top, which is aligned with holepunch and socketsupplies ssc, both of which are peer to peer web runtimes.

The research is closely tied to a variety of research spike implementations we have successfully been able to implement, utlizing many cross browser core web API features, such as **servie workers**, which enable and constitute the backbone of our bridge architecture, otherwise **CSP** and **CORS** rules to secure the bridge and **Dedicated Workers** and **Shared Workers**, to offload execution from the main renderer thread, and allows us to maintain long-running sessions without memory leaks bringing down the system. It also allows us to more easily use state of the art **sandboxing**, aligned closely with work from the **Agoric Team (endo)** and the **MetaMask team (lavamoat)** to properly isolate code execution from each other. We otherwise make heavy use of **windows** and **iframes** and **workers** and communicate between them and **shared workers** and **service workers** via the native **postMessaging system** and **Message Channels**, which also utilize **Atomics** and **SharedArrayBuffers** to optimize for performance when dealing with the large peer to peer datasets.
The browser bridge can essentially be seen as some sort of **"p2p web kernel"** that developers can use to build web apps which do not require a server-side backend and allows user to stay in control of all of their data using a standardized data wallet. Those apps also natively support peer to peer data hosting via our substrate based **DatDot** chain service, which we developed quite far, but which we need to integrate with the browser bridge and then further stress test in order to maximize user adoption.

This part of our work is hardest to estimate, because the final break through could happen any time and feels quite close, the implementation then becomes straightforward because of all the spike implementations which basically means things can be implemented in a matter of weeks once the research has been fully consolidated.

# AMMENDMENT PROPOSALS - intro
We are in the process to write an application and a project plan that will allow us to leverage all of the knowledge and experience we gathered over the last 3 years to be able to estimate with bigger accuracy how long project execution will take so we avoid situations like in the past.
The request from the Web3Foundation to submit an ammendment proposal came unexpectedly and the given deadline did not allow us to break down the project into enough detail yet, to be able to precisely estimate.
We plan to continue the process to eventually end up with such a precisely estimated project plan for future applications, but the ammendment proposals below still suffer from the vagueness that inherently comes with leading edge technology (substrate, hypercore) and doing something that has not yet been done and the difficulties in estimations inherent to research in general.

## Proposal 1 - `UX/UX web app` (duration one month)
The first ammendment proposal is to clean up and finalize the entire UI/UX for the user facing app to interact with the DatDot hosting system in a single figma wireframe in great detail, broken down and versioned into individual web components, together with a detailed project plan to implement individual components and how they depend on other components and the repositories with the first few components fully implemented with a preview page for each of those components.
It also includes a series of tutorials to teach how to implement the remaining components in a style compatible with the peer to peer data sharing technology we use, using the "protocol-maker" module and some other conventions, so any frontend developer can continue the work, by watching the tutorials, following the laid out project plan and implementing component by component based on the finalized wireframe that details all components including the finished DatDot hosting app. Additionally they can use the first components that have already been implemented as an example to follow.

## Proposal 2 - `lastest hypercore upgrade` (open ended, pay when finished)
The second ammendment proposal alternative is for us to finish upgrading the DatDot service, which we submitted for the previous milestone to the latest version of hypercore v10, which includes changes to the cryptographic proof format, the addition of forking and truncating in hypercores to simplify collaborative data structures build on top of hypercores and some API changes. This is work in progress and we mostly finished that work, but we are still debugging several smaller issues, because we are getting duplicated connections infrequently and the data encoding is sometimes incorrect and we haven't yet figured out the exact constellation in which that happens, so it is a bit difficult to estimate how quickly we can deliver that. It might be a few weeks away, but as with debugging "heisenbugs" in distributed systems in general, it can sometimes take a longer time, so it is a bit risky and might not be doable within the next month, and we also might discover new bugs as soon as we fixed the current one.

## Proposal 3 - `datdot browser bridge` (open ended, pay when finished)
The third ammendment proposal alternative is to finalize the bridge architecture and implement it as planned.
This was our original goal in the previous ammendment and is still what we focus most of our time on to finalize the bridge architecture to seamlessly integrade the different concerns:
1. user data wallet
2. web access to udp based hypernet
3. dev UX api standardized and in line with ssc and holepunch runtimes
4. permissioned access to DatDot chain api for p2p hosting service

Given the variety of web apis involved, service workers, shared workers, dedicated workers, iframes and postMessage, shared buffer and message channel communication across different domains with the right set of CSP and CORS rules and sandboxing to make it all work in a secure way and given how much legacy the web api surface carries including the many weird APIs with all their exception, it is quite a tricky and finicky job to puzzle it all into place as some sort of "p2p web kernel" that connects all of the above mentioned concerns in one unified and simple developer experience.

We made a lot of progress, testing every aspect with short tech spikes, coding things up to verify they are feasible, we are 100% certain it is doable, but we do not dare to put an estimation on this task. It is our favorite option and we would just ask for patience.

We will at some point deliver the browser bridge - the missing puzzle piece to make DatDot work and we would hope to be paid the final milestone, but we are not sure to be able to deliver it in a fixed timeframe.