# NGI grant application for Solid Data Modules
With Jackson, Angelo, Reza and Michiel.

## Grant details
https://nlnet.nl/entrust/

## Web Form Answers
### General
Thematic Call: NGI Zero Entrust
Contact Information: Michiel de Jong
Project Name: Solid Data Modules
Website: https://solidproject.org

### Abstract: Can you explain the whole project and its expected outcome(s).
The Solid Project enables a "Bring your own Data" architecture, but this is only useful if apps understand the data they find on the pod. The client-client specs are the crucial but underdeveloped core part of the Solid project that needs urgent attention now.

Even though most Solid apps are still demo apps, we already see data in the user's pod being stored in multiple different and incompatible ways. That's why we will develop a number of "polyglot" JavaScript modules, one per data type.

These modules will build on the existing remoteStorage modules work (funded by NLnet in 2011) and the (partially NLnet-funded) Solid Application Interoperability spec.

They will support the data types already documented in Michiel's https://pdsinterop.org/conventions/overview/ initiative and
Jackson Morgan's https://shaperepo.com/ project.

Apart from making data accessible, reliably updating index files, and preventing data corruption, the Solid Data Modules will also automatically show the app developer which fine-grained Data Grants to request.

We hope to finally stop the bad practice of demo apps that request root access to your pod.

### Have you been involved with projects or organisations relevant to this project before? And if so, can you tell us a bit about your contributions?

Michiel de Jong has worked on Solid server implementations, semantic interoperability tools, the Solid test suite, Solid OS and several stand-alone Solid apps, also first as Inrupt employees and later as independent open source developers.

Michiel de Jong has furthermore worked on personal data stores since 2010, first in the (primarily NLnet-funded) Unhosted project, and later on the Solid project, developing various servers and apps, and setting up the Solid test suite, also with NLnet funding. Michiel is also the founder of the Ponder Source Foundation, an 8-person post-capitalist open source company which has recently (in the context of the Solid Working Group formation at W3C) become a W3C member.

Jackson Morgan is the founder of O-Team, another important player within the Solid project, author of famous Solid apps like LiqidChat, producer of popular Solid app dev tutorials, and he previously worked on the NLNet-sponsored https://nlnet.nl/project/SolidUsableApps/. He also worked on Solid server implementations (together with Michiel at the time, in Inrupt's Pod Server team), semantic interoperability tools like shaperepo.com, and he is the main author of solid-auth-fetcher and the Solid WebHooks (Notifications) specification and its implementation in the Community Solid Server.

Angelo Veltens is a core member of Tim Berners-Lee's Solid OS team, renowned Solid app developer, inventor of the "micro apps" vision for Solid and co-organizer of this year's Solid Symposium. Outside the Solid project, he is also an accomplished full-stack web developer and IT consultant.

Reza Soltani is an accomplished open source frontend (React) developer, and currently working alongside Michiel in a prestigious project that the Ponder Source Foundation is carrying out for SURF. When that project completes, he will be a welcome addition to the Solid Data Modules team, also providing fresh frontend developer eyes, dealing with data on Solid pods for the first time.

### Requested amount
50,000 euros

### Explain what the requested budget will be used for? Does the project have other funding sources, both past and present?
* develop one data module (small includable JavaScript library) per data type, and make it respect data formats that existing demo apps use.
* document which data types get stored on Solid pods by all the currently existing (demo) apps. Actively reach out to app developers,
try out Solid apps, then see which data structures they leave behind on the pod.
* continue the development of tooling like https://pdsinterop.org/conventions/overview/ and https://shaperepo.com/
* participate in the interoperability panel, together with elf Pavlik's three times NLnet-funded work from https://nlnet.nl/project/Solid-Interop/
* Implement Data Grants (https://solid.github.io/data-interoperability-panel/specification/#data-grant) in an Authorization Agent that can run on top of Solid-Nextcloud or other implementations (see https://solidservers.org)
* Contribute to the Data Grants spec at the Solid WG level (the Ponder Source Foundation is a W3C member since today)
* Establish and repair interop between existing demo apps, both inside and outside Solid OS

Other funding will come from:
* Stichting Nederlandse Datakluis (collective Dutch media partners) have contacted Ponder Source for consultancy and possible Solid app development, so if their 50 mln euro "groeifonds" grant is awarded (to be confirmed in June 2023), this could become a considerable
source of funding for us to do much open source work on interoperable Solid app over the coming years.

### Compare your own project with existing or historical efforts.

This work builds on remoteStorage modules, solid-logic (the business logic module of Solid OS that Michiel started), Pod OS (the interoperable WebComponents library that Angelo promotes to Solid app developers), our two existing shape documentation efforts, the interoperability spec, and the existing efforts to document client-client specs.

SolidLabs Flanders will also soon published a shapes repo tool but that is currently still in local development. We already made contact with them and agreed to work together as soon as they are ready for this, probably around June 2023.

A lot of work and attention is going to the Solid project, but the apps we develop still rarely get past the "demo" status and into day-to-day use by any significant user base, and to our slight embarrassment, our own demo apps are often developed in isolation and not even compatible with each other yet.

And for additional embarrassment, a lot of existing work Solid apps also still request root access to the user's pod.

Millions of euros are now being invested in Solid by several EU member states and organizations. But if we don't fix both these embarrassing holes in our work, then the entire Solid project is at risk of staying as unusable as it unfortunately still is.

In response to questions about how to deal with the limits of the "Document Box" nature of Solid pods, there has been a strong statement from Tim Berners-Lee to emphasize the importance of client-client specs for Solid, which is also a motivation for us to focus more on this topic now.

### What are significant technical challenges you expect to solve during the project, if any?)
We have to work with multiple existing practices, they might be based on incompatible assumptions.
Existing data formats may not be easily mappable onto each other.
The code may become complex with lots of if-else-statements in order to support lots of different models.
There may be unknown unknowns that explain why data import and fine-grained Access Grants have proven so illusive in the Solid project so far.
We may be met with a "not my problem" attitude in Solid app developers, and may have to convince them of the urgency of both issues. Using Solid is useless if your app cannot import data that was written by other apps. And requesting root access to a user's pod is an inacceptable anti-pattern. Yet many demos of Solid talk over these two topics as if they were just details. Our challenge is not just to write code, but also to open up these uncomfortable but necessary conversations, and convincing others to do extra work to make their app into a true usable Solid app.

### Describe the ecosystem of the project, and how you will engage with relevant actors and promote the outcomes? (E.g. which actors will you involve? Who should run or deploy your solution to make it a success?)
Angelo, Jackson, and Michiel are all well-connected core figures in the Solid project, who regularly attend (online) meetups and regularly talk with (industry) stakeholders in the project. We will thus disseminate our results in direct conversation with existing Solid app developers, and through the forums and meetups, both online and in-person. Even this proposal was written at the successful Solid Symposium 2023 which Angelo co-organized in Nuremberg yesterday, and which reached over 100 in-person participants.

### Attachments
(none)