---
# System prepended metadata

title: 'Data Shell: Project Outline & Roadmap'
tags: [datashell, ' datdot']

---

---
tags: datashell, datdot
---

# Data Shell: Project Outline & Roadmap


--------------------------------------------------------------------------------

## 3. Prototyping environment project
Prototyping web apps with data vault

### 3.1 What & How

#### Goal

1. This project aims to grant users full control, self-authentication and data portability by providing developers with an open standard for secure data vaults and an open prototyping environment for creating web apps based on that standard.


2. This project aims to provide developers with an open standard to make secure data vaults and an open prototyping environment for creating web apps based on that standard, to create an ecosystem that grants end users full control, self-authentication and data portability.


---

3. DataShell project aims to provide developers with an open prototyping environment to build web apps which separate user data from the apps themselves and comply with a data vault standard to grant users full control, self-authentication and data portability.

---

COMMENTS (about above):

1. Use the first sentence when you want to emphasize the user benefits of the project, specifically full control, self-authentication, and data portability. The sentence begins with "This project aims to grant users..." which puts the user benefits upfront. The sentence also mentions that the project provides developers with an open standard and an open prototyping environment, but these are secondary to the user benefits.
    
2. Use the second sentence when you want to emphasize the developer benefits of the project, specifically the open standard for secure data vaults and the open prototyping environment. The sentence begins with "This project aims to provide developers..." which puts the developer benefits upfront. The sentence also mentions the user benefits, but they come after the developer benefits.
    
3. Use the third sentence when you want to emphasize the technical aspect of the project, specifically the separation of user data from tools and compliance with a data vault standard. The sentence begins with "This project aims to provide developers with an open prototyping environment to build new tools which..." which puts the technical aspect upfront. The sentence also mentions the user benefits, but they come after the technical aspect.



#### Problem

1. **data sovereignity**
- No existing p2p app decouples user data from the application, because there is no existing standard for how to do so and to enable many apps to share the mechanism

- it is necessary to fullfill the promise of users-own-their-data so they can use data with other apps, export and manage access to their data

- Once upon a time, in the early days of personal computers, users had all of our data on our disks and all software was reading and writing data to disk. Computers were new and operating system were not stable, disks were often crashing and backups were difficult to make and restore. Users were therefore happy when they could move more and more of their data into the cloud and didn't have to take care of backups themselves. But as users started producing more and more data, it became clear that hosting all this data isn't going to be cheap. Apps and services continued giving users free storage space but in return they got full access of their private data. User data is now a precious asset and the primary source of income for such companies and users have little control over where and how their data is stored and used online. 

But today machines and systems are a lot more stable and powerful. Furthermore, peer to peer technologies have matured and they enable users to securely backup and easily restore our data across devices. What is missing is an environment and standards about how to build web apps which decouple user data from the application itself so developers can start building services that bring back the power to the users.

- As soon as you sign up with a free email provider or install an operating system, you usually get some cloud storage space. Accessing your data through an online environment has become commonplace, for both, businesses and individuals. You can share everything from a grocery list with your significant other or even store sensitive documents in the cloud so you can access them from work or on the road. But to be able to have users upload, edit or create files online and pass them around, many cloud services require full access to users data. Like the extensive (and sometimes unreadable) privacy policies you already had to wade through to open your 'free' account, users do not have many options. Either you click yes and the cloud is yours, or you deny the apps all access and are left to your own devices.

- The fact that you want to access your data online, does not mean you need to store it in a place where the provider requires you to hand over the keys. Especially when it is unclear who can use the keys to look around in your documents, or analyze sensitive documents simply for the sake of personally profiled advertising, which quite certainly is not what you signed up for. You would want to know where your data is, lock the doors to it yourself and keep the keys somewhere safe (without any intricate key management or cryptographic busywork).

- Data portability is a personal right established in the General Data Protection Regulation (GDPR). This is in direct conflict with the desire of companies to retain users and their data. Data is often tightly coupled to the application, complicating transitioning data between services.

- Big Tech companies currently dictate how digital identities are used. As a result, they have amassed vast amounts of private user data. Movements like Self-Sovereign Identity aim to give users control over their online identity.

- Internet users have little control over where and how their data is stored and used online. Big Tech companies store gigabytes of data about you and know exactly which online services you use. User data is a precious asset and the primary source of income for such companies.

-  In our system, users are not reliant on Big Tech companies to authenticate themselves or store and host their data. User data is stored in a data vault on a device under the control of the user

- The goal of this project is to design a system that gives users sovereignty not just over their identity, but also over their data.
- As long as your data is on a remote server, it is not truly under your full control.

2. **p2p/dat ecosystem perspective**
- It requires a lot of time and effort from a dedicated team to build p2p apps and package them cross-platform

- development cycle is long, because getting users to download, try and give feedback to apps is hard because users hesitate to download unknown apps to their computers

- users don't want to download anything on their computer to just try and play around
- users see that if they want to commit themselves to p2p they will need to download many apps

**data sovereignity**
- each app will have a different way of dealing with data and a different identity
- users expect to have some sort of data vault and full control over their data on each app + easy way to export/import identity

**p2p development**
- developers don't know how to think p2p and have no guidance and best practices on how to use APIs
- developers need to be very experienced in many fields to build a p2p part, frontend and to package the app 
- developers can't quickly prototype and just play around and try to make app because it takes too much effort 

**p2p adoption**
- bringing dat in the browser has been tried many times, but developers still didn't massively start building p2p apps in the browser, why?


---
- your data is stuff you care about
- mostly you only get to interact with it in places owned by corporations
  - a bit like living in someone elses house
- you dont get to choose ho can see your stuff
- malicious actors can follow your activities and harass you
- owners of spaces can record what you do and sell information
- because it is not your space, you cant decide how anything works
- features can disappear overnight
- your data can be changed or deleted without your consent

- what if you and people you care about could band together
- have your own place for your data to live
- where only people who see your stuff are people you trust
- nobody is selling your privacy
- you decide how things work and when they should change


--------------------------------------------------------------------------------

### 3.4 Roadmap 

#### INITIAL MILESTONES

v0.1 - Core Environment
-----------------
- Core: portal (web server)
  - web server configuration standard
  - includes index.html with single script tag
  - includes shim.js to import and boot kernel script (loaded via script tag)
  
- Core: dpm OS (default WebApp OS, using web kernel)
  - includes .data/* folder on portal to host all OS customization code
  - defined via package.json dependencies and entry script
  - uses entry script for advanced customization 

- Core: web kernel (web runtime patch)
  - installs itself as a service worker and reloads the page
  - serves the OS system html page
  - serves scripts offline to load a visual shell
  - can add more scripts based on package json to customize OS
  - allows user shell code to be executed in runtime
  - provides dpm (data package manager) to be used by user via shell commands

- Core: shell
  - includes a command log so user sees history of interacting with the shell
  - includes a command line to type in javascript and commands for execution
  - indicates vault account user is logged in with (dummy account by default)
  - allows to list all data available on a specific portal (stored just in memory for now)
  - add advanced formatting to all basic types of shell messages for easy debugging
  - allow commonjs based import of data modules in executed code snippets

- Core: dpm (data package manager)
  - add basic data integrity verification mechanism
  - implement support for some very limited basic commands:
    - help command to list all available package manager commands
    - commands to list all available data
    - commands to download more data
    - commands to create a user project folder and data inside it
    - commands to download some data from the browser to the users disk on the device they use
    - commands to start a specific app available (apps have access to commonjs import mechanism)

- Core: vault:
  - add vault mode to kernel
  - specify vault message protocol to talk to kernel and apps
    - based on `comlink` library or similar logic
    - includes transferable arraybuffers
  - implement vault message protocol to communicate with kernel and apps
  - add basic permission hooks
  - add basic user permission prompts (all ow all vs. deny all)


- Core: sandboxing:
  - adds as basic iframe based sandboxing mechanism to kernel runtime
  - allows user shell code to be executed in such an iframe sandbox
  - add sandbox message protocol to enable communication with sandbox
  - add commands to list and spawn and shut down sandboxes


v0.2 - Basic UX/UI 
-------------------
- vault
  - wireframe basic responive data vault UI
    - list all apps/software installed
    - list all datasets created
    - indicate permissions set for a specific dataset
    - indicate permissions per portal
  - implement basic iframe view and fullscreen vault view
  - add UI for user to trigger actions available via data package manager such as:
    - add actions to change permissions
    - add actions to delete or add datasets
- dpm OS
  - nav bar
    - user account
    - offline/online indicator
    - open/close/collapse shell
    - data status indicator
  - content area
    - layout design


Initial unstable release
------------------------

### FUTURE MILESTONES

v0.3 - Basic Editor App
------------------------
- basic text editor
- no syntax highlighting
- actions: new/open/save file
- allow multiple editor tabs

v0.35 - dpm intermediate
------------------------
- allow installation of complex apps with dependencies from web server cache
- allow export of app with multiple dependencies created via text editor in vault
  - to be downloadable as zip file, which can be extracted to update web server cache

Pre-alpha release
-----------------

v0.4 - Intermediate Sandboxing
-------------------------------
- based on inter in-process sandbox ocapn (object capability network)
- using research from endo/lavamoat/agoric/SES
  - to harden the global primordial objects
  - utilizing javascript proxies and the with statement to restrict access
  - using eval in this restricted context to run code sandboxed
- integrate commonjs module import with in-process sandboxing
- will not have DOM access

v0.5 - Advanced Sandboxing
----------------------------
- implement fast and lightweight virtual dom
- implement a DOM communication protocol between sandbox and renderer thread
  - to shuttle dom updates from sandbox to renderer
  - to shuttle dom events from renderer to sandbox
- include basic signaling library and protocols to ease usability


v0.6 - Intermediate Vault
--------------------------
- implement a few basic demo apps to refine permission system
- make concept for different kinds of data types and structures supported to be used in apps
- make concept for meaningful user permission prompts based on data
- implement test library to semi-automate testing and simulating user interaction scenarios
- design access control interaction
- test permission system

Alpha release
--------------

v0.7 - Basic Deamon
--------------------
- create basic cross-platform installer for "localhost p2p data daemon"
- add hyper-dht-relay to data daemon installer
- add kernel code to detect and start communication with data daemon

v0.8 - Advanced dpm 
-------------------
- allow download and publish of data packages to p2p network
- enable export of data packages as p2p archives for web server cache
- enable dpm to download data packages from web server cache

v0.9 - Advanced vault
----------------------
- enable user self-authentication and self-authenticating data structures
- wrap all vault data in hypercores or similar p2p datastructures
- sync them from browser cache to device disk via localhost daemon
- enable sharable p2p dataset links for others to access vault data if permission is granted by user
- create p2p address book of trusted peers
- enable downloading datasets from peers given access was granted by those peers
- enable subscription to dataset updates 

Early Beta release
-------------------
- known bugs fixed
- release

--------------------------------------------------------------------------------

#### 3.4.2 Project outline 


portal (web server)
-------------------
- [ ] distributed network of **"portals to download & explore dpm OS customized with data packages"** many can host for users to have options
- [1] a portal is a static web server which hosts:
  - [1] index.html (or 404.html)
  - [1] shim.js (which imports & boots bootkernel runtime)
  - [2] (optional) .data/* (folder)
  - [1] package.json (config/manifest file)
    - [2] to define normal and dat dependencies
    - [1] (optional) entry_script.js (=init script) (to customize system by auto installing and executing whatever is bootscript)
- [1] public portal (static web servers)
  - used by developers and operators to publish/promote and share new p2p web apps and bootstrap node addresses
  - used by users to discover and quickly test p2p web apps and possibly share them
  - hosted by trusted portal operators
  - [1] work out-of-the-box in offline-first mode without the need to install anything on local system
  - [ ] securely accessible under a public domain based portal url
- [ ] private localhost home portal (needs native portal daemon installed)
  - used by users and developers for maximum security when working with their apps and data
  - used by users to securely browse the dat (=p2p network)
  - run by users
  - [ ] works after installation of the portal daemon by the user
  - [ ] securely accessible under http://home.localhost

Bootkernel/runtime web shim (dpm OS)
------------------------------------
0. Everything is a log! (not a file nor an object, but a log)
1. Code is tolerated inside the bootkernel only if moving it outside (e.g. permitting competing implementations), would prevent implementation of the system's required functionality.
2. Make each module do one thing well. To do new jobs, compose and build afresh rather than complicate old modules by adding new "features"
4. Expect output of every program to become input to another yet unknown program. Don't clutter output with extraneous information. Don't insist on interactive input.
5. We should have some ways of connecting programs like garden hose--screw in another segment when it becomes necessary to massage data in another way. This is the way of IO also.
6. Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them.
- [1] user's initial portal url visit requests browser to contact portal web server to download `index.html` and start installation
- [1] INSTALLATION: request -> portal web server: index.html + shim.js -> import + boot bootkernel.js -> restart
  - [1] download and evaluate `domain1/index.html` with single script tag `domain1/shim.js`
  - [1] download and evaluate `domain1/shim.js` to import and boot `domain2/bootkernel.js`
  - [1] download and evaluate `domain2/bootkernel.js` will install itself as `domain2/bootkernel.js#service_worker` (BKSW) and reload page
    - SEE MORE: => in renderer session
- [1] LOAD: shared kernel worker `bootkernel.js#shared_worker`
  - [1] hosts all kernel functionality and offer syscalls API
    - [1] Kernel code handles sensitive resources and implements the security and reliability barriers between applications
  - [1] wait for renderer session to connect
  - [1] (re)establish & maintain kernel connection to user cached or default data vault's shared kernel shared worker (in vault mode)
    - [1] do this by using "kernel execution hook"
    - [1] @TODO(1): test if shared vault worker can be kept alive solely by a foreign shared worker holding onto a message channel
      - [1] if not: make sure to at least keep one vault iframe around in one of the app tabs and re-establish if needed 
      - [1] vault (re)connection hook (optional use in case shared kernel worker has no vault connection yet)
        - [ ] spawn a vault iframe (purse mode) and use it to connect + change when user logs in or transitions to different vault
        - [ ] @TODO: maybe a vault shared worker can send a last message to vault user tunnels before last vault closes to respawn a vault
          - [ ] unless a vault user tunnel is enough to keep vault shared worker alive  )see @TODO(1)
        - [1] receive vault url and message port to forward to spawned vault url iframe to keep alive until told to shut down iframe
          - [1] double check security implications of message channel interception
  - [1] load package.json (from cache or fetch) via data-vault
    - SEE MORE: => in data vault
  - [1] detects boot mode based on url and environment to prepare script loading & initialization
    - [ ] embed mode environment (app iframe) -> just for CORS)
      - [ ] SCRIPTS: viewport, (API: sandbox, package manager, vault)
    - [1] plain mode environment (app window)
      - [1] SCRIPTS: viewport, shell, (API: sandbox, package manager, vault)
    - [1] vault mode environment (vault window)
      - [1] SCRIPTS: viewport, vault, (API: sandbox, package manager)
    - [1] purse mode environment (vault iframe) -> just for CORS)
      - [1] SCRIPTS: (API: sandbox, vault)
  - [1] load and initialize (extension shared worker modules) according to mode and extend kernel and offer additional "syscalls"
      - [ ] e.g. think: IOCTL => for uncommon hardware devices -> "device driver shared workers" to manage additional other resources
      - [ ] based on user settings and/or renderer session request (prompt user for permission)
  - [1] sends SCRIPTS: (custom cached and/or default) init and user scripts to new session (e.g. renderer)
    - [1] includes mode SCRIPTS (always includes: vault hook script + sandbox)
      - [1] e.g. refresh/reload script to any of the renderer session (e.g. browser tabs), e.g. to trigger a "reload"
    - [1] init script (entry script from package.json)
      - [1] init.js can be shim.js by just stopping exec via throwing error and cutting away initial 2 lines later on.
    - [1] user script (user defined customization)


Renderer Session
----------------
- [1] BOOT: (offline first page)
  - download and evaluate `BKSW/index.html` from bootkernel serviceworker which includes `BKSW/renderer.js`
```htmlembedded
  <!doctype html><html><head><meta charset="utf-8"></head><body><script src="renderer.js"></script></body></html>
  ```
  - download and evaluate `BKSW/renderer.js` script to load (spawn or connect to) shared kernel worker `domain2/bootkernel.js#shared_worker`
- [1] INIT: spawn user (sub)session via `BKSW/renderer.js` as main renderer process and run BOOT SCRIPTS
  - [1] setup kernel worker execution hook to listen for initial session BOOT SCRIPTS from bootkernel worker
    - [1] Renderer is a "container-frame" (browser-vm) connected to bootkernel shared worker [BK.shw]
  - [1] load `bootkernel.js#shared_worker` shared kernel worker (see below)
  - [1] receive initial session SCRIPTS from bootkernel worker and execute them
  ```javascript=
  // renderer "firmware"
  window.kernel = new SharedWorker('https://domain2/bootkernel.js#shared_worker')
  kernel.port.onmessage = ({data}) => eval(data) // renderer bootloader
  ```
- [1] available APIs depend on protocol scripts sent by kernel based on custom renderer apps 


dpmOS
-----
@GOAL: develop system in system
1. create first portal with meaningful data in hand coded portal
2. using legacy `umblical_fetch` (legacy fetch for https) to pull in legacy code, turn it into "hyper" MODULES in dpm/store to later export into new portal

shared worker bootkernel
```javascript=
const renderer = new WeakMap()
onconnect = event => {
  const [port] = event.ports
  renderer.push(port)
  port.onmessage = kernel_protocol
  // @TODO: define and check connected renderer type
  // @TODO: maybe based on "kernel mode" too
  const mode = "plain"
  port.postMessage(`(${default_renderer_JS})('${mode}')`)
}
function kernel_protocol (event) {
  // ...
}
////////////////////////////////////////////////////////////////////////////////////////////////////
;(async () => {
  document.body.replaceChildren()
  document.head.replaceChildren(Object.assign(document.createElement('style'), { textContent: `
    *, *::before, *::after { box-sizing: border-box; }
    body, h1, h2, h3, h4, p, ul, ol, li, figure, figcaption, blockquote, dl, dd { margin: 0; }
    body { display: flex; flex-direction: column; min-height: 100vh; }
    iframe { border: 0; flex-grow: 1; }
    img, picture { max-width: 100%; }
    table { border-collapse: collapse; }
    html:focus-within { scroll-behavior: smooth; }
  ` }))
  document.title = 'Playground'
  
  const { port1, port2 } = new MessageChannel()
  port2.onmessage = event => {
    console.log('[kernel]:renderer>', event.data)
  }
  port2.postMessage('hello renderer')
  const kernel = { port: port1 }
  default_renderer_JS('plain', 123)
  
  async function default_renderer_JS (mode, id) { // basic renderer // @TODO: unique renderer ID
    // https://markkarpov.com/post/the-identicon-package.html
    // https://terrykwon.com/blog/visual-hashing/
    // https://jdenticon.com/#icon-Dylan
    // https://morioh.com/p/91ef450fe563
    // http://jsfiddle.net/pHv6W/
    // https://github.com/MyCryptoHQ/ethereum-blockies-base64
    // https://github.com/laurentpayot/minidenticons
    // https://siva.dev/identicons/
    // https://github.com/Lobstrco/stellar-identicon-js
    // https://dev.to/krofdrakula/improving-security-by-drawing-identicons-for-ssh-keys-24mc
    // https://www.cssscript.com/svg-identicon-generator/
    // https://www.cssscript.com/canvas-geometric-identicon-generator/
    // https://www.cssscript.com/identicon-generator-jdenticon/
    // https://www.cssscript.com/github-identicon-avatars-squareicon/
    // https://www.cssscript.com/random-cartoon-avatars-svg-faces/
    // use: package.json#name + location.href + source code hash or hyperurl and version (log index)
    if (mode !== 'plain') return document.body.innerHTML = `<h1> '${mode}' is not supported yet </h1>`
    kernel.onmessage = renderer_protocol
    function renderer_protocol (event) {
      console.log('[renderer]:kernel>', event.data)
      // ...
    }
    document.onkeydown = event => {
      const { altKey, ctrlKey, shiftKey, /*metaKey, which,*/ code, key, /*charCode, keyCode, location, */ repeat } = event
      console.log({ altKey, ctrlKey, shiftKey })
      if (altKey && ctrlKey && shiftKey) {
        console.log('shell')
        shell.hidden = true
      }
      console.log(JSON.stringify({ code, key, repeat }))
    }
    ////////////////////////////////////////////////
    // FEATURES:
    // 1. ctrl+alt+shift   => toggle shell
    
    ////////////////////////////////////////////////
    // GOAL:
    // 1. sandboxed viewport (iframe sandbox)
    // 2. default shell, replaceable by custom shell
    ////////////////////////////////////////////////
    // RENDERER KERNEL:
    const shell = sandbox(shell_JS, shell_protocol) // shell_JS is not an address, but a script
    const viewport = sandbox(viewport_JS, viewport_protocol) // viewport_JS is not an address, but a script
    document.body.append(viewport, shell)
    return
    ////////////////////////////////////////////////
    async function shell_JS (port2, ID) { // script
      document.head.replaceChildren(Object.assign(document.createElement('style'), { textContent: `
        html { background-color: black; color: lime; font-family: mono; font-size: 8px; }
        *, *::before, *::after { box-sizing: border-box; }
        body, h1, h2, h3, h4, p, ul, ol, li, figure, figcaption, blockquote, dl, dd, pre { margin: 0; }
        body { display: flex; flex-direction: column; min-height: 100vh; }
        iframe { border: 0; flex-grow: 1; }
        img, picture { max-width: 100%; }
        table { border-collapse: collapse; }
        html:focus-within { scroll-behavior: smooth; }
        input, button, textarea, select { all: unset; max-inline-size: 100%; color: white; flex-grow: 1; }
        .shell { flex-grow: 1; display: flex; flex-direction: column; height: 100%; }
        .log { display: flex; flex-direction: column; flex-grow: 1; overflow: auto; max-height: calc(100vh - 9.5px); }
        .cmd { width: 100%; display: flex; }
      ` }))
      console.log('[renderer/shell_JS]', ID)
      port2.postMessage('ping')
      port2.onmessageerror = event => console.error('error', event)
      
      port2.onmessage = onmessage
      
      document.body.append(Object.assign(document.createElement('div'), { className: 'shell', innerHTML: `
        <div class="log"></div>
        <div class="cmd">🌐alexander@seraseed.com:/dpm/store/foobar#>&nbsp;<input placeholder="type command"><a title="Keyboard Shortcuts:\n ctrL+shift+alt: toggle shell">❔</a></div>
        <!--<div>👤alexander📍seraseed📂🗂️🐚/dpm/store/foobar🛠️▶<input placeholder="type command">❓❔📡⚫🌐🌎🌍🌏 ⚪🔴🟢🟡🔵🟣🟠</div>-->
      ` }))
      const [log, { children: [input] }] = document.body.lastChild.children
      input.onkeypress = event => {
        if (event.key === 'Enter') return input.value = evaluate(input.value) || ''
      }

      Array.from({ length: 30}).map((x,i) => (input.value=i,input.onkeypress({key: 'Enter' })))
      await new Promise(ok => setTimeout(ok, 200))

      return
      function onmessage (event) {
        console.log('[renderer/shell_JS]:system>', event.data)
        // ...
      }
      function evaluate (data) {
        log.append(Object.assign(document.createElement('div'), { innerHTML: `<span>></span> <span>${data}</span>` }))
        log.scrollTop = log.scrollHeight
        port2.postMessage({ type: 'exec', data })
      }
    }
    function shell_protocol (port1, ID) { // onready
      // @TODO: messageport .close() and .onmessageerror
      console.log('[system]:renderer/shell', 'init', ID)
      port1.postMessage('hello')
      port1.onmessageerror = event => console.error('error', event)
      port1.onmessage = event => {
        console.log('[system]:renderer/shell>', event.data)
        port1.postMessage('pong')
      }
    }
    function viewport_protocol (port1, ID) { // onready
      // @TODO: messageport .close() and .onmessageerror
      console.log('[system]:renderer/viewport', 'init', ID)
      port1.postMessage('hello')
      port1.onmessageerror = event => console.error('error', event)
      port1.onmessage = event => {
        const { type, data } = event.data
        console.log('[system]:renderer/viewport>', type, data)
        port1.postMessage('ack')
      }
    }
    async function viewport_JS (port2, ID) { // script: viewport is augmented no-script iframe
      document.body.style.backgroundColor = 'darkblue'
      console.log('[renderer/viewport_JS]', ID)
      port2.postMessage('ping')
      port2.onmessageerror = event => console.error('error', event)
      port2.onmessage = event => console.log('[renderer/viewport_JS]:system>', event.data)
      await new Promise(ok => setTimeout(ok, 200))
    }
    ////////////////////////////////////////////////
    function sandbox (script, onready) {
      if (typeof script !== 'function') throw new Error('no script function provided')
      if (typeof onready !== 'function') throw new Error('no ready callback provided')
      const ID = `${Math.random()}`.slice(2) // @TODO: where to get real pids from?
      const srcdoc = `<!doctype html>
      <html>
        <head><meta charset="utf-8"></head>
        <body><script>(${firmware})(${ID}, '${location.origin}', ${script})</script></body>
      </html>`
      const opts = { srcdoc, sandbox: 'allow-scripts' } // @TODO: what about CSPs and other flags?
      const iframe = Object.assign(document.createElement('iframe'), opts)
      window.addEventListener('message', onmessage)
      console.log('[kernel]', 'spawn:iframe')
      return iframe
      function onmessage (event) {
        if (event.source !== iframe.contentWindow) return
        event.stopImmediatePropagation()
        window.removeEventListener('message', onmessage)
        const { data, ports: [port1] } = event
        console.log(`[kernel]:sandbox-${ID}>`, data)
        console.log({ port1 })
        onready(port1, ID)
      }
      async function firmware (ID, origin, script) {
        const name = `[sandbox-${ID}]`
        const AsyncFunction = (async () => {}).constructor
        const source = opener || parent
        const { port1, port2 } = new MessageChannel()
        try {
          const run = new AsyncFunction('port', 'ID', `await (${script})(port, ID)`)
          console.log(name, 'init:script')
          source.postMessage('ready', origin, [port1])
          // @TODO: could use OCAPN to sandbox scripts
          const code = await run(port2, ID)
          // @TODO: maybe await script and then close/shutdown worker/iframe and notify all users or it's user          
          console.log(name, 'exit:script', code)
        } catch (error) {
          port2.postMessage(error)
        }
      }
      function ready (source, data) {
        if (source !== iframe.contentWindow) return console.log('@TODO: wat pxeboot?')
        console.log('[kernel]:iframe>', data)
        source.postMessage(pid, origin, [port2])
      }
    }
    ////////////////////////////////////////////////
  }
})()
```



SUMMARY
-------
0. portal (web server)
    - hosts index.html/shim.js/init.js/package.json/data/*
1. bootkernel (web runtime) [dpmOS]
    - install kernel service + vault connection for pkg.json + send init scripts to renderer
    - load kernel module extension shared workers 
2. renderer session
    - connect to kernel service to receive init scripts and execute them
3. sandbox
    - ...
4. data vault
    - ...
5. dpm (data package manager)
    - ...
6. shell UX
    - ...
7. viewport (UX)
    - ...
8. deamon
    - ...
9. Data registry and p2p hosting (datdot)
    - ...
10. Prototyping Environment
    - ...
11. Apps/Tools
    - ...
12. About Page
    - ...




sandboxed runtime
-----------------
- [1] enables apps/tools to run in a sandbox container
- [1] coarse grainded iframe sandboxes
- [ ] finegrained ocapn in-process sandboxes
  - [ ] super minimal & ligtweight
  - [ ] endo and lavamoat inspired (in proces sandbox per module inside a single app)
  - [ ] ocapn (object capability network) based security
- [ ] dat/hypercode compatible
- [ ] enable to pause/resume/rollback/replay/reload from logs
**runtime/system**
- [ ] Users can switch from one application to the other because data is decoupled from applications by design.
- [ ] allow users to use their data in web apps without authorizing apps to access user's whole data vault
- [ ] consists of a container system single-page application which opens remote components/apps/data-adapters in sandboxed iframes
- [ ] communicates with and between components/apps via HTML 5 message channels
- [ ] provides data api and system api to apps to communicate with data vault
  - [ ] operations like: new, open save, ...
- [ ] always waits for users explicit requests
- [ ] always asks for users explicit permission
- [ ] a minimal environment for testing and easy deployment of single apps
- [ ] allows **easy integration** of single-page web apps and provides them with **data acceess** operations and the required **user interface** for permissions
- [ ] respects freedom of computer users
- [ ] goal is composable programmable user configuration/services
- [ ] provides a **consistent user experience** for all apps, while freeing the app developer from **great amounts of work**.
  1. [ ] Apps are loaded in iframes inside the window.
  2. [ ] Once the app is loaded, it connects to data-vault through an async-enabled messaging channel.
  3. [ ] On connection (immediately after the app is loaded), the app sends to the data-vault its manifest, which consists of various meta data about the app and the description of its capabilities according to this API.
  4. [ ] After connection, the app receives commands from the shell, which it executes and returns the resuls, or throws an error. The result or the error is then transmitted back to the shell. Use `WebShellApp.command` in the app to register command handlers.
  5. [ ] The app can also send events to the shell. Use `WebShellApp.trigger` in the app to send an event.
    - [ ] Commands, expected results, errors and events are defined in various app APIs.
- [ ] your data, your choice
  - [ ] true data ownership
    - [ ] users should have the freedom to choose where their data resides and who is allowed to access it. By decoupling content from the applications itself, users are now able to do so.
  - [ ] modular design
    - [ ] because applications are decoupled from the data they produce, users will be able to avoid vendor lock-in, seamlessly switching between apps and personal data storage servers, without losing any data or social connections.
  - [ ] reusing existing data
    - [ ] developers will be able to easily innovate by creating new apps or improving current apps. all while reusing existing data that was created by other apps
- [ ] lets people store their data securely in decentralized data stores called pods. pods are like secure personal web servers for data. when data is stored in somebodies pod, they control which people and apps can access it
- [ ] data vault: all your data under your control
  - [ ] any kind of data can be stored in a data vault, from structured data to regular files that might be otherwise stored in a google drive or dropbox. people can grant or revoke access to any slice of their data as needed




Data Vault
----------
One of the main goals of this work is to help users regain control over their data and thus over their privacy. The first step is to standardize how apps are separated from user data to then enable users to put their data into a data vault, which they can self-host. The system is specifically designed to not rely on cloud infrastructure. All data is stored with and hosted by the user. This makes the vault GDPR compliant by design, so there is no data storage for processing by third parties eliminating the need for Privacy Officers and Data Protection Officers. This stops data-hungry companies from running machine learning algorithms over user data and learning users’ behavioral patterns. This has the added benefit of disrupting Big Tech from monetizing user data.
- [1] loaded from a vault portal
- [1] initializes bootkernel in the same way as a plain portal environment
- [1] receive `vault.js` and run it
  * [ ] if user wants to change vault portal, that's one thing => trust in portal developer (e.g. use home portal)
  * [ ] if user wants to change vault script, that's another thing (e.g. change user interface and features of vault)
    * [ ] keep baseline to reset vault script
    * [ ] but keep standards based data and basic kernel protocols used by any vault script
    * [ ] hence run vault script in sandbox and log all it's interaction with data
    * [ ] vault script always has all permissions and forwards to vault users

==> NEEDS SANDBOX FIRST !
==> then probably needs PACKAGE MANAGER first (used by kernel by default to install package.json)
    ==> maybe that's enough to assume shit is there and we can think about package manager later
    ==> i guess package manager api and vault is the same or at least related (npm vs. npmjs.org) or (git vs. github)
        ==> git can also push to remote git repos ...interestingly (maybe hypershell is the answer!)
        
LESEZEICHEN: do not receive `vault (re)connection hook` but instead `custom or default vault script`


- [ ] when user logs in to dat pages, they connect the page to the vault (user sets the permissions for every page)
- [ ] requires the kernel/runtime to start in "vault mode"
  - [ ] it works similarly to the page environment, but because runtime script is in the vault mode, this page acts as a vault
- [ ] enables self-authentication: stores user's public/private keypair
- [ ] signs new data created (but private mode is also supported)
- [ ] password protected
- [ ] on this page, user doesn't log in, because this page is a vault
  - [ ] maybe "if" a user can login with a different vault, then maybe for the purpose of switching to a different vault
- [ ] it takes care of storing your data 
  - [ ] in the browser cache until you install the dat deamon
  - [ ] when using a dat deamon the vault stores data on your device
- [ ] apps/tools from the shell get downloaded to the vault cache (shell install)
- [ ] types of data stored in vault: 
  - [ ] user data from the apps
  - [ ] source code of installed app
  - [ ] source code of all visited portals
  - [ ] user's own projects
  - [ ] audit logs: 
    - [ ] shell log
    - [ ] access log
      - [ ] Tamper-proof access log: Access control is completely automateable without the intervention of the data vault owner. This makes it impossible to keep track of who has been given access to which files. A policy editor lets your view, create and update access policies for a selected pieces of data.
      - [ ] The tamper-proof access log does not provide indisputable proof that a requester retrieved a file. The record only claims that the requester could access a set of files based on the provided VCs.
  - [ ] keypairs
- [ ] data is version controlled
- [ ] includes:
  - [ ] indicator next to each data slice to show which data is synced and which is just local (diff)
  - [ ] indicates the authorship (write & read/read only)
  - [ ] customization file for every app you have installed
  - [ ] log of shell commands 
- [ ] shim
  - [ ] loading the kernel in the vault mode
- [ ] permissions - [ ] access control management
  - [ ] fine-grained access control (for every chunk of data)
  - [ ] enable portability
  - [ ] **access control management**
    - [ ] local access policy meta data information associated with each piece of user data. Giving the user fine-grained access control allows the user to have specific disclosure policies at the desired granularity level, down to the file level. This comes with great responsibility, as there is the opportunity to make mistakes when defining access policies and disclosing data to unintended parties. The challenge is to make the user experience simple and intuitive to minimise the chances of mistakes.
    - [ ] share it safely: anyone or anything that accesses data in a data vault uses a unique ID, authenticated by a decentralized extension of keypairs. data vault control system uses these keypairs to determine whether a person or application has access to a resource in the data vault
    - [ ] Pod owners have granular control over who has access to the data. Solid uses the WebAccessControl system, which is based on access control lists with user identification by WebID, to grant and revoke access to any slice of data contained in a Pod to individuals, organizations, or applications. WebID is a protocol that allows persons, organizations or other types of agents to create unique identities and embed links to other people or objects using Resource Description Framework. WebID makes it easy to make arbitrary claims about yourself, but those claims are not trustworthy as they are not verifiable. At best, they provide authentication by proving possession over a private key. Access control rules can be based on the agent’s properties in their profile document.
- [ ] **Vault new features**
  - [ ] When opened, a password is required to decrypt the data vault. This prevents unauthorised access even if someone gets physical access to the device and launches the application.
**data vault: access control**
- [ ] data vault provides fine-grained access control for the user’s locally stored data.
**data vault**
- [ ] For now, data can be opened and saved from the local data vault.
- [ ] data vault stores user keypairs
- [ ] privacy-first (offline) secure data vault for personal storage
- [ ] allows users to store all their data on their smartphones and control with whom they share it
- [ ] data vault gives users back control of their identity and all their data
- [ ] a starter set of data system permission adapters
- [ ] A data vault with self-authentication standard would be a secure data storage system that includes a built-in mechanism for verifying the authenticity and integrity of data stored within it. This mechanism could use cryptographic hashing or digital signatures to ensure that the data has not been tampered with or altered in any way. The data vault would be designed to provide users with full control over their data, while also ensuring that it can be shared securely and transparently with other parties. The use of self-authenticating data and a secure data vault can help to protect users' privacy and ensure that their data remains under their control at all times.
**user control data sovereignity**
- [ ] Nearly five decades after the invention of public key cryptography, we still lack a good solution for people to manage their digital identity and efficiently share encrypted data directly with each other, certainly at a massive scale. Various movements aim to halter Big Tech’s power and give back control to the users. These movements are powered by technologies like blockchain and Self-Sovereign Identity (SSI), which promise to improve how we interact with online services and each other.
- [ ] This work aims to solve these problems by developing a data vault with advanced data sharing capabilities that leverage SSI to provide users with true sovereignty over their data.

- [ ] A system where users have true sovereignty over data has to have the following properties:
    - [ ] Data storage has to be decentralised on devices under the control of the data owner.
    - [ ] Access control has to be verifiably authenticated, finegrained and resolutely enforced.
    - [ ] Data has to be decoupled from applications


Data package manager (dpm)
--------------------------
- [ ] all data is stored in the browser cache or vault (after once downloaded from portal server or p2p network)
- [ ] can be accessed via shell
- [ ] API
  - [ ] .list 
    - [ ] list available programs for installation
  - [ ] .install 
    - [ ] stores in data vault cache
    - [ ] install the dependencies from package.json
    - [ ] entry script auto runs dat install when the page opens
    - [ ] if user is connected to the dat network (through the deamon) they can install more dependencies
  - [ ] .init
    - [ ] creates a new project folder with a package json in the data vault
  - [ ] .export
    - [ ] download project folder as zip file from browser to disk
  - [ ] .run
    - [ ] opens a program in sandbox (e.g. sandboxed iframe)
- [ ] supports upgrades and rollbacks of data
- [ ] npm install experience (in the browser)
- [ ] **package manager**
  - [ ] supports atomic upgrades and roll-backs which make an effectively unbreakable system
  - [ ] goal is distributed package distribution
- [ ] **package manager**
- [ ] a universal functional data package manager
- [ ] focuses on bootstrappability and reproducability
- [ ] gives users strong guarantees on integrity of full softwarestack running

shellUX
--------
- [ ] special js console
  - [ ] main feature: fetching dat stuff and downloading it to vault cache 
    - [ ] if no deamon you fetch it from this page's web server (github page's repo)
    - [ ] if deamon use hyperswarm
- [ ] shows the account you are logged in with 
- [ ] logs your commands and stores them in the vault
- [ ] lists all the data allowed by the vault 
- [ ] has access to the portal code (data in the repository of the portal)
- [ ] through shell you interact with:
  - [ ] dpm (`list`, `install`, `init`, `export`, `run`, )
  - [ ] data vault (`ls`, `cd`, `whoami`)
  - [ ] kernel
- [ ] you can run apps/tools from other people (in the shell)
- [ ] you can use it to do any js stuff (fetch etc.)
      - [1] enables running all the code without any additional build tools




- [ ] shows shell UX (=special devtools/js console)
    - [ ] shell(ux) spawned by renderer (can be replaced for different shell if needed)
      -  stdin/stdout/stderr
      - => shell translates all into input/output log messages (back & forth)
      - : user -keyboard-> "rendered shell (shellux)" -stdin/on[X]-> shell -write()-> pseudo terminal master or bootkernel

taskbar
- give commands (shortcuts in jump lists)
- access resources
- view program status
- 

[user]@[node]:[path]#[task]>


DATASHELL UX:
- RUNLOG: everything is a log (not object, not file, but log!)
  - sends user input to connected node(s) (e.g. app)
  - receives and displays output from connected node(s)
- COMMANDLINE:
  - PROMPT: [user(=e.g. vault)]@[node(=e.g. tool)]:[path(=e.g. project/folder)]#task(=e.g. roadmapping/todo)> [ACTION/PARAMS]
  - APIDOC: eval, ...
    - ACTION: commands/program or api call (defines messages type)
    - PARAMS: all data entered after a space following the command
```bash
user@domain:path> asdf
user@domain:path> asdf1
user@domain:path> asdf2
serapath@shell:~#live>
~: vault user home
/: normal path local to app
.. and . only exist in commands, not in "path" segment of prompt
user@domain:~/path/to/dir> asdf
user@domain:path> asdf2
user@domain:path> asdf3
```
```javascript=
input.onkeypress = e => {
  if (e.key !== 'Enter') return
  const { value } = input
  const { user, node, path, task } = prompt
  const msg = make(user, node, path, task)
  shell(msg)
}
```
```javascript=
const task = { // task & task protocol
  pid: project.id,
  tid: taskid(),
  ins: [],
  out: [],
  log: [],
}
```



viewport (UX)
--------
  - [ ] viewport
    - [ ] users use the default app or tool defined in the entry script defined by the developer of the page
    - [ ] can be customized by portal operator
      - [ ] shell
      - [ ] apps/tools defined in the entry script (if none, shell is full screen) 
      - [ ] apps/tools installed by the user using `shell install`
    - [ ] can be customized by user
      - [ ] installing other apps by running 'dat install'
      - [ ] customizing the look and feel of the page via a `user_script`
      - [ ] all the adjustments/customization to the page is stored in the user's data vault
  - [ ] user account bar
    - [ ] user account
      - [ ] shows your username (default username/you can rename it)
      - [ ] first time you get a default account (keypair and username) 
      - [ ] if no deamon, it says, this is just a demo account
      - [ ] default vault url is chosen by the admin
      - [ ] you can logout
      - [ ] login with existing (by pasting your own vault url) 
      - [ ] click to open the vault page (opens a new tab or iframe)
    - [ ] indicator for p2p offline/online
      - [ ] includes an indicator, showing if you are offline or connected to dat network through a deamon
    - [ ] indicator for data status (synced/not synced)
      - [ ] green for all up to date, red for when there is uncommited data
    - [ ] open/close the shell 
      - [ ] when closed, input filed is still available



Deamon
------
- [ ] enables you to connect to the dat network (hyperswarm) (gets you 'online' in the p2p sense)
- [ ] provides storing on the device instead of browser cache
- [ ] can run additional sandboxed services (other p2p network / blockchain nodes)
- [ ] enables data sharing
  - [ ] selected data from user vault is available to the other users when this user is online/connected to the dat network (using the deamon)
- [ ] data is protected in transit with noise protocol
- [ ] **p2p/Hypercore**
  - [ ] Data is also protected in transit with noise protocol. Packets are encrypted with the recipient’s public key and signed for authenticity and integrity.
  - [ ] In the future, users will be able to securely backup and share data in the peer to peer network.

Data registry and p2p hosting (datdot)
-------------------------
- [ ] enables selected datasets to be always available
- [ ] makes all the published data publicaly available (data can still be encrypted, but it can be always accessed unless it is unpublished from the registry)
- [ ] uses public ledger to store data addresses and a network of peers volunteering to host data for other peers
- [ ] verifies if data is really available
- [ ] uses fairness ratio to avoid spam and malicious attacks
- [ ] DatDot registry uses public ledger to store data addresses and a network of peers volunteering to host data for other peers
- [ ] **DatDot data registry / p2p hosting**
  - [ ] There are several threats to data availability. The first threat is the lack of redundancy. All the user’s data is on a mobile device that can temporarily or permanently be out of service for a number of reasons. If the data vault is not backed up on a more reliable medium, the user risks losing data if the device becomes permanently inaccessible. 

Prototyping Environment
----------
"Encourage to write programs that do one thing and do it well. Write programs to work together. Write programs to handle json, because that is a universal interface."
- [ ] Just like when you start up your laptop or home computer, you start up the prototyping environment and browse your data vault, open an application to edit something, and switch it off again.
- [ ] Unopinionated open customizable web based developer environment for prototyping p2p web apps with a data vault
  - [ ] next generation internet environment to build next generation internet
  - [ ] decouple content from app logic instead
  - [ ] user friendly runtime/system customization
- [ ] includes links to support chat/forum to get fast feedback loops
  - [ ] so users can provide data vault feedback
  - [ ] developers can provide building apps feedback
  - [ ] we can co-develop and improve the standards with the audience
- [ ] enables web apps shareable via link so users can try quickly and give feedback
  - [ ] no need to download apps first
- [ ] prototyping environment is built with the prototyping environment
  - [ ] source code of apps build by developers using it are stored in their data vault 

Apps/Tools
----------
- [ ] work by default without the build tools
- [ ] can be configured seed other peers' data (chat or social networking or collaboration tools) 
- [ ] apps are downloaded from p2p network or portals (package.json includes links to community/get-in-touch/issues)
- [ ] default apps: 
  - [ ] code editor
- [ ] **system**
  - [ ] allows full operating system including desktops
- [ ] **default apps**
  - [ ] add first apps users can use with the data vault
- [ ] **UX/UI**
  - [ ] complete data control and user-friendly data access in an environment you are already familiar with: an online version of your desktop.
**app ecosystem**
- [ ] Data vault’s open architecture encourages the development of new decentralised applications that can read and write data to the user’s data vault. Different applications can provide different ways of interacting with data in users’ vaults. This makes for a more competitive ecosystem as user data is completely portable between applications.
- [ ] goal: create interoperable ecosystem of applications and data
  - [ ] data stored in data vault can power ecosystems of interoperable applications where individuals are free to use their data seamlessly across different applications and services
**prototyped apps**
- [ ] apps implement serialization and deserialization of individual file content
- [ ] a starter set of demo apps 
- [ ] instead of these apps requiring full access to your data that is hosted somewhere in the world, you can use your data vault
- [ ] apps never operate on your data directly. The apps only securely open and save your data and only with your express permission.
**prototyping/developing apps**
- [ ] Anyone is able to develop new decentralised applications that can interact with data in your data vault.
**web2 data importer apps**

About Page
----------
- [ ] **For users/For developers**
  - [ ]  show features on the website
  - [ ] https://websh.org/sandbox/
- [ ] sitemap:
  - [ ] help
    - [ ] getting started
    - [ ] api docs
  - [ ] npm packages
  - [ ] plan: milestones
  - [ ] about: info + funding
- [ ] publish on a presentation page using roadmap + project outline hackmd as backend
- [ ] link page in application


--------------------------------------------------------------------------------
