Secure Data Storage Features

EDV Client Features

• Perform client-side encryption of data such that EDV servers (storage providers) cannot view the data
• Search metadata on EDV servers using encrypted indexes (search by media type, tags, etc.)
• Use Authorization Capabilities to perform CRUD operations on EDV servers
• Perform peer-to-peer replication for the data they have access to

EDV Server Features

• Support Authorization to create/read/update/delete resources. EDV Servers are purely a policy enforcement points (authorization), not policy decision points (authentication).
• Authorization is supported using Authorization Capabilities (zcaps)
• Pluggable replication allows different replication strategies to be implemented.
• Simple counter-based replication for all stored objects.
• Set/get standard global configuration settings (default chunk size, supported features, etc.)
• Provide CRUD for encrypted indexes to clients for the purposes of searching/indexing
• Provide CRUD access to private encrypted data (not shared)
• Provide CRUD access to shared encrypted data (multi-party sharing)
• Provides delegation/attentuation for authorization to do CRUD operations via zcaps
• Support complete replacement for data (last write wins)
• Support version history for data
• Logging of the lowest level operations of the EDV (e.g. for the purposes of auditing)

Hub Features

• DID Authentication based on the pub key refs in a DID Doc. Support Authentication to create/read/update/delete resources. Hubs are both a policy enforcement points (authorization) and a policy decision point (authentication).
• Perform (filtered) peer-to-peer replication
• Multi-instance peer-to-peer active/active replication between instances.
• Differentially replicate subsets of the logical whole dataset across different instances.
• Provide access to public plaintext data
• Provide access to shared encrypted data
• Provide access to private encrypted data
• Enable the identity owner and multiple external identities (e.g. via OCaps) to modify the same objects within the Hub in a convergent way.
• Limit writes/reads by external entities. An example of this would be Alice, the datastore owner, being able to limit writes and reads of objects she specifies to Bob and others, in keeping with access and permissioning criteria she sets.
• Keep private objects out of public circulation.
• Support Last-Write-Wins complete replacement of objects
• Support objects that can be seamlessly merged via conflict-free replicate data mechanisms.
• Implementations can run in the Web platform
• Implementations can run in native/mobile app
• Implementations can run in remote server environments
• Provide inferential queries on semantic data that can be directly addressed and fetched via type or a set of well-known metadata values (e.g. an HTTP GET that gives me all of Alice's public https://schema.org/SocialMediaPosting objects).
• Authorized parties can listen for new changes on objects they are authorized to have

Decentralized Twitter (Dewitter) Requirements List

Assumptions, Principles, Requirements, and Other Considerations

Abbreviations

• EDV = Encrypted Data Vault
• EDV App = Encrypted Data Vault-based App (Application or Service)

General Assumptions

Dewitter Use Cases: 1
A. User Case 1b. No centralized servers are required to Encrypted Data Vaults for phone, tablet, or laptop-based application
Actors, Roles (Personas), and Roles (Followers, Following, Neighborhoods)
Dewitter Use Cases: 2-4
B. Use Case 2. Users of an EDV-based application can have multiple accounts (identities) associated with themselves and their EDV Apps.

EDV Server Instances, EDV Data Vaults, and Containers

Dewitter Use Cases: 5
C. Use Case 5a. Local EDV Service Instances are capable of running standalone on a personal device (e.g. phone, laptop, or tablet).
D. User Case 5c. Each Local EDV Service Instance can host (have attached to it) 1 or more EDVs
E. Use Case 5d. Each EDV has the capability of storing resources into separate and secure resource Container (or the equivalent). An EDV can container 1 or more Containers.
F. Use Case 5x. Local EDV Service Instances automatically start and restart (after the device resumes from hibernation).

Personal Data Vaults and Containers

Dewitter Use Cases: 6
G. Use Case 6d. A resource is securely stored in a Container in an EDV.

Personal Agents

Dewitter Use Cases: 7
H. Use Case 7e. Personal Agents can talk to each other and exchange messages (e.g. using DIDCOMM).
I. Use Case 7f. A Personal Agent (or the equivalent) mediates all access to a personal Local EDV Service Instance, the attached EDVs, the Containers of resources contained in the EDVs.
J. Use Case 7h. Personal Agents automatically start and restart (after the device resumes from hibernation).

Dewitter Data Vault

Dewitter Use Cases: 8
• No requirement items

Dewitter Data Model Definitions

This section describes how tweets are stored in Dewitter’s fully decentralized architecture.
Dewitter Use Cases: 9-18 (Use Cases: 19-22 Reserved)
K. Use Case 10. Any resource, optionally, can have plain-text properties associated with the resource for the purpose of querying and retrieving a single resource or a collection of resources (Simple Resource Key).
L. Use Case 10. A resource property (Simple Resource Key) can have sub-properties (Resource Subkeys) – called a Resource Complex Key.
M. Use Case 10. A specific Container can be queried for a single resource or a collection of resources given a Simple Resource Key or a Complex Resource Key.
N. Use Case 14. A Local EDV Service Instance has an atomic, transaction-safe capability for updating the value of a property on a resource in a Container (e.g. incrementing the value of “counter” property on a resource).
O. Use Case 14. A Local EDV Service Instance supports a general-purpose, atomic, transaction-safe capability for updating the value of a property on a resource in a Container based on an arbitrary property value changing computation on the specific resource.
P. Use Case 15. A Local EDV Service Instance has a capability for storing, updating, querying, and retrieving stream resources to a client app or service (e.g. images, audio streams, video streams).

Personal Agent to Local EDV Server Instance Communications

Use Cases: 23
Q. Use Case 23a. Capability for a Personal Agent to directly access a Local EDV Server Instance running on the same device using the Layer B EDV Trusted Content Storage Services API (not via one of the Layer B Trusted Content Storage Service remote access service endpoints (e.g. HTTP, Bluetooth, etc.).
R. Use Case 23b. Capability for a Local EDV Server Instance to talk directly to the Layer A Trusted Content Storage Kernel via an API (which, in turn, talks directly to the EDV Microkernel).
S. Use Case 23c. An EDV Microkernel securely manages all access and operations against each of the attached EDVs.

Dewitter Protocol Operations

NOTE: A Compound Query is a query parameter that includes multiple Resource Keys (Simple and Complex) and their key values.
Use Cases: 24-35

Create/Update a Tweet Item or Stream Item in a Personal Local EDV Service Instance

T. Use Case 24a. Capability to create a new resource in a Container in a Data Vault attached to the Local EDV Service Instance.
U. Use Case 24b. Capability to update a new resource in a Container in a Data Vault attached to the Local EDV Service Instance.
V. Use Case 24c. Capability to tombstone a new resource in a Container in a Data Vault attached to the Local EDV Service Instance.
W. Use Case 24x. Capability to ensure no one but the owner can update a resource in a Container in the Data Vault attached to the Local EDV Service Instance.

Query a Personal Local EDV Server Instance for a List of Tweet Keys or a Collection of Tweet Items

X. Use Case 25a. Capability to query a Local EDV Server Instance for a list of Resource Keys given a Simple Resource Key or a Complex Resource Key or a Compound Query.
Y. Use Case 25b. Capability to query a Local EDV Service Instance for a list of Resource Items given a Simple Resource Key or a Complex Resource Key or a Compound Query.
Z. Reserved

Query Another Personal Agent or List of Personal Agents for a List of Tweet Keys

AA. Use Case 26a. Capability to query another Personal Agent for a list of Resource Keys given a Simple Resource Key or a Complex Resource Key or a Compound Query.
BB. Use Case 26b. Capability to query a List of Personal Agents for a list of Resource Keys given a list of Simple Resource Keys or a Complex Resource Keys or a Compound Queries.

Query Another Personal Agent or List of Personal Agents for a List of Tweet Items

CC. Use Case 27a. Capability to query another Personal Agent for a list of Resource Items given a Simple Resource Key or a Complex Resource Key or a Compound Query.
DD. Use Case 27b. Capability to query a List of Personal Agents for a list of Resource Items given a list of Simple Resource Keys or a Complex Resource Keys or a Compound Queries.

Block a Query from Another Personal Agent

EE. Use Case 28a. Capability for a receiving Personal Agent to block the receipt and acceptance of a query for a resource from another Personal Agent.

Query a Personal Local EDV Server Instance for a List of Stream Keys or a Collection of Stream Items

FF. Use Case 29a. Capability to query a Local EDV Server Instance for a list of Resource Keys given a Simple Resource Key or a Complex Resource Key or a Compound Query.
GG. Use Case 29b. Capability to query a Local EDV Service Instance for a list of Resource Items given a Simple Resource Key or a Complex Resource Key or a Compound Query.

Query Another Personal Agent or List of Personal Agents for a List of Stream Keys

HH. Use Case 29y. Capability to query another Personal Agent fora list of Resource Keys given a Simple Resource Key or a Complex Resource Key or a Compound Query.
II. Use Case 29z. Capability to query a List of Personal Agents for a list of Resource Keys given a list of Simple Resource Keys or a Complex Resource Keys or a Compound Queries.

Query Another Personal Agent or List of Personal Agents for a List of Stream Items

JJ. Use Case 30a. Capability to query another Personal Agent for a list of Resource Items given a Simple Resource Key or a Complex Resource Key or a Compound Query.
KK. Use Case 30b. Capability to query a List of Personal Agents for a list of Resource Items given a list of Simple Resource Keys or a Complex Resource Keys or a Compound Queries.

Update a Resource Item in Another Personal Local EDV Service Instance

LL. Use Case 35. Capability, given a Resource Key, update an existing Resource Item in a Container in an EDV Data Vault attached to a personal Local EDV Server Instance.

Dewitter Use Cases

Use Cases: 36-51
MM. Use Case 48. A Local EDV Service Instance has an atomic, transaction-safe capability for updating the value of a property on a resource in a Container (e.g. incrementing the value of “counter” property on a resource). (Same as Requirement Item N.)

Notes on Notes

Dewitter Notes: 1-4
NN. Note 1. Support for offline Personal Agents.
OO. Note 2. Support for offline Local EDV Server Instances.
PP. Note 3. For secure logging on an EDV-by-EDV basis: all actions against the Containers in a particular EDV stored in a secure Logging Container for that EDV.
QQ. Note 3. Support for EDV-to-EDV replication at the Container-by-Container level.

DO NOT EDIT BELOW

Identity Hub Requirements OLD PLEASE DONT EDIT - FOR POSTERITY

• 1. Add support for DID Auth and encryption/capabilities based on the pub key refs in a DID Doc.
  • DID Authn - authn - Not EDVs, possibly Hubs
  • Encryption - in scope, EDV clients will do this
  • capabilities - in scope, EDVs will do this = Layer A - Core Secure Content Storage + Layer B - Core Secure Content Storage Service/API?
  • EDVs will not support DID Auth (no authn, authz only) (DZ: I suspect DID Authn will be a Hubs thing.)
  • EDVs will do authz via zcaps, but can Hubs reuse this?
  • EDVs will provide encryption in transit and at rest using public keys
  • Out of scope for EDVs and Hubs (this is an application):
    • Allow Alice to purchase an EDV account using a DID instead of an email address and password as her identifier. Either way, Alice needs to send the EDV host a payment such as a credit card.
  • Objections: None
• 2. Multi-instance peer-to-peer operation, with active/active replication between instances.
  • Replication = Layer D - Replication Services?
  • EDVs - pluggable replication, but no peer-to-peer replication in EDV Spec v1
  • Possible to do peer-to-peer replication using EDV clients – Hubs could utilize that capability OR Hubs could provide their own pluggable replication
  • Two EDVs can reach eventual consistency if they are connected, aware of each other’s service endpoint, and have a filter in place.
  • Concern: Need to clearly delineate responsibility for which aspects / modes of replication EDVs handle, and which aspects Hubs handle. (can't have them clashing.)
  • Objections: None
• 3. The ability to differentially replicate subsets of the logical whole dataset across different instances.
  • Replicate - Layer D - Replication Services?
  • No support for EDVs in v1
  • EDVs - only supported through EDV client-based replication (with access to document metadata).
  • Two EDVs (servers) can do filtered replication based on a scope schema.
    • Possible through EDV encrypted indexes
  • EDVs can use encrypted indexes + replication rules to determine what goes where; if we support configuring filtered
    replication on an EDV server, the server could ensure certain documents are replicated to other EDVs when
    encrypted attributes match
  • Objections: None
• 4. Three modes for data: public plaintext data, shared encrypted data, private encrypted data.
  • Three modes for data - Layer A - Core Secure Content Storage + Layer B - Core Secure Content Storage Service/API?
  • Not EDVs - no support of public plaintext data, Hubs or some other higher layer does that
  • EDVs - support for shared encrypted data and private encrypted data
  • An EDV document or database entry can be unencrypted pointer can include the decryption key (hashlinks-style) decryption key is passed out of band
  • Objections: Daniel (extreme disappointment, not objection - smells bad), Adrian (confused), Michael (concerned that we may not require it to be implemented)
• 5. Multi-writer ingest, with the ability for the owner of the datastore to authorize any entity to write that they choose.
  • EDVs - Yes, supports multi-actor reads, writes, updates, deletes (all operations)
  • Will EDVs have ACID transactions?
    • No, just transactional integrity across single resource, single EDV, single instance
    • Might not even be at the Hub layer, perhaps higher (OR lower, probably lower)
  • Can authorization be attenuated to just reads or just writes, or r/w?
  • EDV access authorization supports Alice-to-Bob use-cases where Alice and Bob are different DID controllers using different clients or user agents.
  • Can attentuation utilize the indexes? Don't know, need more discussion
  • Objections: None
• 6. Ability to limit writes/reads by external entities. An example of this would be Alice, the datastore owner, being able to limit writes and reads of objects she specifies to Bob and others, in keeping with access and permissioning criteria she sets.
  • EDV access authorization supports arbitrarily fine-grained scopes.
  • Layer C - Authorization Service + Layer B - Core Secure Content Storage Service/API?
  • Additional authorization might happen at the Hub layer – TBD
  • Objections: None
• 7. Ability to keep private objects out of public circulation.
  • All EDV access requires authorization
    • is protected by a policy enforcement point (don’t count on encryption alone to keep objects out of public circulation).
      • Q: When things are protected via PEP, encryption is just a matter of policy? A: Encryption is to prevent host from seeing data, JWE's ar e used; that's a part of the spec
    • Concerns: Why "encrypted" other than to protect vendor/host?
  • Layer C - Authorization Service? plus?
  • Objections: None
• 8. Support for both Last-Write-Wins complete replacement of objects and objects that can be seamlessly merged via conflict-free replicate data mechanisms.
  • EDVs will support complete replacement of objects
  • EDVs will support version history for objects
  • EDVs will NOT support CRDTs for v1.0, but can support CRDTs if the data structure is built on top of the base layer of storage
  • Layer D - Replication Services - Inbound Processing - Conflict Resolution?
  • We will discuss EDVs supporting a notification mechanism that allows p2p replication conflicts to be adjudicated by a user or the user’s agent.
  • Objections: None
• 9. Implementations that can run in Web platform, native/mobile app, and remote server environments.
  • PROPOSAL: Skip this for now
  • NOTE: Even though we're starting with an HTTP API, the goal is to expose the operations and interfaces through other types of APIs (local native APIs, Bluetooth, NFC, etc.)
  • NOTE: Could be a EDV-kernel-like interface (e.g., C-based API callable through a variety of mechanisms)
  • Deployment/Supported Platforms? …not a layer thing
  • EDV accounts are substitutable across multi-tenant cloud hosts and native, occasionally connected mobile device hosts.
  • EDV accounts do not presume or benefit from access to a HSM or IntelSGX infrastructure.
  • Objections: None (for skipping, but with notes)
• 10. An API that provides inferential queries on semantic data can be directly addressed and fetched via type or a set of well-known metadata values (e.g. an HTTP GET that gives me all of Alice's public https://schema.org/SocialMediaPosting objects).
  • EDVs will not support this feature (except to the degree that using an encrypted index can address this use case).
  • Hubs (or some higher layer) are expected support this feature
  • Layer B - Core Secure Content Storage Service/API + Layer D - Search Query Service? Also Deployment
  • Objections: None
• 11. An authorized user can search against an EDV index.
  • EDVs will support searching on encrypted indexes in v1
    • Applies specifically to encrypted indexes
    • Indexes are HMAC'd attributes that sit beside the JWEs
  • Objections: None
• 12. Logging of the lowest level operations of the data vault (e.g. for the purposes of auditing)
  • We should have a future call on logging for EDVs and logging for Hubs
  • Objections: None
  • GitHub issue for documenting positions: https://github.com/decentralized-identity/confidential-storage/issues/186
• 13. Replication concerns based on the above?
  • We should have a future call on replication for EDVs and replication for Hubs
  • Github issue for documenting positions (and proposals)
  • Objections: None
• 14. The ability for authorized parties to listen for new changes (via feed, pub/sub, etc.) on public objects and encrypted objects (individually or sets) they are authorized to have.
  • Documentation for the Github issue: https://github.com/decentralized-identity/confidential-storage/issues/202
    • NEED: We need to have a call on sequence numbers and resources before we can have this discussion.
    • EDVs will have the ability to detect and track changes along these lines in v1
      • "I'm at sequence #4, give me every change from there."
      • Update sequence numbers are used to track changes to any and all objects in the EDV
      • Version history is just the addition of additional objects that are linekd together to produce the version history?
      • Version 1 will define identifiers related to changes over time for resources
      • Version 1 will define a data model for a change
    • EDVs might expose a more complex history API w/ ability to rewind to certain changes later
      • Fundamental to a stream API "I need to catch up before I stream"
    • EDVs might expose a mechanism to provide a feed of changes to objects that a subscriber has access to later
      • What happens when you connect and miss changes?
    • Layer C - Authorization Service + Layer D - Replication Services - Outbound Processing?
      • Authorized parties (Bob) that own their own EDV account somewhere can use the filtered replication feature of EDVs as a way to listen for changes to Alice’ EDV.
  • We need one or more github issues raised on this particular feature with a concrete proposal for providing this feature in EDVs or Hubs before we can tackle this requirement
  • Objections: None