ADR-002 v.2 - HackMD

--- title: ADR-002 v.2 id: adr-002 displayed_sidebar: resourcesSidebar --- - **Author(s):** [Brayden Cloud](https://github.com/mcjcloud), [Josh Long](https://github.com/joshLong145) - **Last Updated:** June 21, 2022 - **Status:** Draft - **Deployment:** `integrate/adr-002` # ADR-002: IPLD Objects With JOSE [![hackmd-github-sync-badge](https://hackmd.io/iomfscGXR1ycI_67NrLvMw/badge)](https://hackmd.io/iomfscGXR1ycI_67NrLvMw) ## Summary Implement Object Schemas based on [IPLD](https://ipld.io) which support [JOSE](https://jose.readthedocs.io/en/latest/) operations. ## Abstract Objects on the Sonr network provide a way for users and applications to persist data. The problem with modern software architectures is that the security of user data is not guaranteed—a problem Sonr aims to solve with the help of JavaScript Object Signing and Encryption (JOSE). Additionally, today’s software stack is not built with interoperability in mind. An entire layer must be added to any application to support communication with other applications. IPLD provides a standard and decentralized way for any motor node to make use of the data stored on Sonr. The combining of these technologies begets a system in which user data is always encrypted and can never be owned by one application, not even the application that created it. ## Primer The content of this ADR is based on the works of ADR-001 as well as a few core technologies—all of which should be well understood before reading ADR-002. Here is a list of resources. * [ADR-001](/articles/reference/adr-001) * [IPLD Primer](https://ipld.io/docs/intro/primer/) * [IPFS Concepts](https://docs.ipfs.io/concepts/) ## Glossary #### `JSON Web Signature (JWS)` JavaScript Object Notation (JSON) data structure that represents a cryptographic key. This specification also defines a JWK Set JSON data structure that represents a set of JWKs. Cryptographic algorithms and identifiers for use with this specification are described in the separate JSON Web Algorithms (JWA) specification and IANA registries established by that specification. #### `JSON Web Encryption (JWE)` JSON Web Encryption (JWE) represents encrypted content using JSON-based data structures. Cryptographic algorithms and identifiers for use with this specification are described in the separate JSON Web Algorithms (JWA) specification and IANA registries defined by that specification. Related digital signature and Message Authentication Code (MAC) capabilities are described in the separate JSON Web Signature (JWS) specification.  #### `JSON Web Key (JWK)` A JSON Web Key (JWK) is a JavaScript Object Notation (JSON) data structure that represents a cryptographic key. This specification also defines a JWK Set JSON data structure that represents a set of JWKs. Cryptographic algorithms and identifiers for use with this specification are described in the separate JSON Web Algorithms (JWA) specification and IANA registries established by that specification. #### `Schema` A type definition for data stored in Sonr. These structures conform to a standard of key value pairs where the key is a string and the value is one of a set number of `Types`. #### `Identity` An identity is derived from the user's provided key upon registration. ## Objective The objective of ADR-002 is to define how Sonr manages the creation and access of data on its network. A proper solution must have these attributes. * Enable _motor_ nodes to store data on behalf of a user. * Enable devices which belong to the same user to access the same data. * Enable certain data to be shared with other users. * Allow for revoking access to both devices and other users. * Maintain that all data is encrypted in transit and at rest. ## Changes From ADR-001 To accommodate the features proposed by ADR-002, a few changes to terminology will be made from ADR-001. #### Rename **Object** to **Schema** _Object_ in ADR-001 represents a type definition in Sonr. This name is misleading as it creates ambiguity between the type definition and the data which is of that type. With the addition of IPLD, the term _Schema_ will better express the concepts that were represented by _Object_ while creating consistency with the libraries we employ. ## Client-Side Encryption with JOSE A core principle of Sonr is to never let private user data pass unencrypted through the network. To achieve this, the _motor_ node must be responsible for encrypting and decrypting all data. This section describes how a user may securely store data on Sonr and maintain the ability to access that data from any _motor_ associated with their account. ### Encryption In order for encrypted data to be accessed from multiple devices, there must be a [shared key](https://en.wikipedia.org/wiki/Symmetric-key_algorithm) for each piece of data that will be accessible to each _motor_. #### Key Creation The [JWK](https://jose.readthedocs.io/en/latest/#jwk) used during the encryption process is determined by the access levels for the data. As the owner of data, the user `bucket` will contain a map of encrypted JWKs: `map[CID]map[PK]JWK` [henceforth referenced to as the Object Encryption Key Map (OEKM)]; where CID is the CID of the object, PK is the public key of a member who has access, and JWK is the JWK used to encrypt the data. For objects shared with other users, the OEKM may exist in a dedicated bucket, but this is an implementation detail for the developer. #### Client 1. An object is created using a _Schema_—this is what will be stored. 2. The _motor_ encrypts the data using its shared key for private data. If the data is meant to be shared with a party, a key specific to that party is used. * Any property in the data that is represented as a separate _Schema_ will be encrypted separately and included with the POST request. * Each encrypted _schema_ will also have a corresponding CID. Because CIDs are generated predictably, this will be computed by the _motor_. 3. The _motor_ then creates a copy of the key it generated and encrypts it with each public key in the root-level DID. This is ultimately what allows another _motor_ to decrypt the data. 4. A request can now be crafted to Highway with the encrypted data and set of encrypted keys. * The request should include each encrypted _schema_ along with the list of encrypted shared keys. [![Writing encrypted data](https://i.imgur.com/aPIJLSe.png) ](https://www.figma.com/file/URTnRygutbDRaSK0swKVyV/Reading%2FWriting-Encrypted-User-Data?node-id=190%3A148) #### Server 1. When a request is received, each object is added to IPFS. 2. The CID for each object will be verified, creating an error if one does not match. 3. If successful, each CID is added to a record on-chain. Alternatively, each CID may be be part of a bucket update and not stored individually on-chain. ### Decrypting The process for decrypting data becomes simple with the above provisions. The _motor_ will first obtain the shared key—accessible via the _vault_ (more on that later)—which will be used to encrypt data fetched through IPFS. The entire process is as follows. 1. Fetch the shared key and decrypt it. * The shared key will have already been added to the _vault_ by the uploading device. This process is outlined in the next section. 2. Through IPFS, fetch the relevant data via CID. Then, fetch each referenced CID in the result. Continue this process until all objects have been recovered. 3. Decrypt each object using the shared key. 4. Compose the original _schema_ by replacing the IPLD Link with the referenced object. ### Revoking Access If any device or party is to have its access revoked, a new key must be used for the next update to that object. This means the process for revoking access is to simply upload the object again, with the revoked party omitted from the OEKM. #### A Note on the Vault Each user on Sonr will have a bucket created for them when they create an account, known as their _vault_. The vault acts as a collection of all the user's personal keys or private data. Buckets and vaults will be covered in great detail in ADR-003. ## Adding Verification Methods The process of securing user data starts with the verification method. Simply put, verification methods are a member of the DID Document which contain the public keys used for authentication. New verification methods can be added in order for a user to authenticate with new or different keys. Later sections will detail how these methods are used to create JWKs for secure data storage. :bulb: The important thing to note about Verification Methods in ADR-002 is that the process for adding a Verification Method must now include encrypting all shared keys again with the new method’s public key. ## Schema Model A _schema_ added to Sonr will have the following format. ```go= message SchemaReference { // the DID for this schema string did = 1; // an alternative reference point string label = 2; // a reference to information stored within an IPFS node. string cid = 3; } message SchemaDefinition { // Represents the types of fields a schema can have // the DID for this schema string creator = 1; // an alternative reference point string label = 2; // the properties of this schema repeated SchemaKindDefinition fields = 3; } message SchemaKindDefinition { string name = 1; // Type of a single schema property SchemaKind field = 2; } type SchemaKind int32 const ( SchemaKind_Invalid SchemaKind = iota SchemaKind_Map SchemaKind_List SchemaKind_Unit SchemaKind_Bool SchemaKind_Int SchemaKind_Float SchemaKind_String SchemaKind_Bytes SchemaKind_Link SchemaKind_Struct SchemaKind_Union SchemaKind_Enum SchemaKind_Any ) ``` #### DID The DID that uniquely identifies the Schema. #### Label A name for this schema. #### Fields A list of properties for this schema. The property types are defined as `SchemaKind`. Each SchemaKind maps to an `IPLD` type. ## Object Model Object data on Sonr will be encrypted, but IPFS will contain a wrapper of that data containing some additional metadata. `CID` is a look up to data stored within IPFS. ```go= type Object struct { Did string Cid string Creator string SchemaDid string EncryptedJWKs map[PublicKey]EncryptedJWK } type PublicKey string type EncryptedJWK string ``` #### SchemaDid The DID of the schema this object adheres to. #### Content The raw content of this object. May be encrypted with a JWK. #### EncryptedJWKs A mapping of each public key that has access to the data with the JWK encrypted with said public key. ## Object Schema API (Blockchain) The following methods will be added to support creating Schemas and Objects. ### `CreateSchema(label, fields)` #### Parameters `label (string)` : The label for the Schema. Must be alphanumeric and without whitespace. `fields (map[string]int32)` : The properties to be included in the Schema. Each key must be alphanumeric and without whitespace. The value is an integer corresponding to a `SchemaKind`. #### Description Creates a _schema_ for use throughout Sonr. Creating a _schema_ requires a DID and DID Document to be created. > The model itself will be stored in IPFS, while the DID-CID pair is stored on-chain. ### `GetSchema(did)` #### Description Returns the schema. Requires no blockchain transactions. #### Parameters `did (string)` : The DID of the schema. ### `UploadObject(schemaDid, data, encryptedJwks)` #### Parameters `schemaDid (string)` : The DID of the schema this object adheres to. `data ([]byte)` : The raw data to be added to IPFS. This can be plaintext or encrypted with the user's root JWK. `encryptedJwks (map[PK]JWK)` : A map of encrypted JWKs for anyone who should have access to the data. Each JWK must be encrypted with the PK of the shared party. #### Description Uploads raw data and adds to it IPFS. A CID is returned for addressing the content. > There is no blockchain interactions with this call. ### `DeprecateSchema(did)` ### `GetObject(cid)` #### Parameters `cid (string)` The CID which addresses the content on IPFS. #### Description Returns the object previously uploaded to IPFS, including the map of JWKs. The map can be used to look up and decrypt the appropraite JWK in order to read the data. > This call requires no blockchain interaction. ## WhoIs Updates ``` go // Metadata attached associated with the document // data is arbitrary, and thus user specified. Metadata map[string]string ``` ## DID Document Verification Definition ```json // Connected Motors Webauthn credentials get stored as verificationMethod { "verificationMethod": [ { // Set to Motor Nodes Wallet Address "controller": "did:snr:123", // Id of Key set to unique value and operating system/architecture "id": "did:snr:123#ios-arm64-1", // JWK generated from WebAuthN Credential "publicKeyJwk": { "crv": "P-256", "kty": "EC", "x": "UANQ8pgvJT33JbrnwMiu1L1JCGQFOEm1ThaNAJcFrWA=", "y": "UWm6q5n1iXyeCJLMGDInN40bkkKr8KkoTWDqJBZQXRo=" }, "type": "JsonWebKey2020" } ] } ```  ### IPLD Codec DAG-JOSE is a codec that implements the IPLD Data Model as a subset of `JOSE`, which complies with the `DAG-CBOR` specication. Some additional constraints for hash consistent representations. DAG-CBOR also adds a "link" type using a CBOR tag to bring it in line with the IPLD [Data Model](https://ipld.io/glossary/#data-model). [go implementation](https://github.com/ceramicnetwork/go-dag-jose) ### IPLD Pathing Paths are composed of a series of segments, and each segment is an instruction on how to navigate deeper into the filesystem. With filesystems, each step is over a "directory" and leads you to either a "file" or another "directory"; for IPLD, each step is over a "node" and leads you to another "node"! ## DID Methods for CBOR-JOSE support The following methods will need to be implemented within **sonr-io/sonr/pkg/did** for DAG-JOSE support for Sonr DID Documents. * Authenticate * CreateJWS * DecryptJWE * EnctyptJWE * VerifyJWS **Models** ``` CreateJWSOptions { did: string protected: map[string]interface{} (optional) linkedBlock: string (optional) } ``` ``` DagJWS = { payload: string signatures: []JWSSignature link: CID (optional) } ``` ``` VerifyJWSOptions { atTime: Date (optional) disableTimecheck: boolean (optional) issuer: string (optional) capability: Cacao (optional) revocationPhaseOutSecs: number (optional) } ``` ``` EncryptJWE { cleartext: []byte, recipients: []string, protectedHeader: map[string]interface{} aad: []byte (optional) } ``` ``` DecryptJWEOptions { did: string (optional) } ``` ## Methods ### Authenticate #### Description DID Authentication communicates with the remote origin of DID document to gain the `JWS` or `siganture`. Said `JWS` is then `verified`. the following operations are important for proper authentication compliance - Check the returned `kid` includes the did from the verification payload. - Check experation of the signature #### Parameters **provider** - DIDProvider. **paths** - didUrl(s) to authenticate **aud** - Audience (JWS claim) #### Returns A JWS with general serialization containing the following properties: `nonce` : The random string which was given as a challenge `did` : The DID which authentication was given for `paths` : The paths which was given permission for `exp` : Unix timestamp after which the JWS should be considered invalid. `aud` : optional audience for the JWS, should match the domain which made the request. ### CreateJWS Generates a JSON web signature object composed of a multi signature key. ```json { "link": { "/": "bafybeig6xv5nwphfmvcnektpnojts33jqcuam7bmye2pb54adnrtccjlsu" }, "payload": "AXASIN69ets85WVE0ipva5M5b2mAqAZ8LME08PeAG2MxCSuV", "signatures": [ { "protected": "eyJhbGciOiJFUzI1NksifQ", "signature": "SiYGXW7Yi-KxbpIlLNmu0lEhrayV7ypaAC49GAcQ_qpTstZW89Mz6Cp8VlUEX-qVsgYjc-9-1zvLcDYlxOsr1g" }, { "protected": "eyJhbGciOiJFUzI1NksifQ", "signature": "Q8PdTE5A5N3a0ktO2wNdUymumHlSxNF9Si38IvzsMaSZC63yQw-bJNpKf-UeJFPH7cDzY7jLg2G_viejp7NqXg" } ] } ``` ### Params `payload` : The payload to sign, json object or base64url encoded string. `protected` : The protected header, json object. `did` : The DID that should sign the message, may include the key, fragment. `revocable` : Makes the JWS revocable when rotating keys, boolean default to false **Returns**: `DAGJws` ### VerifyJWS ### Description We can compute the `SHA-256` representation if the specified `secret` is present within the generated token. Upon querying for an ecrypted entry within ipfs, the DID controller of said object should be able to `verify` the signature as it should be aware of valid signatures relating to itself. ### Params `jws` : signature to verify `options` : VeriftyJWSOptions **Returns** # Object Schemas ## IPLD Object Schema IPLD represents data types which can be stored in IPFS nodes as “Kinds” the following are types which “Kinds” support: * `boolean` * `integer` * `float` * `map` * `list` * `string` * `null` * `bytes` * `link` Kinds in IPLD are similar to data types supported within “JSON” but also add “bytes” and “link” where link maps to “CID” type. ## DSL Representation of Object Schema Node builders allow you to declare schema within an object representation that then can be transformed to various types. Builders then can be represented in DSL which is a declarative syntax for IPLD schemas similar to JSON. The following is a method stub for creating an object schema for later usage. ## CreateObjectSchema Creation of an object will define a schema for objects to be later stored in IPFS. `Schemas` define the shape of data for a given object. ## PutObjectSchema ``func (i *IPFSProtocol) PutObjectSchema(doc *ot.ObjectDoc) (*cid.Cid, error)`` ### Implementation using IPLD Node Builder Stores a DSL representation of a defined object schema with the following relationship denoting a property of the persistent schema. ``` { Label: string Type: Kind } ``` This object will be stored within an aggregate (array structure) within the ObjectDoc and should be iterated over and stored within a node builder, then serialized to a byte representation. ## GetObjectSchema  ### Function Signature ```func (i *IPFSProtocol) GetObjectSchema(cid *CID) (Node, error)```  ### Implementation (string repersentation of schema) Instead of using a `node builder` to build our schema up from a tree of related objects, the schema structured can be `loaded` from a string repersentation of the `ipld` schema: ``` ' { "foo": string, "baz": { "prop": integer } } ' # Keepers A Keeper is an abstraction that manages access to the subset of the state defined by various modules. The `x/Schema` `Keeper` holds functionality for accessing and persisting `Schemas` on chain. These Schemas can also be queried for by functionality present with this `Keeper` Both `message` endpoints and `Query` endpoints are accessible through `grpc` and the `cli` The following endpoints are both accessible through `GRPC` and the `sonrd` cli see [here]() for information on running an instance of the `sonrd` block chain locally ## Keeper Message Enpoints ### `CreateSchema(SchemaDefinition)` Register's a new type definition for a given application. this type defention is then asserted against when uploading content ```Text (`string`) Creator (`string`) Label (`Map`) Fields ``` - `Creator` The identifier of the application the schema is registering for - `Label` The human readable description of the schema - `fIelds` The data `Schema` being persisted see [here]() for schema data types Returns a `WhatIs` --- ### `DeprecateSchema(MsgDeprecateSchema)` Allows for Schemas to be depricated. Depricated schemas are still accessible but will allow schemas developers to indicate it is no longer supported. ```gp { Creator string Did string } ``` - `Creator` The Account Address Singing this message - `DID` The identifier of the Schema Returns a `status code` and `message` detailing the operation. ## Usage ### `CreateSchema` `GRPC` ```bash $ grpcurl -d '{"creator": snr1234, "Label": "Message schema v1" "fields": {"message": 0, "icon": 2}}' \ sonrio.sonr.schema.Msg/MsgCreateSchema ``` ### `DepicateSchema` `GRPC` ```bash $ grpcurl -d '{"creator": snr1234, "did": "did:snr:123"}}' \ sonrio.sonr.schema.Msg/MsgCreateSchema ``` ## Query Methods Query methods are used for accessing state kept within the `Keeper` ### `QuerySchema` For cases it is dersired to lookup a Schema Definition for verifying on an uploaded object. ```go { Creator string Did string } ``` - `Creator` identifier of the schema owner which will be a `Application` - `DID` identifier of the schema being queried for Returns a `SchemaDefinition` --- ### `QueryWhatIs` For cases where applications need to verify existing data, or verify a schema belongs to a certain `Creator`. `QueryWhatIs` should be used over `QuerySchema` when only verification of the data is needed. ```go { Creator string Did string } ``` ` - `Creator` identifier of the schema owner which will be a `Application` - `DID` identifier of the schema being queried for Returns a `WhatIs` --- ## Usage ### `QuerySchema` `GRPC` ```bash $ grpcurl -d '{"creator": snr1234, "did": "did:snr:123"}}' \ sonrio.sonr.schema.Query/QuerySchema ``` ### `QueryWhatIs` `GRPC` ```bash $ grpcurl -d '{"creator": snr1234, "did": "did:snr:123"}}' \ sonrio.sonr.schema.Query/QueryWhatIs ``` # Interaction Methods for Highway to Motor Communication The following are interface methods that should be defined on `Host` and implemented for Host targets. ## `AccessApp()` User authenticates a Registered Application on Sonr with their DID Based Multisignature key for all their devices. Creates a new Bucket inside the User bucket for the newly provisioned Application. ## `InteractObject()` Maps user defined object to a `object schema`. Which is then pushed to a bucket. ## `ReadObject()` This method begins the request process for reading the individual object values of another User's app bucket. Provisioned users will automatically get access (if they called RequestBucket already) and unprovisioned users will send the RequestBucket to the corresponding peer. # Messages ### `CreateSchema(SchemaDefinition)` Register's a new type definition for a given application. this type defention is then asserted against when uploading content ```Text (`string`) Creator (`string`) Label (`Map`) Fields ``` - `Creator` The identifier of the application the schema is registering for - `Label` The human readable description of the schema - `fIelds` The data `Schema` being persisted see [here]() for schema data types Returns a `WhatIs` --- ### `DeprecateSchema(MsgDeprecateSchema)` Allows for Schemas to be depricated. Depricated schemas are still accessible but will allow schemas developers to indicate it is no longer supported. ```gp { Creator string Did string } ``` - `Creator` The Account Address Singing this message - `DID` The identifier of the Schema Returns a `status code` and `message` detailing the operation. ## Usage See `Keepers` section for examples of message endpoint usage. ```gp { type: 'category', label: 'Schema', items: ['build-apps/modules/schema/overview', 'build-apps/modules/schema/state', 'build-apps/modules/schema/keepers', 'build-apps/modules/schema/messages', 'build-apps/modules/schema/events', 'build-apps/modules/schema/client'], collapsible: true, collapsed: true, }, ```