owned this note
owned this note
Published
Linked with GitHub
# Sync protocol
* Discovery phase
* a json blob found during MDNS discovery, on an HTTP request, or on a USB stick
```js
{
"version": Number,
"deviceName": String,
"deviceType": Enum,
"projects": [
{
"projectId": String,
"deviceId": String,
"data": { // Optional
"hypercores": [
"key": String, // public key for the hypercore
"seq": String // highest sequence number known
],
"blobs": [
....
]
}
}
]
}
```
* Stream: handshake & data
* API should expose raw handshake data
* diff of data -- who has which pieces of data?
* size
* multifeed sync already sends all of the hypercores and the highest sequence number, we can make a good estimate
* all the media and file sizes, and do a diff
# API
* ``/projects?data=Boolean``
* list of projects
* ``/projects/:projectId?data=Boolean``
* metadata about a project
# Projects
A project is a directory with the following structure.
* __tiles__: contains the subdirectories
* __media__: a blob store
* __data__: a multifeed
## data
A multifeed which the hypercores of all of the other devices that you've synced with. All of the data is encoded as JSON.
### Common Attributes
Most key-value objects in the database share these common attributes. Objects that share these attributes are listed below with `Common &`
```js
{
id: String, // Unique ID (most are 64-bit but not guaranteed)
version: String, // hypercoreID + Seq
createdAt: DateTimeString, // ISO 8601 when document was first created
modifiedAt: DateTimeString, // ISO 8601 when this version created
userId: String, // optional 64-bit ID of user record
type: String, // Type identifier, defines the schema and used by indexed
links: Array<String>, // Array of versionIds
schemaVersion: Number, // Integer starting at 1, defines schema, along with the `type`
value: {
...
}
}
```
#### `value` property
The contents of the `value` property is the object that is written by the client. The top-level properties are managed by the Mapeo core database. This is done so that in the client API `get()` returns the entire object but `update()` only passes the contents of `value`. The top level properties are special because:
* `id` must not change with an edit.
* `version` is assigned by the backend
* `createdAt` is assigned by the backend (and can never change)
* `modifiedAt` is assigned by the backend
* `userId` if this was left on the `value` prop it could easily be forgotten about and returned with the update. The client app should be explicit about setting the userId, and if there is no userId then it should be undefined. If this property was on `value` then the client would need to delete `value.userId` and could forget to change it.
* `type` is an enum which defines the schema of the object and must not change
* `links` is assigned by the backend if not passed by the client (the default if missing is to merge known heads). Similar to `userId` we do not want clients to accidentally pass back the same links in an update as received by a get(). The client should always explicitly set this.
* `schemaVersion` is defined by the backend. All writes to the database must follow the most recent schema supported by the backend. The backend will always transform data from previous schema versions and pass as the most recent, so the schemaVersion passed to the client will always be the latest.
#### API ideas
* Namespace api by object type, e.g. `api.node.get()`
* `api[objectType].get()` returns the top-level object.
* `create()` and `update()` pass only the value as an argument, and optional `opts.userId` and `opts.links`.
### observation
```js
Common & {
type: "observation", // enum
value: {
lat: Number, // can be null or undefined
lon: Number, // can be null or undefined
metadata: { // Cannot guarantee any properties here
},
refs: [{
id: String // id of node/way that this obs is about
}],
attachments: [{
id: String, // Identifier to find the image
type: String, // mime type e.g. image/jpg
}],
tags: {}
}
}
```
### user
```js
Common & {
type: "user",
prevIds: [String], // Array of user IDs if users are merged (e.g. if a user is created twice on two different devices)
value: {
fullname: String // User name
}
}
```
### device
```js
Common & {
id: String, // The id for a device is the hypercore ID
type: "device",
value: {
displayName: String // User defined name of device,
deviceType: Enum // One of: `'cloud', 'mobile', 'desktop'`
}
}
```
#### deviceType
One of: `'cloud', 'mobile', 'desktop'`
### sync
Written by both sides after a successfully completed sync. (should we also write this after a failed / errored sync?)
**Note**: Sync events do not have a
```js
{
type: "sync"
deviceId: String, // ID of the other device successfully synced to
timestamp: DateTimeString // DateTime when sync successfully happened
}
```
The author's deviceID isn't written because it can be derived by looking at the hypercore ID (public key) of the record's author.
### node
```js
Common & {
type: "node",
}
```
### way
### changeset
### relation
### presets & tiles
punting on presets and tiles