owned this note
owned this note
Published
Linked with GitHub
# Ceramic core-devs call - Biweekly Meetings
[📅 Meetings calendar](https://calendar.google.com/calendar/u/0/embed?src=c_4dae4b001a353ace604a8f3fd1a41ec718b0e18f1d12c64fb7fd4b61b22283ab@group.calendar.google.com&ctz=Europe/Lisbon)
<!-- Template
# YYYY-MM-DD
### Agenda
* CIP discussions
-
* Ecosystem updates
- Us3r network
- Orbis
- Hirenodes
- 3Box Labs
* Open floor
### Attendees
### Notes
---
-->
# 2024-05-02
## Agenda
* Important announcements
* Ecosystem updates
* Protocol updates
* Upcoming events
## Notes
### Initial Announcements
None to speak of today.
### Ecosystem Updates
#### Orbis
- Main focus is finalizing migration from existing solutions to the new OrbisDB setup
- Main lift is recreating plugins from Orbis Social in OrbisDB
- Working on a concept of "presets" in OrbisDB that essentially pre-specify sets of models, plugins, and views to pre-load on a node. Will be an easy way for users of existing products to get a familiar experience up and running ASAP.
- Estimating another 2-3 weeks of work.
- Could imagine many such presets. They're currently living in the OrbisDB repo, contributors welcome.
- Presets are somewhat analogous to ComposeDB composites and could be a means of "distributing" configurations for participants in an application or ecosystem.
#### Points SDK (3Box Labs)
- [SDK blog](https://blog.ceramic.network/web3-points-library/) distributed 5/2
- A follow-up tutorial (walk-through of one of the example apps that uses the SDK) is already drafted and will go live alongside a YouTube video, likely week of 5/5
- New [example app](https://github.com/ceramicstudio/solutions-sdk/tree/main/demo/fullstack-app) added to SDK (this is what the tutorial and video mentioned above will walk through)
- The [ComposeDB Sandbox](https://developers.ceramic.network/docs/composedb/sandbox) has also integrated usage of the points library
#### Hirenodes
- Upped monitoring on node storage to prevet any future disk issues.
- Generally quiet few weeks!
#### Index Network
- Will be demonstrating a langchain library today. We have a Python SDK today!
- Olas will be using Index as an agent-first search engine.
The team showed a live demo of their LangChain SDK, which exposes Index as a source for RAG, and their JavaScript SDK, which exposes simple programmatic access to search via Index Network.
### Protocol Updates
#### 3Box Labs
Two major updates:
- rust-ceramic
- Been tracking down a handful of stability and performance issues.
- Performance issues now largely isolated to CAS, we are working on a handful of improvements on CAS.
- In-order delivery is the last remaining feature to be completed.
- rust-ceramic beta landing May 20.
- Data Feeds API
- Resumability lands tomorrow, May 3.
- rust-ceramic users will not be able to rely on it until we ship in-order delivery for rust-ceramic (May 20.)
- js-ceramic users can use it starting May 3.
#### CIPs
No updates for now!
### Upcoming Events
#### EthCC
3Box is currently working on finalizing its plans for EthCC, more on that to come soon! We'll be hosting a dinner at EthCC.
#### GenAI
Conference in SF, we'll be hosting a dinner there, inviting key stakeholders from AI companies. Details and invite list information to come later this month. This will be the first in a series of dinners we'll be hosting in the coming months!
#### 3Box Labs x Lit Hackathon
Currently coordinating a small hackathon with the Lit team end of May. More info to come.
# 2024-04-18
## Agenda
* Important announcements
* Ecosystem updates
* Protocol updates
* Upcoming events
## Notes
### Ecosystem Updates
* **Points SDK** (3Box Labs)
* Aggregator + Allocation primitives which use a shared hosted node (for test purposes)
* [Express server demo](https://github.com/ceramicstudio/solutions-sdk/tree/main/demo/server)
* Available as [npm package](https://www.npmjs.com/package/@ceramic-solutions/points?activeTab=versions); Link to [GitHub repo](https://github.com/ceramicstudio/solutions-sdk)
* Will be integrated into ComposeDB Sandbox on docs
* If time permits, additional fullstack example app will be included as an example as well that consumes the library
* Initial customers want to allocate points for research contributions
* If supports DataFeed API, OrbisDB can provide shared points instance (check with Golda)
* **Index** (Index Network)
* Released TS/JS SDK, to create index wrapping ComposeDB to expose nat lang interactions with CDB models
* Going to release Python SDK
* Langchain search coming soon
* Already 2 teams using SDK today
* Sometimes getting errors on Data Feed API, but resolve once CDB node restarts; maybe K8S doesn't like long lived connections networking between pods? (From Hirenodes)
* Working on more configurable embeddings and developer experience
* **Orbis**
* Working on shared instances, making it easy to onboard new devs; already 2 developers using them; BD for additional users
* Onboard DAOSpace to DB, want to create 300K streams from scraped data from governance platforms (Optimism 20K streams, ...)
* Might be using ODB CSV Uploader plugin
* Testing new plugin for ENS domain names - using ENS name plugin to reverse lookup address - making it easier to issue credentials to users
* Going forward: Autmoatically create preset views that everyone can query. Specific views for specific use cases -- e.g. get posts for social use cases
* Need to migrate from old tile documents by indexing that content into new shared
* **Hirenodes**
* Continuing to run nodes; everyone on v5.5.0
* Got serverside events running over nginx
### Protocol Updates
* Rust-Ceramic (3Box Labs)
* Almost production ready
* Shorter runs hit targets, longer runs hit some issues
* Looking to sustain much higher loads for much longer periods of time; 100tps for a week is the first benchmark
* If you have a use case or scenario that stresses Ceramic with performance or correctness - please get in touch with team
* Orbis was trying to dump 1-2M solana txs per day into ceramic; 3Box Labs to connect with team because this use case is within bounds
* DataFeed API (3Box labs)
* Looking to clean APIs and library abstractions by untangling js-ceramic + composedb into rust-ceramic + some library to create documents from events that DBs/indexers can built on top of
* First, aiming to make DF API stable and production ready, then will look to what comes next
* 3BL Team will connect with DF API users in order to design the next version
* CIPs
* Multi-Prev and After for stream ordering, nothing net new, but feeling more confident they can solve problems for users in the future
* Conflict Resolution Change: Changing the rules when updates happen so time event isn't interfering with stream updates. No changes for devs, just less likely to get a broken/corrupt stream.
### Upcoming Events
* ETH Boston | Boston, MA
* Next weekend 27th and 28th
* Spencer giving Ceramic talk and will have a Ceramic booth
* GenAI Summit | San Francisco, CA
* Index Network is going
* FarCon | Los Angeles, CA
* As of now, no one is going
# 2024-03-21
### Agenda
* Ecosystem updates
- Us3r network
- Orbis
- Hirenodes
- 3Box Labs
* Open floor
*
### Attendees
- Michael Sena (3BL)
- Justina Petraityte (3BL)
- Joel Thorstensson (3BL)
- Aaron Goldman (3BL)
- David Estes (3BL)
- Paul Le Cam (3BL)
- Radek Sienkiewicz (3BL)
- Golda Velez (3BL)
- Julissa Dantes (3BL)
- Spencer Brody (3BL)
- Rohan Kochhar (3BL)
- Sam Doyle (3BL)
- Mohsin Zaidi (3BL)
- Nick Kammerdiener (Hirenodes)
- Donat (Orbis)
- Charles Levecque (Orbis)
# 2024-04-04
### Agenda
* CIP discussions
-
* Ecosystem updates
- Us3r network
- Orbis
- Features
- Shared instances
- Developers can open up their node to other developers
- Orbis will also stand up a default Orbis node
- Depending on your context, you will create different data
- All connecting to same Orbis db default server, but data partitioned by context
- Plugins: CSV uploader to apply large amount of data to Ceramic node
- two types of orbisdb instance
- 1) you run your own orbisdb + ceramic node yourself (e.g., molecule wants their own ceramic node and orbis instance). you run a non shared version where you are the only user. only one db with a set of admins. issues = everything needs to be set up yourself
- 2) diff settings configuration. shared instance. still one ceramic node, one orbis instance, but multiple slots in the orbis db instance, each slot has its own database. could have multiple slots and credentials for each database.
- are there scalability concerns? can one project degrade performance of another?
- initially, orbis will probably be the only one opening up slots.
- could rate limit projects
- Hirenodes
- Running recon.
- More requests for testnet nodes.
- mostly small projects kicking the tiers.
- Proxy issue
- Problem connected to data feed api behind proxy
- TODO: hirenodes to follow up with data feed api lead on this
- 3Box Labs
- looking into longevity issue on rust-ceramic, issue with database access. needs to be resolved before vprime is released. should be resolved within 2 weeks
- building out event validation on rust ceramic side (currently happens on js-ceramic side)
- Request for comments on forum posts
- After: I am interested in events only after x date
- Discussion of ways to modify CAS so you dont have to send all your events to CAS
* Open floor
### Attendees
Aaron D Goldman(3box)
Joel Thorstensson(3box)
![image](https://hackmd.io/_uploads/ryaJhEn1A.png)
### Notes
# 2024-03-21
### Agenda
* Ecosystem updates
- Us3r network
- Orbis
- Hirenodes
- 3Box Labs
* Open floor
*
### Attendees
- Michael Sena (3BL)
- Justina Petraityte (3BL)
- Joel Thorstensson (3BL)
- Aaron Goldman (3BL)
- David Estes (3BL)
- Paul Le Cam (3BL)
- Radek Sienkiewicz (3BL)
- Golda Velez (3BL)
- Julissa Dantes (3BL)
- Spencer Brody (3BL)
- Rohan Kochhar (3BL)
- Sam Doyle (3BL)
- Mohsin Zaidi (3BL)
- Nick Kammerdiener (Hirenodes)
- Donat (Orbis)
- Charles Levecque (Orbis)
### Notes
No CIP updates to discuss
* Orbis updates from Charles
- want to start with recon
- avoiding migration is important
- moving orbis social to use MIDs from tile docs
- 1-1.5 million streams per day broke ceramic (another reason to use recon).
- using postgres for time being
- orca, drift, jupiter transactions (https://sol.party/)
- interpreted solana tx data offchain was idea
- sign in with solana => **required for WS1 event validation**
- questions about how to best migrate old tile doc streams to MIDs (new streams are using it)
- possibly plan to import them as read only
- aware that recon will not support tiledocs or caip10 streams
* Nick from Hirenodes
- gitcoin is running well
- some interest after ethDenver (indirect marketing?)
- spin up demo nodes => easy
* 3box updates (Joel)
- new season kicking off. primary focus on making rust-ceramic (ceramic-one) stable
- verifying signatures, time events
- self anchoring
- option to submit tx to blockchain without relying on 3BL anchoring service
- exploring submitting a merkle root hash rather than all events to reduce bandwidth
- getting rust-ceramic production ready
- alpha, beta releases.
- beta target to hit benchmarks of 300 transactions per second sync from node A to B (not network tps) for prolonged periods (i.e. 1 million writes/hour)
* Open floor: "interest in decentralized CAS"
- not a lot of clear interest by builders (unclear how well known it is)
- too much else to worry about
- node runners (nick) are more interested, particularly if CAS issues
- people might want it but "where's the wallet" might deter actually doing it
---
# 2024-02-29
### Agenda
* CIP discussions
- CIP-137 updates
* Ecosystem updates
- Us3r network
- Orbis
- Hirenodes
- 3Box Labs
* Open floor
### Attendees
- Cole Brown (3BL)
- Justina Petraityte (3BL)
- Joel Thorstensson (3BL)
- Aaron Goldman (3BL)
- Nathaniel Cook (3BL)
- Paul Le Cam (3BL)
- Mohsin Zaidi (3BL)
- Baptiste Greve (Orbis)
- Liang Qiao (US3R)
### Notes
🦘 🐇 happy leap day 🐸 🦌
#### rust-ceramic/recon API (CIP-137):
Had discussion yesterday to finalize http api
API currently exposes EventID, plan is to hide that as an internal detail
Need to update the CIP with the finalized API design
Right now Recon feed only returns CID of the root of the event to apply, the additional blocks need to be fetched one at a time with multiple round trips. In the future the feed will return a CAR file with all the blocks needed to apply an event.
Want to update CIP by next core-devs call so we can share it with OrbisDB and others. Will make a forum post.
#### OrbisDB Beta Launch
Officially announced!!!
Website updated, branding updated, public annoucement.
Repo is open source
6 or 7 alpha testers, running their own instances
Charles is in discussions with other teams about running in production and/or building plugins.
Two focus points in next month:
- Public shared OrbisDB gateway
- Migrating existing Orbis Social users onto OrbisDB
Old version of Orbis had a centralized API that everyone used. Bad for centralization, but good for developer quickstart. We want some kind of public gateway that devs can use to start playing with OrbisDB. Will be some sort of variation of OrbisDB b/c main OrbisDB assumes only one app is in control.
OrbisDB gateway as a service. In the future others can run shared OrbisDB nodes as well.
Working with Hirenodes to run OrbisDB as a service.
Next want to migrate Orbis Social users onto OrbisDB.
17k profiles
100k posts
on Orbis social today
Right now OrbisDB stack is kubo, postgres, js-ceramic, orbisdb
Want to simplify this stack
Data feed resumability:
easiest path uses the CDB database
We want to test if we can use the same Postgres instance for both ComposeDB and OrbisDB database.
Might be time to start thinking about a minimal aggregation library. Will require cleaning up the relationship between the state store and the index db.
#### US3R:
OrbisDB vs ComposeDB:
Shared modeling layer
Can they serve different use cases?
ComposeDB might offer finer controls
Thinking about S3 hosting OrbisDB as well as ComposeDB
Main focuses:
- driving adoption
- releasing the S3 development experience to end users
S3 more suitable to advanced users who are already familiar with Ceramic and ComposeDB
Talking with Michael about integrating docs
Building our own social protocols using ComposeDB
S3 is dev stack
U3 is social protocol
U3 already integrated with Farcaster and Lens
Can build whole social protocol (social graph, profiles, etc) on ComposeDB
- hoping to use interoperable data models with Orbis social
Thinking about Points use cases
Lens vs Farcaster vs Bluesky vs ActivityPub
- ActivityPub and bluesky have wider adoption
- but lens and farcaster are more crypto-native. Much more likely to have a wallet set up.
#### 3BoxLabs team update
ComposeDB v0.7 is released, with support for SET account relation, field locking, and unindex.
In final week of season 8, getting ready for season 9
Next season primary focus is:
- launching rust-ceramic and Recon in prod
- scaling and performance work
- CAS performance
- Breaking ground on designing what a future decentralized CAS looks like
#### Protocol minimalization
For now aggregation will stay out of the rust codebase, might change in the future.
rust-ceramic is just about event distribution, signature verification, and timestamping.
indexers have to do aggregation (possibly with the help of a common aggregation library they can use)
wen CRDT?
Once CIP-137 is finalized and signature verfication pushed into rust-ceramic, then new types of data structures in Ceramic become possible.
# 2024-02-15
### Agenda
* CIP discussions
-
* Ecosystem updates
- Us3r network
- Orbis
- Hirenodes
- 3Box Labs
* Open floor
* points
*
### Attendees
- Michael Sena
- Cole Brown
- Justina Petraityte
- Joel Thorstensson
- Baptiste Greve
- Seref Yarar
- Aaron Goldman
- Sam Doyle
- Donat
- Rohhan Kochhar
- Serhat
- Karahan Sahin
- Adam Corrado
- Radek Sienkiewicz
- Danny Zuckerman
- Mohsin Zaidi
### Notes
- 3Box Labs
- Two major projects concluding with our current season
- rust-ceramic (without anchoring functionality) reaching production readiness
- Data Feed (fka Firehose) API reaching production readiness (with resumability support)
- Orbis
- OrbisDB plugin ecosystem is growing
- Official launch of OrbisDB coming Feb 26, accompanied by a new website to make sure developers have a clear on ramp and don't get conflicting information
- New branding!
- OrbisDB SDK will be open source, separate package from the "instance" which is the indexing service itself
- Aiming to make the SDK as simple to use as their current tooling
- Index
- Vector embeddings will be stored in a queryable index
- Looking at ComposeDB / Ceramic as an agent-orchestration mechanism
- Agents will be using NLP to process and trigger off of events
- Aiming to deliver composability of indexes by ETHDenver
- Community discussion points
- Michael: Points Working Group
- Not trying to pick a winner with regards to schema or other implementation details. We believe these are emergent properties. Winners will be determined by *distribution*, not by technical decisions like this.
- If you're interested in taking part, comment on the [thread in the forum](https://forum.ceramic.network/t/storing-points-on-ceramic/1470) or in the Points WG Telegram group.
- Joel: Protocol minimization
- Conversation happening in the [Protocol Minimization](https://forum.ceramic.network/t/protocol-minimization/1483) forum post.
-
- Joel: CAS Scaling
- Not as controversial, but more a dump of our insights.
- Find it [here](https://forum.ceramic.network/t/proposal-for-cas-scaling/1507).
- Joel: Proposal for flat Recon
- Conversation happening [here](https://forum.ceramic.network/t/proposal-for-flat-recon/1503).
- Current approach relies on a sortable key, but the problem is it's difficult to choose hierarchy when structuring these keys. There are tradeoffs to every configuration. Do we care about grouping by models? Controllers? etc. It's always wrong from one approach.
- Flat Recon rejects this entirely—let's stop thinking of the space as a ring, but rather as a potentially many-dimensional landscape.
- Recon / Ceramic itself would flatten this hierarchy and remove any intrinsic information from the keyspace
- Nodes are then given option to either linearly scan and filter this keyspace when reconciling with other nodes
- OR, pre-index the keyspace based on their preferences
- One could imagine certain indexers (e.g. OrbisDB, ComposeDB) preferring separate indexes over the event keyspace to optimize their specific dataflows.
- We see logical use cases for these indexes, but don't have any community members really asking for them right now. So it is hard to determine whether now is the time to implement.
- Don't want to add too much compute overhead on Ceramic nodes by requiring them to compute arbitrary indexes.
- When do we feel it is important to prioritize this?
- Answer is not immediately obvious
- If anyone feels strongly about sharding servers across fields beyond the model, share with us and that can help us prioritize.
---
# 2024-02-01
### Agenda
* CIP discussions
-
* Ecosystem updates
- Us3r network
- Orbis
- Hirenodes
- 3Box Labs
* Open floor
- Protocol minimization
### Attendees
![Screenshot 2024-02-01 at 16.14.20](https://hackmd.io/_uploads/BJw_7SYca.png)
- Michael S (3boxlabs)
- Joel T (3boxlabs)
- Liang Q (U3er Network)
- Yunhe Z (Us3r network)
- Baptiste G (Orbis)
- Donat (Orbis)
- Charles L (Orbis)
- Paul C (3box labs)
- Sam D (3box labs)
- Cole B (3box labs)
- James P (3box labs)
- Mark K (3box labs)
- Rohhan (3box labs)
- Nathaniel C (3box labs)
- Spencer (3box labs)
- Mohsin Z (3box labs)
### Notes
**CIP updates**
Joel
* Followup ticket for Recon to update the CIP
Nathaniel
* Discrepancy btwn current status and difference between now:
* Will likely tweak event id
* Spec is also lacking definition of interest id
* Should be updated in 2 weeks
**Ecosystem updates**
**Orbis**
Baptiste
* Opening OrbisDB alpha version to customers next week
* New SDk interacting with OrbisDB is working
* Will come with indexing + querying features
* will be open source
* Public announcement in late March / early April with updated branding
* Waiting on event type
Spencer
* Release with event type should be going out within 1-2 days
**OrbisDb Plugins**
Sena
* Potential Dune plugin
Baptiste
* Could work with OrbisDB
Sena
* Open questions about indexing and exposing the entire network
* What is the level of compute
Baptiste
* How does it work?
Joel
* Everything is indexed with custom views on top
Baptiste
* Hard to anticipate which fields specifically would be interesting for developers
**OrbisDB <> ComposeDB Feature Parity**
Spencer
* Data modeling feature parity with ComposeDb and OrbisDb
* e.g., interfaces, SET account relation, field locking (immutable fields)
* OrbisDb support expectations?
Baptiste
* If the info is all available in the schema of the model, we can eventually get it
* OrbisDB is in early stages right now
* Can eventually sync with the features in the future
Spencer
* Interfaces: Most consequential one will be interfaces. Can impact model design and architecture. Powerful primitive
* Field locking: should be more trivial, especially since it's enforced at the Ceramic level. Just need to expose the feature to OrbisDB users
* Set account relation: Orbis would need some feature development to ensure that users could create new models using this feature.
* Ideally full production launch of OrbisDb would have feature parity, though not needed for Alpha
**User network**
Liang
* Working on workflow for composites
* Reorganized UX to make more sense to new developers
* Adding user guide: Explore, Compose, and Build
* Main product focus is S3
* Also dogfooding own tooling by building a social network complement using Ceramic
* Currently supporting farcaster and lens but both those protocols need new users to pay to use service, so trouble onboarding
* Working on an Guest mode for ceramic. Profile + social graph without paying. Able to bookmark content from farcaster
* Excited about points, working on design spec. SDK to quickly roll out points. Standard use case across products - leaderboards, google analytics like tracking system, then define and map events collected from users to actual points
* All points can/should be built on Ceramic
Michael
* Need a record + some algorithm over the record
* System of recording actions could be streams on Ceramic
* Useful not just for points, but also for audit logs e.g., Discord moderation panel
* Verifiable actions that are recorded on Ceramic and then computed on top of
* Points primitives could be as important as an ERC-20 type token.
**3box labs**
Cole
* Busy week
* Working on getting the Firehose API out, resumability feature is next
* And other features in ComposeDB: Locked fields, Set account relation
* Moving forward on Recon, marching towards goal of js-ceramic being sunsetted. When ready, partners can help test it
* Getting ready to start focusing on CAS scalability
* Potentially a path to unifying js-ceramic/composedb code bases
Sena
* Announced Proof of Data Summit, co-hosting with Tableland
* Should coordinate some EthDenver and growth plans with Orbis
* Ramping up social account activities
**Ongoing forum discussions**
(missed the beginning of this)
Spencer
* Should aggregation be pushed down into rust ceramic?
* Where should state store live?
* Standardization around aggregation is important
Sena
* Data modeling is a lower level primitive than composedb. so composedb and orbis need to implement standards at same time. otherwise ecosystem can become fractured.
Joel
* Standards bodies agree on standards but different implementations
* Different teams should come together and agree on standard rather than being pushed by one team
Spencer
* Someone who wants to build an indexer shouldn't need to have to build their own aggregator
* e.g., OrbisDb shouldnt need to re-implement model instance documents
* Ceramic protocol vs data product could be thought about separately
* Can node js deliver production level quality at scale?
Nathaniel
* Not convinced that indexers can build on top of an opaque aggregator vs instead indexer being some type of aggregator
* Not sure if there are 2 or 3 layers, prefers 2 layers
Sena
* e.g., Orbis building on a document change API is easier than building on an event change api
Nathaniel
* but is that easier than building an aggregation layer
* can you build a stateless aggregation layer? if yes, then build it
Joel
* aggregation layer is like a library
Nathaniel
* Binary or library? Binary can't be a product necessarily
---
# 2024-01-18
### Agenda
* Ecosystem updates
- Us3r network
- Orbis
- Hirenodes
- 3Box Labs
* Open floor
### Attendees
- Aaron D Goldman (3boxlabs)
- Michael Sena (3boxlabs)
- Liang Qiao (U3er Network)
- Yunhe Zhang (Us3r network)
- Baptiste Greve (Orbis)
- Paul le Cam (3box labs)
- Donat (Orbis)
- Sam D (3box labs)
- Spencer (3box labs)
- Cole Brown (3box labs)
### Notes
No CIP updates.
#### Ecosystem updates
**US3R**
Working on node as a service. New developers work better on the composit level then the model level. In order to make it easyer to share a composite we would need a way to share them as an object in the network. There is not curently a gobal registry for composits.
We releast a proof of consept to store a composit as a data model. Requesting feedback in the forum [post](https://forum.ceramic.network/t/rfc-enhanced-composite-management-and-sharing-in-s3-console/1461).
Feedback for this shema to make sure it solves the general case but a minimal viabal it the ferst step.
To test the node as a servise you need to request with an email workflow.
**Orbis**
Working on the SDK portal. This is the way to write and read data. The goal is to be developer frendly. Sharing a demo of the SDK next week. In feb we are leting the first devs on the platform. Probaly five teams at first. We are makeing the UI so that dev don't need to use the CLI to do the inital setup. Hirenodes will work with this SDK to run the DB instance on the same server as the Ceramic node. Distributed as a docker img that can be co-located on the server.
Waiting on 3box for the firehose API. We will be testing the new firehose shortly. Are there any blocking work prerelease.
Context header will be a StreamID byte string. Joel will look at updating the PR for context.
**3box**
Firehose is out v1 with no resume yet.
The recon speed and metric are improving.
It is not clear what the numbers to use are for advertising the limits of ceramic.
How to limit spam in the events?
# 2024-01-04
### Agenda
* CIP discussions
-
* Ecosystem updates
- US3R network
- 3Box Labs
* Open floor
### Attendees
- Aaron D Goldman (3boxlabs)
- Michael Sena (3boxlabs)
- Spencer (3boxlabs)
- Cole Brown (3boxlabs)
- Sam D (3boxlabs)
- Mohsin Zaidi (3boxlabs)
- Joel Thorstensson (3boxlabs)
- mcclurejt (?)
- Charles Levecque (Orbis)
- Baptiste (Orbis)
- Liang Qiao (US3R Network)
- Yunhe Zhang (US3R Network)
### Notes
#### Context
The context is a header that used to seperate streams that are relivent in a spesific context.
For now this might be lmmited to a StreamID so that a context could be fetch to understand it better.
#### Server Sent Events (firehose)
The ability to send events to an listener when they are discovered.
Basic server-side functionality is built. Http-client support needs to be built, and it needs end-to-end testing.
Orbis would like to have this released in 2 weeks.
#### Costs for node as a service (US3R)
What is the cost to run a node?
What is the willingness to pay for customers?
# 2023-12-21
### Agenda
* CIP discussions
- None to discuss today
* Ecosystem updates
- Us3r netowkr
- Orbis
- Hirenodes
- 3Box Labs
* Open floor
- [RFC: Native support for single-write documents with field locking](https://forum.ceramic.network/t/rfc-native-support-for-single-write-documents-with-field-locking/1398)
### Attendees
- Joel Thorstensson
- Aaron Goldman
- Cole Brown (scribe)
- Baptiste Greve
- Charles Levecque
- Michael Sena
- Spencer Brody
- Nathaniel Cook
### Notes
- OrbisDB
- Key feature is the user-friendly interface. Don't have to be a developer to meaningfully interact with orbisDB
- Much more decentralized than their current indexer—every user can run their own Ceramic node and get their own orbisDB dashboard operating over it.
- "Contexts" concept allows users to organize models into various... contexts
- Sub-contexts represent models w/in a context
- Sub-contexts can leverage plugins, e.g. Gitcoin plugin to verify that controller of a stream has a Gitcoin passport
- Other plugins include ability to query an API and add fields from the result
- e.g. Hitting CoinGecko for asset prices and storing them every 10m
- UI has a built in query / view editor that allows you to query orbisDB streams live
- Plugins truly extend the functionality of orbisDB—they can even add their own API routes
- e.g. CSV uploader plugin based on "generate" primitive
- Upload a CSV, parse by row, write streams
- These streams are owned by a did controlled by the plugin itself in the CSV uploader, but the plugins can implement many other behaviors re: stream ownership
- ChatGPT plugin allows users to transform records based on ChatGPT queries, e.g. classifying a book by interest
- Can update a record or create a new one
- Team currently exploring future options for multiwriter streams
- Contexts support participants by mulitple controllers
- They have themselves a controller who updates their metadata
- Their state are stream based
- Discoverability—Compose is discovery by model, Ceramic by stream. OrbisDB provides context as a new content discovery mechanism.
- In discovery w/ customers, migration from standard SQL DBMS to Ceramic is key
- Need to be able to handle the load and have a smooth onramp to support this
- Some users interested in maintaining their centralized db, esp for private data, but willing to export subsets of their data. Good onboarding opportunity.
- All models are now using ModelInstanceDocuments
- MIDs force programmer to choose a model
- Dynamic model creation / evolution
- Could do a search of existing models that match req's
- See this as a separate product almost
- Ideally we help people converge at the same models
- Ceramic
- Ceramic v4 will be breaking compatibility with v3 and earlier
- In part doing this because we are setting extremely high performance standards for rust-ceramic and Recon that we want to hit from day one
- Targeting subset of functionality then adding it all back in as we become more performant.
# 2023-12-07
### Agenda
* CIP discussions
- CIP-124 Recon
- EventId and sorting
* Ecosystem updates
- Us3r netowkr
- Orbis
- Hirenodes
- Intuition
- 3Box Labs
* Open floor
### Attendees
* Aaron D Goldman
* Cole Brown
* Yunhe Zhang
* Nathaniel Cook
* Charles Levecque
* Cuyler
* Baptiste Greve
* Liang Qiao
* Spencer
* Donat
* Oak
* William Luedke
### Notes
#### CIP-124
sorting by (network_id, hash(separator_value), **context_value**, **time**, controller, stream, event)
- aaron: have a PR ron: have a PR ron: have a PR
aaron: explaining the current sort order
"context" value might be relevant
"time" use ethereum block
Joel: position of time? What is time based on?
Aaron: A CIP with a new field for "after" a particular blockhash
Nathaniel: If both times exist how do we pick?
Aaron: the after might be sooner
Nathaniel: if after is defined do we need to follow the chain back?
Aaron: that is more correct, but might make more sense to just use "after"
Joel: UNIX Timestamp vs blockhash?
Aaron: the thing in the after header, if blockhash stronger proof can be generate
Spencer: clarify sep | ctx | time vs sep | time | ctx
Aaron: Context is for further filter beyond just the schema of the data
Instingt is that bc context is developer construted, people are more likely to divide by context
Spencer: I'm still worried about that it's useful for sync purposes, dev wants to know how close they are to catching up
Nathaniel: you can still answer spencers question if time is after context
I expect number of contexts N to be small
Even if N is big you can still do it
Spencer: you still can't do the time thing if you care about all context in the model
How does it work for zero context?
Next step: Aaron to set a PR to the CIP with the updated range
#### Ecosystem updates
**Orbis**:
Making good progress on OrbisDB
Will share a demo with Michael next week
making progress on server and client
Donat is working on clientside iterface
Using serversent events
ETA is Jan
context as part of the metadata
thinking of ways of batching updates / inserts
Aaron: what exactly do you mean by batch update?
Donat: Not exactly sure yet, taking inspiration from traiditional dbs
For now just creating a model for batches
Spencer: an api that acccepts multiple signed events would be easy
an api that takes multiple events with one signature would be more difficult
Donat: single signature is not a requirement here.
Spencer: Have wanted to separate document Id and StreamId fro a long time
Donat: Context and metadata fields?
Spencer: Discussion from CIP-124, support higher leve would come later
Baptise: We actually can't add the field in the Client library
Is rejection on purpose?
Spencer: need to think carefully about how to use it
If you want to set it we can have a discussion
Baptise: adding the field would be the only thing we need now
Nathaniel: Ceramic doesn't need to understand, or don't want it to?
Baptiste: We just need to be able to have data there.
Baptiste: In orbisdb the context is always a streamid
use this as an application field.
**US3R Network**:
Liang: Making good progress on the node as a service initiative
Bunch of PRs from Keramik project
Looking at a demo in the end of the month
One click to provision resources and get a Ceramic url
some of the issues we had we already posted in the forum
visual model editor we already built
Indexing info is specified in the composite and it has to be shared using github or something else
don't want to save indexing infomration in the model
the low level api is there, some issues when trying to invoke those APIs
spencer is probably already aware of this.
Forum post: https://forum.ceramic.network/t/devtools-composite-frommodels-does-not-support-filter/1304
Spencer: Danny has been thinking about this, don't fully understand why it doesn't work
Danny seem to think that the problem is that node is aready indexing the model and you try to re-add it.
Where in the process does it break down?
Liang: we will follow up in the PR discussion.
**3Box Labs**:
Nathaniel: We got rust-ceramic running on Clay testnet, found issues with Kubo interactions it pruned connections
The way we happen to host nodes with AWS was challenging with perf of rust-ceramic
Now, instead of running rust-ceramic as a replacement for kubo we are going to host net new nodes using kubernetes / Keramik
What this means is that there should be new nodes as rust-ceramic comming online
Once this is there means that you could also start using rust-ceramic
biggest advantage will come when recon drops.
# 2023-11-09
### Agenda
* CIP discussions
- CIP-142 Linkfeed - a content feed abstraction standard
- CIP-145 Multi-Prev
* Ecosystem updates
- Us3r netowkr
- Orbis
- Hirenodes
- 3Box Labs
* Open floor
### Attendees
- Aaron Goldman
- Sam Doyle
- Cole Brown
- Michael Sena
- Baptiste Greve
- Liang Qiao
- Julissa Dantes
- Mohsin Zaidi
- Donat
- Paul Le Cam
- Yunhe Zhang
- Nicolas Kammerdiener
### Notes
#### CIP-142 Social Models
- Liang: How to get social model reuse over re-invention.
- Baptiste: Orbis is currently migrating everything to MIDs for OrbisDB, but the migration is not yet complete. Regarding the mobile story, we have made attempts to achieve react-native compatibility, but haven't had any success yet.
- Liang: Team is interested in aligning on a shared model.
#### CIP-145 Multi-Prev
- Aaron: Not much change in the last weeks. One CIP we've been working on is MultiPrev—have folks in the community been running into diverged state? If so, how do they resolve it?
- Liang: My intuition is that conflict resolution should be the responsibility of the application layer. e.g. Farcaster uses a CRDT to sidestep the issue.
- At the moment, the streaming layer resolves conflicts automatically. Data however isn't technically lost, but we don't do anything with that. Do we want to expose this dead data to developers for their own resolution?
- Baptiste: We don't have much motivation for this, since users will more or less self correct data that doesn't update as they'd expect.
#### CIP-124 Recon
- Mohsin: Mostly ready, pipelined behind rust-ceramic work. Should be seeing its first release next season assuming things go well.
- Aaron: From performance benchmarking, a lot of time is spent publishing to and reading from the DHT. This indicates Recon stands to make a significant improvement here.
#### Ecosystem Updates
##### Orbis
- Thinking about UX/DX for OrbisDB, looking at a graphical configuration application to make life easier.
- Been clarifying the mechanism for listening to Ceramic topics.
- Ideally, only subscribe to updates rather than raw events.
- Been in collaboration with 3BL team to discover an ideal API.
- Tentatively calling this API Firehose, it should be agnostic to the backend (Recon or no)
- Developing a plugin architecture to give developers hooks for certain common tasks and make OrbisDB an extensible platform. First four plugins:
- generate: Hook to execute side effects such as creating a Ceramic stream
- validate: Pre-write data validation.
- add_metadata: Attach application or context specific metadata to records.
- post_process: Automatically transform streams on indexing.
- Example they've prototyped: gate collection writes based on the writer's Gitcoin passport.
- [The OrbisDB specification](https://orbisclub.notion.site/Specifications-158aef7c63064b2ca730d6990d390ce4)
##### US3R
- Have started work on keramik, which is a big step towards their work on Nodes as a Service. Hoping for a demo next month.
- Have hit some issues regarding ComposeDB. Recent changes (post 0.5.0) we are no longer able to get index metadata along with model metadata. It seems as though these are now two separate pieces of metadata.
- Seems like the workflow now requires grabbing the indexing and model information separately before indexing a model locally.
- Currently in discussion with Paul on the form [in this post](https://forum.ceramic.network/t/devtools-composite-frommodels-does-not-support-filter/1304).
- Paul: The way indexing was designed, it was meant to be the applications concern, which is why it is separated from model information. Applications can decide on their own indexes for a model. So as a result, they don't exist on the models themselves, only on the model composites.
# 2023-10-26
### Agenda
* CIP discussions
- [Recon (CIP-124)](https://cips.ceramic.network/CIPs/cip-124)
* Ecosystem updates
- Us3r network
- Orbis
- Intuition
- Hirenodes
- 3Box Labs
* ComposeDB node separation
* Open floor
### Attendees
Aaron Goldman
Joel Thorstensson
Michael Sena
Nathaniel Cook
Spencer Brody
Liang Qiao
Zhang Yunhe
James Pham
Donat
Mohsin Zaidi
Danny Browning
Paul Le Cam
John McClure
Sergey Ukstv
### Auto notes
#### Database Updates and Field Discussions
Joel and Michael organized a meeting with discussions on a variety of topics including updates on their shared databases and the possibility of using an application-specific field in the event ID. The decision was made to trial having notes from the Zoom summary function to make note-taking easier. Aaron discussed the tension between ordering events for efficiency and maintaining time-based order for cost-effectiveness. Spencer and Baptiste introduced the idea of adding an application-specific field to the event ID, which was initially referred to as a 'namespace' or 'context'. Nathaniel pointed out the implications of using a fixed-length byte field for this data, suggesting it might limit the scope of their applications. The team agreed that the recon needs to be fixed with the context and might need to do the hashing themselves to maintain any kind of ordering.
#### Model Complexities and Data Handling Discussed
Liang discussed the complexities of using the same models in different applications and suggested that either a field in a model or an index level namespace could be used to solve this issue. He also mentioned an infrastructure upgrade that improved their data handling. Baptiste and Joel discussed the possibility of comparing data streams and Joel shared about an internal dashboard that shows individual rights of data production. Aaron and Spencer raised questions about the efficiency of data loading.
#### Cows, Shipping, and New Project Discussion
Spencer, Aaron, Baptiste, Liang, Joel, and Michael discussed their work, specifically about the pace of their cows, with Spencer expressing surprise. The team then shifted their focus to a new project that Liang proposed, which involves shipping a ceramic note as a service. This idea aims to improve developer experience by enabling users to spin up their own node for testing or development. Baptiste suggested integrating this service into their UI, providing a new note for the user. The team agreed to finalize the specifications for this new project in early November.
#### Modeling Framework for Applications and Databases
Michael and Baptiste discussed the modeling framework for applications, protocols, and composable data across databases. They emphasized the need for backwards compatibility and the use of instance documents. The team, consisting of Baptiste, Spencer, Aaron, Liang, Michael, Nathaniel, Joel, and John, deliberated on the handling of arrays in documents, developer experience, and the challenges of building databases on top of indexing and query systems. The conversation also touched on composite models and the concept of individual models. Towards the end, John from Intuition introduced himself and highlighted his work on leveraging attestations for subjective data. He identified some issues related to aggregate models and the need for a standardized procedure for ingesting data uniformly.
#### Progress, Deployment, Migration, Newsletter, and Feedback
John discussed the progress of their project, Nathaniel provided an update on the deployment of Restoramic in their QA environment, and outlined their plans to push it to Clay and then to main net. Nathaniel also mentioned that the migration process is working smoothly and can work with anyone who wants to migrate a node. Joel's microphone was not working well. Michael discussed his idea of creating an ecosystem newsletter to highlight the work of all teams, and asked for input on how to gather and share information. James asked for feedback on a presentation about the composed of B mode separation.
#### Developing Custom Indexing Solution for Ceramic
The meeting discussed the development of a custom indexing solution for Ceramic. Sergi explained that the aim is to provide a simplified method for users to build their own solutions on top of Ceramic, without needing access to its internals. He also outlined different layers involved in this process, including a streaming layer, an aggregation layer, and an index solution. Sergi highlighted the importance of providing some level of introspection into what's happening internally, such as exposing a feed of stream states and Ceramic events. Spencer suggested that focusing on the feed of changes to documents might be sufficient for most users for now, and deferring the lower level of actual events to a later stage. Joel asked how this relates to the Ceramic API, and Sergi responded that it is directly related but they are approaching it from a different angle.
#### Document vs Event Usage in System: Debate Continues
Aaron, Spencer, Liang, SU, and Nathaniel discussed the use of events and documents in their system. Liang highlighted that the users primarily care about the documents and may not be aware of events or commits. However, for building a new database, events could be more useful. The group also debated whether to push everything needed for processing from the firehose or just the stream ID or C id. SU explained that the idea is to push the stream state, which includes processed events, to eliminate unnecessary requests. Nathaniel pointed out a trust relationship that changes when events are not sent but documents are, emphasizing the need to verify the document's construction from the events. Spencer countered that this doesn't alter the current trust relationship with Ceramic.
#### Explanation of Ceramic Node System for Document State and Event Logging
Spencer explained how the current system works, where the ceramic node provides the current state of a document when asked. Nathaniel agreed with Spencer's explanation. Spencer emphasized that the system also includes metadata about the log of events that went into computing the state. John found all proposed solutions interesting and expressed familiarity with constructing a database from a change data capture feed. However, he indicated that they would not dive down to the event level to verify the integrity of the document. It was determined that stream state changes would be sufficient for their needs. Spencer also mentioned the need to determine the right API for connecting to the node and accessing the feed, but time ran out for further discussion. John clarified that they only operate in a private networking context and do not run much off-browser.
---
# 2023-10-12
### Agenda & Notes
* Implementation updates
- Rust
- Keramik
* CIP discussions
- [Recon (CIP-124)](https://cips.ceramic.network/CIPs/cip-124)
* Open floor
### Attendees
Liang Qiao
Nathaniel Cook
David Galindo
Spencer Broody
Joel Thorstensson
Zhang Yunhe
### Notes
#### Impl upates
NC: still finding bugs in rust ceramic
few have been fixed, few more lingering
have upgraded parts of our own infra to use rust-ceramic
Keramik:
LQ: Nicolas from hirenodes have worked on a majority of the issues
Will ping him for an update. Haven't had time work on Keramik yet. Currently focused on the indexer. Need better recovery when the node goes down
AG: indexer using to pubsub in your own code?
LQ: we are subscribing to libp2p events, but rely on js-ceramic to get the data
there is some perf bottleneck when adding more indexing to js-ceramic. Hit bound of queue
SB: Can't index all of ceramic on one node, curious how cerscan does it
LQ: they don't index everything
we are trying to index all ComposeDB models
JT: they maybe only index pubsub?
Invite Orbis team and Nicolas to these calls
NC: small new features being added from 3BL side as we use it
e.g. wait for k8 status
#### CIP updates
##### Recon
NC: Discussing updating the EventID format. The motivation is to make it easier to observe and debug. e.g. "how synchronized am I?" by putting previous event blockheight into the eventId. Also enables syncing without ggoing back to beginning of time. Disadvantage is that you must know the blockheight to syncronize the event, e.g. you need to know the previous event height to calculate the eventId.
AG: There's a CIP for addign an `after` field to headers. Could point to an ethereum block hash.
NC: high level goal is the ability to observe sync progress
Other proposed change is to add a namespace section to the EventID. Allows apps to put arbitrary data into the eventId. Helpful when there is a many to many reationship between models x users.
Both of these suggestions are pending writing things down.
To update recon CIP
AG: right now we don't know when events happens after some time. I think it makes sense to add an `after`field.
SB: Only downside I can think of is that we need to validate it
JT: Very similar to validate time events
AG: one of the debates I was having with mohsin: Should I use latest eth block or latest block that has an anchor.
JT: store all eth block hashes?
Aaron to follow up on CIPs for after and multi-prev CIPs
#### Open floor
##### associate dapps to models
LQ: exploring ways to get dapps x model mapping. CACAOs was suggested. Not really reliable.
Being able to assiclate models to Dapps enables devs to more easily reuse model.
JT: CACAO approach unreliable why?
LQ: data we get from cacao not really clean,
JT: CACAOs are the only programmatic way, otherwise would need to do it manually
LQ: keep using CACAOs, with some manual mappsings
##### Regarding Recon
LQ: we have a clear use case planned, hypothetically what are the use cases?
AG: should work a lot better if you have a big dump of data, bc of the nature of synchronizing. What are all the evens taht one server has and the other doesn't have. E.g. create a brand new model, as an import of other events, should do decently well. Would to a bit less well if there's a mill events and now a new mill events are added.
LQ: how much of data in a mill events?
NC: eventIDs are 72 bytes now
LQ: when it comes to event data ipfs is still bottleneck?
---
# 2023-09-14
### Agenda & Notes
* Implementation updates
- Rust
- Keramik
* CIP Discussions
- [Multiple previous (CIP-145)](https://cips.ceramic.network/CIPs/cip-145)
- [Recon (CIP-124)](https://cips.ceramic.network/CIPs/cip-124)
- [Ceramic API (CIP-137)](https://cips.ceramic.network/CIPs/cip-137)
* Open floor
### Attendees
Aaron D Goldman
Liang Qiao
Joel Thorstensson
Mohsin Zaidi
Yunhe Zhang
Spencer T Brody
### Notes
#### Implementation updates
Aaron:
rust-ceramic is making progress. js-ceramic runs successfully with rust-ceramic instead of kubo. Seems to give us a small performance improvement as a bonus. Still doing more testing, but getting close to making rust-ceramic the default supported ipfs implementation for ceramic.
Joel:
Uncovered a bug due to the intersection of bitswap and ReCon. Nathaniel put in a fix.
Mohsin:
Built out a test framework for testing with mix of kubo and rust-ceramic backed Ceramic nodes (based on Keramik), to ensure interoperability.
Aaron:
issues to getting Keramic production-ready?
JT: there are issues open and assigned to Nick from Hirenodes to fix.
#### CIP updates
Aaron:
Multi-prev CIP coming along, handles forks in stream history. Includes changes to conflict resolution and branching rules.
**Need community to review and give feedback** (in forum post)
useful for:
- anchors not conflicting with updates
- letting devs/users explicitly resolve write conflicts
- letting users fork away from writes created by a malicious app.
JT: Updates on ReCon or Ceramic API?
no updates
#### open floor
Liang:
Track usage of a Model by apps?
Currently doing this extracting app info from CACAOs.
Is this the best way?
JT: yes - user attests to app in signature
Liang: Some streams are missing this info. Apps that use did:key instead of did:pkh.
JT: how common is this? I thought most data is written with did:pkh
Liang: a significant portion
JT: this is surprising. Would be good to look into which data models tend to have data written with did:key.
Liang:
Reliability of our indexer.
We listen to p2p pubsub, but if indexer is down data is lost.
Aaron: do you use HDS?
Liang: we tried but had issue
JT: worth trying again, HDS has had a lot of fixes lately
Recon will change this a lot and make this a lot easier.
ReCon won't discover data that isn't still hosted by at least one Ceramic node
HDS can find data even if no Ceramic node is hosting it, so long as the underlying IPFS data is still available. HDS can also theoretically detect when data used to exist but is now missing from the network.
---
# 2023-08-31
### Agenda & Notes
- Implementation updates
- Rust
- CIP Discussions
- [Multiple previous (CIP-145)](https://github.com/ceramicnetwork/CIPs/pull/145)
- [Recon (CIP-124)](https://cips.ceramic.network/CIPs/cip-124)
- [Ceramic API (CIP-137)](https://cips.ceramic.network/CIPs/cip-137)
- Keramik questions
- Open floor
### Attendees
Aaron D Goldman
Nathaniel Cook
Liang Qiao
Joel Thorstensson
Mohsin Zaidi
Yunhe Zhang
### Notes
#### Implementation updates
Nathaniel: Not much has changed in the implementation but we added a lot to the tests
moved the ceramic and Kubo api to a single port.
This lets us have openAPI yaml for api description.
Liang: what is the testing strategy.
Nathanel: use keramic to run tests that look for properties of the network.
Then run the property test agents many network shape.
Joel: Paused the push to recon to make sure rust-ceramic is a solid drop in replacement.
Nathaniel: recon will then be able to ship behind a feature flag.
Liang: Composition is about the nodes in the network not the topology.
Nathaniel: We are starting with the fully connected network, testing other topologies will come later.
This is a side effect of how we connect the network not a explicit limitation.
We use the DHT to find other ceramic nodes.
Liang: so even without recon you only connect to ceramic nodes.
Nathaniel: We connect to other ceramic nodes first.
Joel: IPNI we may want to use this instead of the DHT.
#### Go Ceramic
Liang: We are testing the use of keramic.
#### Ceramic API
Aaron: CIP-145 multiple prev to merge streams.
Joel: we will need a API for getting the many tips
Nathaniel: we may have core/convenience API e.g. get all events
#### Keramic
Liang: the spin up of a network should be independent of the test runner.
We need to be able to spin up and connect to PostgresDB.
The metrics are tied to the test but we need then from monitoring a network.
Nathaniel: We also need to have the metrics outside of the simulation for the network.
Joel: feel free to make the change and we will re-view and merge it in.
Liang: need to be able to connect to outside resources. Ethereum, CAS, ...
---
# 2023-08-17
### Agenda & Notes
- CIP Discussion
- [Recon (CIP-124)](https://cips.ceramic.network/CIPs/cip-124)
- [Ceramic API (CIP-137)](https://cips.ceramic.network/CIPs/cip-137)
### Attendees
- Cole Brown
- Aaron D Goldman
- Liang Qiao
- Yunhe Zhang
### Notes
- Recent discussion has been focused around APIs for Recon / rust-ceramic
- Initial effort has been roughly cloning kubo's API for maximum compatibility with minimal effort
- Ultimately we'd like to create our own APIs that more directly reflect what Ceramic is doing
- Aaron & Nathaniel are of the opinion that versioned APIs aren't all that helpful
- Preferred alternative is to simply add routes and avoid conflicts while keeping things easier to understand/recall
- Integral version numbers don't hold much semantic weight
- Primary use cases for Ceramic API
- When users don't require aggregation or wish to aggregate their data in different/novel ways, the Ceramic API gives them primitives for dealing exclusively with events in their raw format.
- Right now we don't have good tools for managing divergent streams
- But we have the capabilities to expose this to a stream controller (e.g. a prompting that stream has diverged, present the tips, prompt for resolution)
- It would be nice to add an API to get all known heads of a given stream
- We'd like to see TileDocuments deprecated over time, in particular in favor of the ModelInstance document.
- US3R: We'd like to learn more about keramik
- keramik is suitable for running nodes for testing as is now
- keramik intentionally spawns an isolated network within Kubernetes
- It's trivial to connect keramik nodes to other nodes, so it's quite possible to use it for prod deployments
- We'll need to collaborate to help facilitate production use-cases, as there will be many warts deriving from it being built for testing
- US3R is very interested in using keramik to automate their nodeops, particularly deploying new nodes
- Recon update
- Mostly MVP-ready, looking good in local tests
- Next step is productionize it and get it deployed via rust-ceramic
- rust-ceramic rollout is incremental
- First we will use rust-ceramic to replace everything we're using kubo for (essentially IPFS content distribution sans DHT)
- rust-ceramic will plug in to js-ceramic, which will use it for content persistance and retrieval
- Second step, we will actually enable recon for content discovery
# 2023-08-03
### Agenda & Notes
* Implementation updates
* Rust
* Go
* CIP Discussion
* [Recon (CIP-124)](https://cips.ceramic.network/CIPs/cip-124)
* Update on Peer discovery (Nathaniel)
* [Ceramic API (CIP-137)](https://cips.ceramic.network/CIPs/cip-137)
* gRPC vs OpenAPI
* A discussion on finalization ([forum post](https://forum.ceramic.network/t/cip-137-ceramic-api/1153/23))
* Open floor
### Attendees
* Aaron Goldman
* Joal Thorstensson
* Oak
* Cole Brown
* David Galindo
* Nathaniel Cook
### Notes
Start: 8:04 AM PT
rust impl:
We are API compatible for the API serfice we call.
We can drop in rust-ceramic in place of kubo.
More testing is still needed.
We have prof of concept for recon syncing thousands of events.
More performance debugging needed.
#### CIP Discussion Ceramic API (CIP-137)
Sencor:
If a project wished to build on top of Ceramic and wants to know when a write is final.
After this moment the event is unlikely to be rolled back.
If anchoring takes to long then the Cacao could expire and invalidate the event.
If a stream forks both forks can live for some time and a late publishing attack
could revert a number of event in stream after the fork.
Do we want to make consumer deal with the divergence of the covered(committed) and uncovered(uncommitted) events.
Joel: How will this efect Recon/API
Aaron: We need to express the time information in the API returned data.
Spencer: We should enrich the API with timestamps.
But this will not solve the data witholding attack.
These writes are anchored but not final.
When you are maintaining a store you would need to have state for tip and anchored tip.
Joel: It is a problem that a new time event can invalidate a outstanding chain.
Spencer: The API need to be able to respond with invalidation events for when a event moved from valid to invalid.
The consumer needs to be able to replay the stream.
Nathaniel: The conflict resolution should not be the ceramic API. Event retrieval and state aggregation are
distinct tasks. Lets not combine them.
Spencer: What dose the API need to expose to be able to aggregation layer.
Nathaniel: By returning events in the order the node learned them rather then the anchor time.
This may be simplified.
Joel: Example. There is a Data event (exp. T1) gets anchored at (T2) the event is expired not valid.
Nathaniel: Can the ceramic API abstract away the need for clients to validate events? probably no, this changes the trust relationship.
Applications need to make their own tradeoffs for eager evaluation.
We can have a logical replay from beginning and then use checkpoint to improve performance.
Spencer: We can keep the anchored state to replay from.
Remember that in the future there will be stream tipes that use curent state not a delta stream.
Nathaniel: two questions
Can I restore state? Can I detect that I am in invalid state? Answer to both is yes.
Spencer: We should be able to operate in ether a trusting or auditing mode.
Joel: It is like the ETH virtual machine. You could pull the blocks and validate yourself but most won't do that.
Nathaniel: This choice dose not force us to take this choice from the client.
Spencer: late publishing means that we can't get away from needing a roleback.
#### GRPC/Open API
Nathaniel:
Both solve a simalar problem.
Both define the API by writing a machine readable spec.
Open API uses YAML
GRPC uses ProtoBuff
you can generate clients for either.
Open API to methods.
GRPC methods to API.
GRPC is built on. http2 and may give useful performance with long lived channels.
GRPC has multiplexing baked in to its world view.
David: Having to build ProtoBuff is not a real problem.
Nathaniel: It is posible to have both if there are use case.
If the first version is easy to adopt later vertions can give a more performant option.
Joel: Should we merge the PR for using Open API?
Nathaniel: yes, but it is not the final form.
# 2023-07-20
### Agenda & Notes
* Implementation updates
* Rust
* Go
* CIP Discussion
* [Recon (CIP-124)](https://cips.ceramic.network/CIPs/cip-124)
* Open floor
### Attendees
Lian Qiao
Yunhe Zhang
Mohsin Zaidi
Aaron Goldman
Joel Thorstensson
### Notes
#### Rust impl
- Sqlite to persist events for ceramic
- Nathaniel working on first iteration of Ceramic API
#### Go impl
- Made some progress finally, have to implement a new version of kubo ipfs
- Decided to go back to kubo, "kubo++"
- Discussion on go architecture
- Discussion of Recon, eventIds, and ranges
- Is there a protocol interface for Recon?
- More discussion about internals of recon
#### CIP: Recon
##### fencepost
Need to pin down how fencepost are pinned down and also discovery
##### Both sort-key/value in eventId
Aaron is going to update spec and implementations
---
# 2023-07-06
### Agenda & Notes
* Implementation updates
* Rust
* Go
* CIP Discussion
* [Recon (CIP-124)](https://cips.ceramic.network/CIPs/cip-124)
* ~~Update on Peer discovery (Nathaniel)~~
* An approach to js-ceramic compatibility PoC (Zach)
* [Ceramic API (CIP-137)](https://cips.ceramic.network/CIPs/cip-137)
* gRPC vs OpenAPI
* ~~A discussion on finalization ([forum post](https://forum.ceramic.network/t/cip-137-ceramic-api/1153/23))~~
### Attendees
- Joel Thorstensson
- Aaron Goldman
- Zhang Yun He
- Mohsin Zaidi
### Notes
#### Go implementation
- Team is working on the roadmap.
- Will wait for Liang to provide more detailed updates.
#### Rust implementation
- Recon can work on arbitrary byte arrays instead of just strings.
- Specifically event IDs.
- Nathaniel has added peer discovery so it can use the IPFS DHT to find other nodes on Recon.
- When you first start Ceramic it will look up the same key as it uses for the pubsub topic and find other Cerqmic nodes.
- Will retrieve advertisements for "I am a Ceramic node" and see if it supports Recon and which version.
Aaron:
- If we get to a very large number of Ceramic nodes, we can use the DHT to separate nodes with similar interests.
- For now, since the number of nodes is small, we can just find all Ceramic nodes instead of having to make individual requests for interests.
Joel:
- Would lean towards the event ID approach because it gives us the future path.
- Let's follow up with a detailed forum post.
#### CIP Discussion
##### js-ceramic PoC
- Have an initial implementation working with Recon.
- In a position to start benchmarking it.
Joel:
- Is there a way to handle Tile documents?
- Maybe telling the Ceramic node to handle certain ranges.
Zach:
- Makes sense, can use family as separator to do that.
Joel:
- Should work with IDX.
- There are some use cases for subscribing to individual streams.
- Should be able to call Recon API and subscribe to a stream.
Aaron:
- Using JSON?
Zach:
- Trying to keep it similar to the current Ceramic API at least for testing.
##### Ceramic API
Joel:
- Have been having an interesting discussion about this.
- Once Nathaniel has documented his thoughts on this we'll continue the discussion on the forum.
- Spencer had some thoughts about finalization that we'll continue discussing next week.
---
# 2023-06-22
### Agenda
* Go implementation
* CIP discussion
* [Recon (CIP-124)](https://cips.ceramic.network/CIPs/cip-124)
* [Ceramic API (CIP-137)](https://cips.ceramic.network/CIPs/cip-137)
### Attendees
- Joel Thorstensson
- Cole Brown
- Nathaniel Cook
- Aaron Goldman
- Liang Qiao
- Cherry Yang
- Zhang Yun He
- Spencer Brody
- jack
- bufan
- Michael Sena
### Notes
#### Go Implementation
Impression / goal at the first stage is to replace the IPFS dependency with a libp2p dependency. In particular, looking at replacing BitSwap protocol. Goal is to diverge from the Kubo (golang) implementation and move towards the approach offered by the Rust implementation.
Main question to be answered right now is regarding roll out—is this a change to be phased out in testnet or implemented in one shot and observed in a testnet environment for a while.
Joel on 3BL's current thoughts regarding moving away from IPFS and towards Recon:
- *describe phased plan to move to rust-ceramic*
We are eager to collect community input throughout this process to help inform the design of the upcoming Ceramic API.
Liang: Should we prioritize CIP-124 (Recon) for early compatibility, or should we build directly towards CIP-137 (Ceramic API)?
Nathaniel: From ease of development perspective, there are two two potential technical approaches:
1. Fork kubo and modify its behavior, maintaining the fork.
2. Build on top of [Boxo](https://github.com/ipfs/boxo), the newer modular approach to building on top of IPFS.
Nathaniel: "I don't see \[3BL\] moving away from IPFS completely in the short term or, potentially, at all."
There is consensus around libp2p being the appropriate layer in the stack to operate within, but boxo and similar efforts at PL do make it quite a bit easier leverage protocols and concepts (e.g. CIDs and the DHT implementation) present in IPFS.
Liang: Core question is will we be relying on CIDs and content addressing in the long term?
Joel: Currently running into a bottleneck regarding the number of DHT queries and inserts required to add content to IPFS. Under heavy write loads, IPFS nodes are chewing up their CPU capacity. One of the goals of Recon is to allow us to enable efficient content discovery for Ceramic events by their Ceramic IDs, not just by Content IDs.
Cole & Spencer: We likely won't be inserting CIDs into the DHT, meaning Ceramic content would not, by default, be accessible/discoverable from within the IPFS network. BitSwap would mean IPFS peers could *request* data from us if they (1) knew its CID and (2) knew to ask our node for it.
Joel: Circling back to CIP-124 vs immediate implementation of CIP-137, a nice place to start development could be with event validation logic. As a group, we could develop test vectors for event validation to arrive at a standard.
Aaron: What is the goal of creating a go-ceramic implementation?
Liang: To create a Ceramic building block we can create products on top of. One such product we've been entertaining is stream processing over Ceramic streams.
Joel: With that in mind, focusing on CIP-124 and event stream validation are probably the two most valuable things to work on.
Nathaniel: We have testing infrastructure we've been developing, so please reach out when you're ready to start testing your implementation. We can help.
#### Recon
*Discussion regarding details and semantics of Recon and its notion of stream sets / stream IDs ensues. More information surrounding Recon can be found in [CIP-124](https://cips.ceramic.network/CIPs/cip-124) and in the prototype implementation in [rust-ceramic](https://github.com/3box/rust-ceramic/tree/main/recon).*
To confirm, in the near-term building off of something like Boxo will be an accelerant, as we likely won't be moving beyond BitSwap in the next 12 months.
Joel: Prototype for Recon and rust-ceramic is rough. We currently have an open PR to codify the data structures surrounding [event IDs](https://github.com/3box/rust-ceramic/pull/53). Near term question is how do we get js-ceramic speaking with rust-ceramic so we can leverage Recon without fully rearchitecting js-ceramic. In the mid-term, the js-ceramic repository will likely *become* ComposeDB, as more Ceramic functionality is pushed down to rust-ceramic.
Joel: One thing not particularly clear in the CIP-124 spec is the binary protocol spoken between nodes. Right now, messages are CBOR-encoded lists of keys.
Nathaniel: We're using a framing library to frame messages sent over the wire. From the protocol perspective, it's all message passing.
Cole & Aaron discuss backwards compatibility. Self describing data makes backwards compatibility quite easy in some regards, whereas there is a simplicity to using multistream's stream versioning to dictate compatibility. An interesting problem space to explore is running multiple versions of the protocol at the same time, backed by the same on-disk store.
#### Ceramic API
Aaron: We would love feedback on [CIP-137](https://cips.ceramic.network/CIPs/cip-137) from our community to make sure it enables other use-cases.
Joel: To emphasize, the API offered in CIP-137 is very rough. For example, we've pointed to JSON-RPC as an RPC standard, but we aren't set on that nor are we convinced it's the correct solution. We will likely introduce an experimental implementation of a Ceramic API as a means of getting Recon out in the wild, though the standard itself will stil be malleable at that time.
---