HackMDCAPER — Content-Addressed Protocol for Efficient Retrieval
TODO
- re publishing on the top-level specs site: could the site be repurposed so as to only contain the minimal viable specifications set? does that mean that it would import that multi* specs so as to be essentially complete?
CAPER is a minimalist iteration over IPFS, focused on providing a bedrock interoperability layer for typed content-addressed data in a way that is efficient, well-defined, and reasonably easy to implement.
Note: The name is intended solely to be used as a convenient handle as this discussion progresses.
Motivation
IPFS is a wonderful family of technologies that compose to expose a set of operations on content-addressed data. That what IPFS provides is desirable is undeniable: the ecosytem directory speaks for itself.
Over the years, the set of protocols that make up IPFS have undergone many iterations and supported much experimentation. Existing implementations have developed features organically and progressively solidified around some backwards compatibility requirements.
This diversity and variability has taught us a lot and has produced beautiful results. However, it has also turned IPFS into a sprawling and complex stack. Sitting under IPFS, libp2p alone is already an ecosystem unto itself, with a high multiplicity of approaches to peer-to-peer communication.
As a result, it has become hard to tell what IPFS "is" or whether a given piece of software is an IPFS implementation. We lack a shared specification of what a minimal implementation of IPFS is that is guaranteed to interoperate with the rest of the IPFS Cinematic Universe. Composability is wonderful, optionality isn't.
This document therefore lays out requirements for an IPFS bedrock ruthlessly aimed at interoperability and implementability. The underlying thesis is that a robust and well-defined foundation will be better able to support a diverse and composable ecosystem at much higher scales than today.
Semantics
This section details the semantics that IPFS CAPER supports. A protocol concretizing these semantics can then be developed, and other functionality layered on top of it. This approach is (loosely) modelled on RFC 9110: HTTP Semantics.
CAPER provides a uniform interface for interacting with content-addressed data — regardless of its type, nature, or implementation. This interface includes identifying, retrieving, and providing data.
Identify
The heart of content-addressing is content addresses. The fundamental substrate of IPFS is content-addressed blocks identified by CIDs.
CAPER imposes stricter restrictions compared to the broader CID world. Only CIDv1 is accepted. CIDv0 is fully deprecated. Systems that have CIDv0s need to convert them to CIDv1 before using them. (CIDv2 and CIDv3 are mentioned in various places, but they are currently reserved as imaginary creatures.)
issue-cid-content-type
The CID spec is vague about which multicodec values are valid in a CID:
What formats (multicodec codes) does CID support?
We are figuring this out at this time. It will likely be a subset of multicodecs for secure distributed systems.
Based on the decoding algorithm, the content type could be literally anything in the multicodecs table, including another CID or the multicodec for a hash.
One way to look at the content-type part of the CID is that it is similar to the media type used in MIME and HTTP. That makes it essentially an open-ended set of values, with no guarantees that any given client will understand the resource, only that it will be able to access the bytes. However this approach may run up against what might reasonably (?) be expected from an IPFS implementation, notably in terms of graph traversal. It also assumes that CID, as an open-world typed content address format, may feature an arbitrary number of different content types and that those are only meaningful if you can resolve them against the multicodecs table. Is this desirable?
Could this mean to divergent implementations on aspects that we care about (eg. CID → CAR of the DAG)? Basically, the question here is: what do we expect that an IPFS implementation must be able to do, and what can be delegated to another layer?
Alternatively, we could consider that only a small, fixed set of CID content types are relevant and restrict CAPER CIDs to those. All other CIDs would be considered unresolvable over IPFS CAPER. Such a list might look like:
- raw
- dag-cbor
- dag-pb
- car
- dag-json (or just CBOR?)
For everything else we could consider standardizing an IPLD structure with a media type field + bytes. No traversal would be expected at this layer.
Note that if the list includes CAR, then we would have to also address unresolved issues with CAR if we want to provide interoperability guarantees.
Verify
Trustless, but verify.
Every component that receives bytes matching a CID, be it because it retrieved the data or because it is a writable recipient for it, must validate the bytes against the CID hash component.
Without verification we lose the decentralized, trustless nature of IPFS. We could consider surfacing verification failure as a standard message similar to 404.
Retrieve
An IPFS CAPER process can interact with another system (eg. a local store, another process) to verifiably retrieve a CID.
issue-traversal
- Do we want to expect of CAPER implementations that they support DAG traversal, arguably limited to a strict subset of traversable formats?
- If so, do we leave bounding graph traversal up as an implementation detail?
issue-pathing
- Do we want to expect of CAPER implementations that they support path traversal (which requires the ability to traverse graphs) on top of CIDs?
- If so, can we align on a very strictly limited set of pathing methods?
- We could have layers, with path retrieval sitting atop block retrieval, but for that to be practical we would probably still want to limit pathing methods (ie. we shouldn’t end up with twenty *FS variants).
issue-private-retrieval
- How private can we make retrieval in the bedrock?
Provide
An IPFS CAPER implementation must be able to indicate to other implementations what CIDs it is able to provide.
A CAPER implementation may not be able to serve as a provider or receiver of content, it might be restricted to connecting to IPFS processes for verified retrieval and writing. Such implementations are "clients" in the sense that they are peers that cannot be read from or written to.
Receive
An IPFS CAPER implementation must be able to be written to by other processes. Requirements for writability are common across the stack.
Being writable likely means that the system should also support deletion (which it may or may not abide by, and without any guarantees that the CID resource will actually become unavailable if it does).
Advertize
An IPFS CAPER implementation should be able to tell other processes that interact with it which
features on top of the required minimum it supports. (Think OPTIONS.)
A clear way of advertizing additional functionality makes it possible to select specific peers based on what they support. It can also serve as a form of handshake mechanism when optimizations can be applied.
