# SSZ-QL
> [!Note]
> Implementing this endpoint is a part of [EPF6](https://github.com/eth-protocol-fellows/cohort-six) project by [Jun](https://github.com/syjn99) and [Nando](https://github.com/fernantho). You could check our project [proposal](https://hackmd.io/@junsong/HkQ9XBEHel) to get more context ([motivation](https://hackmd.io/@junsong/HkQ9XBEHel#Motivation), progress, etc.).
## SSZ-QL
> [!Note]
> Cases we have to consider from [Light Client Roadmap for Electra](https://hackmd.io/@etan-status/electra-lc#SSZ-query-language):
> - Requesting any sub-tree of an SSZ object (notably, only fields that the client knows about, to stay forward compatible) ✅
> - Allowing to request summaries (hash_tree_root) instead of fully expanded subtrees, on a per-field basis. ✅ 🤔 summary property in the json schema. I think it is not totally correct.
> I guess an example of this would be to request the whole beacon state without the full validators and balances subtree. Only their summaries.
> - Filtering, e.g., a transaction with certain root ✅
> - Simple back-references, e.g., a receipt at the same index in its tree as the found transaction in the other tree ✅ alias property
> - Specification relative to where the proof should be anchored ✅
> [!CAUTION]
> This document is a draft currently under construction.
## 1. Introduction
This specification introduces the Simple Serialize Query Language (SSZ-QL) and its Generic Merkle Proofs (GMP) component, establishing a unified framework for extracting data from any Consensus-Layer object while providing cryptographic proofs that attest to the integrity and validity of the retrieved information.
This specification heavily relies on [SSZ](https://github.com/ethereum/consensus-specs/blob/dev/ssz/simple-serialize.md) and [Merkle Proof](https://github.com/ethereum/consensus-specs/blob/dev/ssz/merkle-proofs.md).
## 2. Terminology & Notation
- **Anchor/root**: a `Bytes32` Merkle root hash thatcommits to of all leaf nodes in the Merkle Tree. It is computed as `hash_tree_root(merkle_tree)` e.g. `beacon_block_root = hash_tree_root(beacon_block)` and `beacon_state_root = hash_tree_root(beacon_state)`. SSZ-QL also allows anchors that are roots of Merkle subtrees e.g. `validators_root = hash_tree_root(beacon_state.validators)`.
- **Leaf node**: a node in a Merkle tree that has no child nodes. **In SSZ Merkleization, a node that corresponds directly to a serialized SSZ chunk (a 32-byte value).** Leaf nodes are the fundamental units from which higher-level branch nodes are derived through successive pairwise hashing, ultimately producing the aforementioned `hash_tree_root` of the SSZ object.
- [**Generalized index**](https://github.com/ethereum/consensus-specs/blob/dev/ssz/merkle-proofs.md#generalized-merkle-tree-index): a generalized index is an integer that represents a node in a binary Merkle tree where each node has a generalized index `2 ** depth + index in row`.
```
1 --depth = 0 2**0 + 0 = 1
2 3 --depth = 1 2**1 + 0 = 2, 2**1+1 = 3
4 5 6 7 --depth = 2 2**2 + 0 = 4, 2**2 + 1 = 5...
```
This representation yields a node index for each piece of data in the Merkle tree.
- [**Path**](https://github.com/ethereum/consensus-specs/blob/dev/ssz/merkle-proofs.md#ssz-object-to-index): a list of nodes that represents the path from the root to the specified node.
- [**Summaries and expansions**](https://github.com/ethereum/consensus-specs/blob/dev/ssz/simple-serialize.md#summaries-and-expansions):
Let `A` be an object derived from another object `B` by replacing some of the
(possibly nested) values of `B` by their `hash_tree_root`. We say `A` is a
"summary" of `B`, and that `B` is an "expansion" of `A`. Notice
`hash_tree_root(A) == hash_tree_root(B)`.
We similarly define "summary types" and "expansion types". For example,
[`BeaconBlock`](../specs/phase0/beacon-chain.md#beaconblock) is an expansion
type of
[`BeaconBlockHeader`](../specs/phase0/beacon-chain.md#beaconblockheader). Notice
that objects expand to at most one object of a given expansion type. For
example, `BeaconBlockHeader` objects uniquely expand to `BeaconBlock` objects.
## 3. Data & Proof Model (transport-agnostic core)
### [3.1 Recap SSZ object model](https://github.com/ethereum/consensus-specs/tree/dev/ssz)
#### TODO in we consider necessary
### 3.2 Query primitives
SSZ-QL exposes five primitives: anchor, path, filter, summary and alias:
#### 3.2.1 Anchoring model
An anchor is the node (root or subtree) the client already trusts and against which proofs are verified. SSZ-QL allows the usage of any anchor/root either it is a `hash_tree_root(CL object/merkle tree)` or `hash_tree_root(merkle subtree)`.
- A request may provide multiple anchors.
- Every query leaf must be a descendant of the anchor.
##### Examples
```json
"anchor": "beacon_block",
"anchor": "beacon_state.validators"
```
#### 3.2.2 Paths
- A path is a dot-notation plus optional [index]/[start:end[:step]] slices.
- Paths must match existing field names or indices in the SSZ type.
##### Examples
```json
.validators[100].withdrawal_credentials
.fork.current_version
.validators[100:200:5]
```
#### 3.2.3 Filters
- A filter is a boolean expression evaluated per element when a path yields a list/vector.
- Elements that evaluate to false are dropped out before proceeding to the proof construction.
- Typical operators are: `==, !=, >, <, in, &&, ||`
##### Examples
```json
"filter": "effective_balance >= 32e9 && slashed == false"
"filter": "withdrawals_credentials == 0x0123456789012345678901234567890123456789"
```
#### 3.2.4 Summaries
A summary replaces a subtree's expanded value with its `hash_tree_root()`.
Summaries are selected on a per-item list as shown in the examples below.
##### Examples
```json
{
"anchor": "beacon_state",
"path": ".",
"summaries": [".balances"]
}
```
```json
{
"anchor": "beacon_state",
"path": ".validators[100]",
"summaries": [".withdrawal_credentials"]
}
```
#### 3.2.5 Aliases
Alias names the current selection's index (or indices) so later can reference them.
Back references use $<alias>.index inside a path.
##### Examples
```json
{
"path": ".transactions[*]",
"filter": ".hash == '0xabc…'",
"alias": "tx"
},
{
"path": ".receipts[$tx.index]"
}
```
### 3.4 Proof model
As per [Merkle Proofs](https://github.com/ethereum/consensus-specs/blob/dev/ssz/merkle-proofs.md) specification in Consensus Layer, SSZ-QL also uses the proving methodology, having single Merkle Proofs as a special case of Merkle Multiproofs where the array of leaves to prove contains only one item.
This "unified" way of providing Merkle proofs provides users with a strong user experience at building system that consumes this proofs.
### 3.5 Input validation
TBD
We need to assess the validity of the queries.
ANALYZE the cascade:
- check if valid anchor
- check if path is valid considering the anchor
- check if filter is valid considering the path
The provided data input is validated by checking if the path is a descendant of the anchor:
```python
def is_descendant(child, ancestor):
while child > 1:
if child == ancestor:
return True
child //= 2
return False
assert is_descendant(leaf_gi, anchor_gi)
```
*Code 1: check if leaf is descendant of the anchor*
```python
def is_filtering_available(path, filter):
assert is_filtering_available(".validators", "*")
```
*Code 2: check if filters is valid*
## 4. Bindings
### API Rest (Beacon API)
A new `SSZQL` namespace is suggested. This namespace is set to hold at least the following endpoint:
- Endpoint: `/eth/v1/beacon/query`
- Method: `POST`
The query object contains the anchors/roots, paths, filters, summaries and aliases.
#### Parameters
This query contains no parameters, all the properties are specified in the request body.
#### Request Body
| Name | Type | Description |
| --- | --- | --- |
| `query` | `object` | A detailed query object. |
| `include_proof` | `boolean` | Whether including a merkle proof or not. Default to false. |
| `multiproof` | `boolean` | Whether using multiproof method for efficiency. Default to false. |
Every request MUST satisfy the JSON schema below.
```json
{
"$schema":"https://json-schema.org/draft-07/schema",
"title":"Query Object for SSZ QL",
"description":"Schema for SSZ query object",
"type":"object",
"properties":{
"query":{
"type":"array",
"items":{
"type":"object",
"properties":{
"obj_type":{
"type":"string",
"description":"The type of object being queried.",
"enum":[
"state",
"block"
]
},
"id":{
"type":"string",
"description":"Identifier of the object being queried (state ID or block ID, depending on obj_type)."
},
"anchor":{
"type":"string",
"description":"The anchor is the root against which the proofs are verified."
},
"path":{
"type":"string",
"description":"A merkle path to the item"
},
"filter":{
"type":"string",
"description":"Optional predicate applied when the path contains a list/vector"
},
"summaries":{
"type":"array",
"description":"Optional list of sub-paths under this item to return as hash_tree_root instead of expanded values.",
"items":{
"type":"string",
"description":"Path (relative to 'path') whose subtree should be summarized (hash_tree_root)."
},
},
"alias":{
"type":"string",
"description":"Optional variable name that stores the index (or full path) of every element matched by this query item, so later items can reference it with the placeholder `$<alias>.index` (e.g. `.receipts[$tx.index]`)."
}
},
"required":[
"obj_type",
"id",
"anchor",
"path"
]
},
"min_items":1
},
"include_proof":{
"type":"boolean",
"description":"Whether including a merkle proof or not. Default to false."
},
"multiproof": {
"type": "boolean",
"description": "Whether using multiproof method for efficiency. Default to false."
}
},
"required":[
"query"
]
}
```
This JSON schema contains an array of:
- *anchor*: A dot notation that resolves to one root or node in the tree.
```json
"anchor": ".beacon_state"
"anchor": ".beacon_state.validators"
"anchor": ".beacon_block"
```
- *path*: A dot notation that resolves to one or more SSZ leaves.
```json
"path": ".validators[100].withdrawal_credentials"
"path": ".fork.current_version"
"path": ".validators[*]"
```
- *filter*: A boolean expression evaluated per element when path yields a list/vector. Any elements that fail the predicate are dropped before proof
generation. Typical operators are: `==, !=, >, <, in, &&, ||`
```json
"filter": "effective_balance >= 32e9 && slashed == false"
"filter": "withdrawals_credentials == 0x0123456789012345678901234567890123456789"
```
- *summaries*: an array of, the selected subtrees are uniquely identified by its Bytes32 `hash_tree_root()`. When a user requests a summary, SSZ-QL must reply with the whole object with the requested subtree summarized. Useful when the dApp consumer wants a commitment but not the full data:
```json
{
"anchor": "beacon_state",
"path": ".",
"summary": [".balances"]
},
{
"anchor": "beacon_state",
"path": ".",
"summary": [".balances", ".validators"]
}
```
- *alias*: saves the index (or full path) of every element matched by this query item, so later items can reference it with `$<alias>.index`. Enables simple back-references like "receipt at the same position as this transaction".
```json
{
"anchor": "beacon_state",
"path": ".validators[*]",
"filter": ".slashed==true",
"alias": "slashed_validators"
},
{
"path": ".balances[$slashed_validators.index]"
}
```
### Response
| Code | Description | Content |
| --- | --- | --- |
| `200` | Success | Response with the [schema](#Response-Schema) |
| `400` | Invalid ID or malformed request | N/A |
| `404` | Not found | N/A |
| `500` | Internal server error | N/A |
| `503` | Beacon node is currently syncing and not serving request on that endpoint | N/A |
See [Example sections](#Example) for example responses.
#### Query Result
The content for success query follows typical Beacon API format: it includes `version`, `execution_optimistic`, `finalized`, and `data`.
For non-multiproof mode and multiproof mode, `data` contains an array with [`QueryResult`](#QueryResult) and the state root (`root`). In case multiproof is disabled, this `data` array one element per query elemente. In case multiproof is enabled, the `data` array contains only one element.
##### `QueryValueResult`
| Field | Type | Description |
| --- | --- | --- |
| `paths` | Array\<String\> | An array of Merkle paths for the queried items. Each path corresponds to the value at the respective index in the `results` array. |
| `results` | Array of any | An array of the actual values in JSON format. |
##### `QueryProofResult`
| Field | Type | Description |
| --- | --- | --- |
| `leaves` | Array\<String\> | An array of the actual values (leaves) corresponding to the queried paths. Each value is typically represented as a **32-byte hash**. |
| `gindices` | Array\<Number\> | An array of generalized indices for each value in the `leaves` array. |
| `proofs` | Array\<String\> | An array of proof that are sorted **descending order** by the generalized index. |
---
### API Binding - examples
For these examples, all the requests target the same endpoint:
POST `/eth/v1/beacon/query`
### 1. SSZ-QL without proof (`include_proof` is false)
```json
{
"query": [
{
"obj_type": "state",
"id": "finalized",
"path": ".genesis_validators_root"
},
{
"obj_type": "state",
"id": "finalized",
"path": ".validators[100]"
}
],
"include_proof": false,
"multiproof": false
}
```
Response:
```json
{
"version": "electra",
"execution_optimistic": true,
"finalized": true,
"data": {
"root": "0xf1f2f3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1",
"values": {
"paths": [
".genesis_validators_root",
".validators[100]"
],
"results": [
"0x4b363db94e286120d76eb905340fdd4e54bfe9f06bf33ff6cf5ad27f511bfe95",
{
"pubkey": "0x8f2a41e5b6c71234abcd5678ef90ff11223344556677889900aabbccddeeff0",
"withdrawal_credentials": "0x00aa2753bbcc...99",
"effective_balance": "32000000000",
"slashed": false,
"activation_eligibility_epoch": "64",
"activation_epoch": "64",
"exit_epoch": "18446744073709551615",
"withdrawable_epoch": "18446744073709551615"
}
]
}
}
}
```
### 2. With proof (`non-multiproof` mode)
Request body:
```json
{
"query": [
{
"obj_type": "state",
"id": "finalized",
"path": ".genesis_validators_root"
},
{
"obj_type": "state",
"id": "finalized",
"path": ".validators[100]"
}
],
"include_proof": true,
"multiproof": false
}
```
Response:
```json
{
"version": "electra",
"execution_optimistic": true,
"finalized": true,
"data": {
"root": "0xf1f2f3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1",
"values": {
"paths": [
".genesis_validators_root",
".validators[100]"
],
"results": [
"0x4b363db94e286120d76eb905340fdd4e54bfe9f06bf33ff6cf5ad27f511bfe95",
{
"pubkey": "0x8f2a41e5b6c71234abcd5678ef90ff11223344556677889900aabbccddeeff0",
"withdrawal_credentials": "0x00aa2753bbcc...99",
"effective_balance": "32000000000",
"slashed": false,
"activation_eligibility_epoch": "64",
"activation_epoch": "64",
"exit_epoch": "18446744073709551615",
"withdrawable_epoch": "18446744073709551615"
}
]
},
"proofs": [
{
"leaves": [
"0x4b363db94e286120d76eb905340fdd4e54bfe9f06bf33ff6cf5ad27f511bfe95"
],
"gindices": [
"65"
],
"proofs": [
"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"0xcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
"0xdddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
"0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee"
]
},
{
"leaves": [
"0x12345678deadbeefcafebabef00dabad12345678deadbeefcafebabef00dabad"
],
"gindices": [
"8796093023233"
],
"proofs": [
"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"0xcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
"0xdddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
"0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee"
]
}
]
}
}
```
### 3. SSZ-QL with multiproofs
Request body:
```json
{
"query": [
{
"obj_type": "state",
"id": "finalized",
"path": ".genesis_validators_root"
},
{
"obj_type": "state",
"id": "finalized",
"path": ".validators[100]"
}
],
"include_proof": true,
"multiproof": true
}
```
Response:
```json
{
"version": "electra",
"execution_optimistic": true,
"finalized": true,
"data": {
"root": "0xf1f2f3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1",
"values": {
"paths": [
".genesis_validators_root",
".validators[100]"
],
"results": [
"0x4b363db94e286120d76eb905340fdd4e54bfe9f06bf33ff6cf5ad27f511bfe95",
{
"pubkey": "0x8f2a41e5b6c71234abcd5678ef90ff11223344556677889900aabbccddeeff0",
"withdrawal_credentials": "0x00aa2753bbcc...99",
"effective_balance": "32000000000",
"slashed": false,
"activation_eligibility_epoch": "64",
"activation_epoch": "64",
"exit_epoch": "18446744073709551615",
"withdrawable_epoch": "18446744073709551615"
}
]
},
"proofs": [
{
"leaves": [
"0x4b363db94e286120d76eb905340fdd4e54bfe9f06bf33ff6cf5ad27f511bfe95",
"0x12345678deadbeefcafebabef00dabad12345678deadbeefcafebabef00dabad"
],
"gindices": [
"65",
"8796093023233"
],
"proofs": [
"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"0xcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
"0xdddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
"0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee"
]
}
]
}
}
```
### 4. SSZ-QL Summary request
Request body:
```json
{
"query": [
{
"obj_type": "state",
"id": "finalized",
"anchor": "beacon_state",
"path": ".validators",
"summary": true
},
{
"obj_type": "state",
"id": "finalized",
"anchor": "beacon_state",
"path": ".balances",
"summary": true
},
],
"include_proof": true
}
```
Response:
```json
{
"version": "electra",
"execution_optimistic": true,
"finalized": true,
"data": {
"result": [
{
"paths": [".validators", ".balances"],
"leaves": [
"0x4b363db94e286120d76eb905340fdd4e54bfe9f06bf33ff6cf5ad27f511bfe95",
"0x0500000000000000000000000000000000000000000000000000000000000000"
],
"gindices": [65, 269],
"proofs": [
"0x5730c65f00000000000000000000000000000000000000000000000000000000",
"0xbe3dc5b7843f6b253970803030a18501814c97ac893ec03560ce4962688f857c",
"0x8d637afd2d258e4d079ca7f00dd4c857a61431af7262ede447b712a25d71e4bb",
"0x09ef197f8757969dce6a9379281f5c5b1ab7aeba924631d6ac5e560e817733c0",
"0x323688e7370b5f72bdadc1cdd2f2f4d9cd7065647a6f54a961def1222684e6d6",
"0xb7edabb0a1e1cb42fe1138558ca73510c890ba5dfc013aca0e2bf9a7c26af0a7",
"0x0400000000000000000000000000000000000000000000000000000000000000",
"0x8e38229b2010e3cde27597c6e4852c4e6cdca82e03574383993cd60250e9ed3a",
"0xe05db90000000000000000000000000000000000000000000000000000000000",
"0x96a9cb37455ee3201aed37c6bd0598f07984571e5f0593c99941cb50af942cb1"
]
}
],
"root": "0x2d178ffec45f6576ab4b61446f206c206c837fa3f324ac4d93a3eece8aad6d66"
}
}
```
### Library
SSZ-QL can also be invoked directly without the need of the API.
```python
class QueryRequest:
root_object: object
items: List[QueryItems]
include_proof: bool = False
```
## Resources
- [Simple Serialize](https://github.com/ethereum/consensus-specs/blob/dev/ssz/simple-serialize.md)
- [Merkle Proof](https://github.com/ethereum/consensus-specs/blob/dev/ssz/merkle-proofs.md)
Sign in with Wallet
Connect another wallet