Interval Merkle Tree: A Circuit-friendly Universal .
This documents describes the APIs, inner working of Interval Merkle Tree (IMT), a cryptographic accumulator that we use to instantiate Nullifier Set which require efficient non-membership proof and member insertion in circuit.
Complexity
For an IMT of height of size , batch insertion size :
Both Add
and BatchAdd
circuit checked non-membership before inserting.
|
Add |
BatchAdd |
Native Execution |
|
|
Constraint Cost |
hash + range_check |
hash + range_check |
E.g. for a Zcash sized Nullifier set and a ~1024 rollup capacity, BatchAdd
requires 66,603
* Hash whereas naively Add
one-by-one requires 131,072
Hash, and Sparse Merkle Tree would require 524,288
Hash! An almost 8x improvement in circuit size compared to SMT (which is what Circom and Aztec Connect use)
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
Challenges
Compared to various non-membership constructions in the literature, our primary goal is low circuit complexity, as opposed to time & space complexity for different operations. Designing circuit-friendly data structures and their algorithms need to pay special attention to the following:
- Input independence: circuit configuration is independent of any input values (namely
(pub_input, secret_witness)
assignments), specifically this implies:
- worse case complexity for loops: constraining an algorithm that contains
for (i=0; i<n; i++)
loop with potential early return/termination requires always checking all n
iterations in circuit.
- combined branch complexity for conditions: constraining an algorithm that contains
if (cond1) {do X} else if (cond2) {do Y} else {do Z}
conditional execution requires always constraining all branches and chained together in a Disjunctive Normal Form: (cond1=T AND cond2=F AND X) OR (cond1=F AND cond2=T AND Y) OR (cond1=F AND cond2=F AND Z)
.
- Circuit-unfriendly Operations: algebraic circuits natively execute finite field arithmetics and many operations could be expensive when translating to field operations, such as pairing in bilinear map, bit-wise operation (AND, XOR, SHL etc.), non-algebraic hash, non-native modular arithmetics. Apparently we want to minimize these operations as much as possible.
Here are some concrete implications:
- Sparse Merkle Tree (SMT) is very deep thus involves many hash operations for each Merkle path (accumulating field elements of 256 bits would require a binary SMT of height 256). Potential optimization leveraging on the observation that: "expected length of Merkle proof with accumulated elements is only " leads to a rather complicated circuit logic dealing with many edge cases and occational longer paths.
- RSA accumulator requires modular arithmetics over a usually much larger modulus (for security) than that of the native field, which leads to high circuit complexity.
- Bilinear-map based accumulator usually involves pairing, another expensive operations. Worse, the number of group points in the Structured Reference String (SRS) required for these accumulators (such as [DT08]) are linear with tree size. Currently, the largest trusted setup ceremony provides about such points, however, even for Zcash's Nullifier set is at least in capacity.
Overview
Core idea 1 (interval as leaf value): transforming non-membership proof in set to membership proof in set .
E.g. for set , we prove non-membership for by providing Merkle proof for the interval (in purple), and an interval check whether (in yellow):
Core idea 2 (replace and append): During insertion of new member, split the containing interval into two smaller intervals. Replace the old interval, then append the other.
E.g. inserting will split into .
Note that:
- The tree is of fixed height, not an incremental tree.
- Intervals are bounded exclusively below and above (namely ).
- Intervals are unsorted/non-consecutive.
- Our IMT is a universal accumulator with dyanmic addition and deletion of elements, and support both membership and non-membership proofs.
- This accumulator is order dependent: different order of insertion of the same set may result in different tree state and digest.
- If were not constraining correct insertion in circuit, we can enforce consecutive intervals (namely shift right 1 slot to make room for ), which will give order independence.
- Unfortunately, to keep the circuit input dependent, we need to ensure a fixed number of "steps" for each insertion regardless of the state of the Merkle tree.
- The most efficient solution is leaving these intervals unordered during insertion, which necessitates another "locator" to help remember the leaf indexes of inserted intervals for fast locating. We instantiate this "locator" with a B-Tree based Map in practice. (detailed description below)
- due to order dependence, we should view IMT as a vector accumulator scheme (rather than a ) and the accumulator as a vector commitment.
- This IMT was proposed way earlier by Philippe et al. in 2008, but there are two problems of this data structure that potentially hinders its wider adoption in practice:
- requiring structural integrity check:
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
TODO (add def)
- order dependence (as mentioned above): this results in either additional locator to help quickly locate insertion position at the cost of extra storage; or require sorting upon insertion & deletion which increase the cost of both operation to .
Construction
We adapt from the definition, syntax and notation of a trapdoorless universal accumulator from [BBF18]:
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
For optimization, we extend the definition above with one more algorithm:
There are auxiliary output for accumulator update algorithms (namely ) to help provers generate proof of correct update, which we highlight with
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
.Note that due to impossibility of batch witness update by [CH09], witness update for each time counter is enough, since info for batch update independent of number of updates (namely ) doesn't exist.
We further assume the existence of an algebraic hash function , which we instantiate using Rescue or Poseidon, modelled as a random oracle.
We denote the bit-size of a prime field with modulus , any number represents a field element unless stated otherwise.
We denote merkle path as the nodes along the path from leaf to root; merkle proof as the slibing nodes along that path.
Our locator search trees should have the following APIs:
Setup
- Inputs:
- security param:
- initial set elements:
- Outputs:
- public param: include field , tree size , hash function for internal nodes , for leaves ,
- initial accumulator value: contains the root value plus a counter keeping track of number of non-empty leaves in the tree.
- Algorithms:
- Initialize locator tree:
let locator = BTreeMap::new();
- Initialize IMT of size with the first leaf being to represent the ideal using field elements; and populate the rest with as EMPTY leaves.
- Derive the merkle root
rt
, and assign the initial accumulator value to signify currently only 1 non-empty leaves.
- Update locator tree:
locator.insert(F::from(0), 0);
.
Add
- Inputs
- current accumulator:
- a new element:
- Outputs
- updated accumlator:
- info used to update proof:
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
prover witness:
- Algorithms
- Query replacement index:
idx = locator.range(x).next().unwrap()
(idx=0, ELEMS[idx]=(0,p-1)
in this example)
- Prepare new leaf to be inserted:
- Prepare replaced leaf :
- Insert new leaf at position :
- add the Merkle proof of
ELEMS[ctr]
(exclude root) AND the new leaf to prover witenss: (purple nodes in diagram)
- append new leaf at position , update internal nodes, and finally compute the new root value
- append to proof update info: where is the merkle path of
ELEMS[ctr]
- in our example, path consists of update values (w.r.t. )
- update locator:
locator.insert((L,ctr)).unwrap();
- Replace
LEAVES[idx]
with :
- add the Merkle proof of
LEAVES[idx]
(exclude root which is ) to prover witness: (dashed nodes in the diagram, note that purple-dashed nodes were updated in the last step during insertion)
- note: is basically the non-membership proof. It might be a little counterintuitive to see such a non-membership proof from an updated tree (with new leaf inserted) instead of the original tree. The main rationale is to unify circuit logic with procedure
- replace
LEAVES[idx]
with , update internal nodes, and compute the new root value (transition from )
- append to proof update info:
- Output
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
- Inputs
- current accumulator:
- a batch of new elements:
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
we assume each batch has fixed size of for simpler circuit later, here we assume
- we further assume for some ,
(the problem with initial (0,p-1)
leaf during setup can be dealt with using some "genesis block" mechanism to pad into multiple of )
- Outputs
- updated accumlator:
- info used to update proof:
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
prover witness:
- Algorithm
- Prepare new leaves to be inserted:
- update locator:
locator.insert((L_i^new, ctr+i-1)).unwrap();
- these intervals might be updated, but their
L
bound won't, thus it's safe to insert their positions to locator now
- Prepare leaves to be replaced:
- query
idx_j = locator.range(x_j).next().unwrap();
LEAVES[idx_j].L
,
- Batch insert all new leaves:
- add merkle proof of the entire subtree to prover witness: (red nodes in diagrams)
- note the root of the empty subtree is a public parameter and can be pre-computed
- insert , compute the new subtree root, then update the overall tree root
- add all new leaves to prover witness:
- Replace leaves: for each , and its respective
idx_j
(searched during step 2):
- add merkle proof of
LEAVES[idx_j]
(against updated roots) to prover witness: (green nodes)
- replace leaf with , update roots
- add all replaced leaves to prover witness:
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
TODO: upmsg
- Output
Del
- Inputs
- current accumulator:
- element to be removed:
- Outputs
- updated accumlator:
- info used to update proof:
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
prover witness: (TODO: will specify this later)
- Algorithm
- Query replacement positions:
idx_0 = locator.range(&9-1).next().unwrap(); idx_1 = locator.get(&9).unwrap();
(in our example, idx_0=1, idx_1=3
)
- Replace
LEAVES[idx_0]
withx(LEAVES[idx_0].L, LEAVES[idx_1].R)`, updax Merkle root
- Replace
LEAVES[idx_1]
with LEAVES[ctr-1]
, update Merkle root
- append updated nodes:
- update locator:
locator.insert((LEAVES[ctr-1].L, idx_1)).unwrap();
and locator.remove(x).unwrap();
- Replace
LEAVES[ctr-1]
with (0,0)
, update Merkle root
- Decrement total non-empty leaves count:
ctr--
- Output
MemWitCreate
- Inputs
- accumulator at time :
- claimed member:
- vector of elements inserted: (unnecessary for us)
- Outputs
- Algorithm
- Query position:
idx = locator.get(9).unwrap();
(idx=3
in our example)
- Return the merkle proof of
LEAVES[idx]
(marked in purple) as
NonMemWitCreate
- Inputs
- accumulator at time :
- claimed non-member:
- vector of elements inserted: (unnecessary for us)
- Outputs
- Algorithm
- Query containing interval position:
idx = locator.get(12).unwrap();
(idx=3
in our example, checking in the same diagram above)
- Return the merkle proof of
LEAVES[idx]
(marked in purple) as
MemWitUp & NonMemWitUp (same algorithm)
- Inputs
- old accumulator:
- old (non)-membership proof: or
- claimed (non)-member:
- update info:
- Outputs
- new (non)-membership proof: or
- Algorithm
- Query index of (non)-member:
let idx = locator.get(x).unwrap();
-
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
TODO: this algorithm is not complete, it's possible your non-membership leaf position gets updated. the above two steps only deal with situation where leaf position wasn't changed (such as all membership proof)
VerMem
- Inputs
- accumulator at time :
- claimed member:
- membership witness:
- Outputs
- Algorithm
- Verify Merkle proof of leaf
- Check
VerNonMem
- Inputs
- accumulator at time :
- claimed member:
- non-membership witness:
- Outputs
- Algorithm
- Verify Merkle proof of leaf
- Check
Constraints
Building blocks
recompute_root(pos, leaf, proof) -> root
: given the leaf position pos
, leaf value leaf
and siblings of its path proof
, recompute the root
of this subtree
compute_root(leaves) -> root
: given leaves, compute the tree root.
Batch Check & Insert Gadget
- Statement: Given a correctly computed old accumulator of the IMT, there exists a vector of new non-members such that applying will result in new accumulator value .
- Public Input:
- Old accumulator:
- New accumulator:
- Root of empty subtree of size :
- Secret Witness:
- New vector of non-members:
- Prover witness from
BatchAdd
:
- Circuit Logic:
- Verify correct -subtree to insert:
- parse from (red nodes)
- enforce:
- Batch insert and update merkle root:
- parse from (red dashed nodes)
- check for
- compute new -subtree root:
- compute new root upon insertion:
- Check and replace, parse (blue dashed nodes) and
LEAVES[idx_j]
(parent of blue dashed nodes) from , let ,
for each , do:
- Verify non-membership:
- enforce
- enforce (
LEAVES[idx_j].L
) AND (LEAVES[idx_j].R
)
- note: we don't need to check because we effectively cover that during insertion
- Check correct replacement:
LEAVES[idx_j].L
- compute updated root after replacement:
- Finally, check new accumulator:
Security Proofs
We show that the structure integrity of our IMT can be captured by the following 3 invariants:
- All left range include all member exactly once plus a negative infinity represented by
0
:
- All right range include all member exactly once plus a positive infinity represented by
p-1
:
- for
Then we proceed to prove that our Add
, BatchAdd
, Del
preserves all 3 invariants.
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
TBDFinally, we argue that our constraints sufficiently enforce the BatchAdd
algorithm.
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
TBD
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
TODO: losing adaptive security (but still sufficient for our use cases) where adv hands you the accumulator and provide valid proof of membership AND non-membership.Caching Strategies
Given the locator B-Tree, we are able to deterministically reconstruct the IMT.
By caching some Merkle path in the IMT, we can achieve space-time efficiency tradeoff during tree update.
When using IMT in a non-universal setting (specifically insertion only, no deletion, such as our Nullifier set), we could enjoy the following pruning:
The main observation is that
- "smaller the interval, less likely the leaf will be updated, so are the nodes on its Merkle path to the root".
- On one extreme, leaves with interval value such as won't be updated.
- Merkle frontier will always gets updated during insertion, thus should be kept in cache.
Adapting to Different Arity
Given different arity (i.e. number of children per internal node) of the Merkle Tree, we need to adapt all algorithms and constraints above.
We first demonstrate 3-ary tree:
TBD: simply literature review/related work, plus some comparison to justify our design and advantanges.
SMT, Balaced Tree, RSA accumulators, Bilinear-map accumulators.