In a prior article - Part I: Bloom Filters - on simple payment verification for light clients, I explored how light clients utilize bloom filters for transaction verification, highlighting the reasons why these filters were implemented and some known flaws in their usage. The challenges are mostly related to privacy leaks, incentive misalignment for full nodes, and trust.
The key highlight from part I is this - BIP37 bloom filters are unsuitable for use and a new method of payment verification for clients must address the design flaws inherent in them. BIP37 peer-to-peer protocol has also been disabled by default on Bitcoin Core.
BIP 157 proposed a new protocol that permits transaction filtering on the client-side for light clients connected to at least one honest node securely verify transactions included in a block; without loss of privacy or reliance on trusted full nodes. It also eliminates I/O asymmetry between light and full nodes, and minimizes the DOS vector that full nodes were subject and vulnerable to. This proposal details probabilistic compact block filters with concrete implementation in BIP 158.
In this article, I will share what I have learned about the need for client-side transaction filtering, what compact block filters are, and how they are used in the network.
With bloom filters, the SPV client constructs a bloom filter and sends it to the full node. The full node then compares each transaction it receives to all of the bloom filters it has received, looking for a match. This means most of the computational overhead is placed on the full node, which makes it incentive-incompatible. With client-slide transaction filtering, this asymmetry is flipped.
Full nodes calculate one compact block filter (CBFs) per block regardless of the number of light clients connected to them. With CBFs, light clients check for matching scriptpubkeys in a set, and download a block only if there is a match or the possibility of a match.
Because light clients download blocks from different peers without worrying about other peers identifying that the downloaded blocks are blocks of interest, and never having to send their wallet addresses to peers, their privacy is preserved much better than with bloom filters.
A compact block filter is a condensed representation of the transactions in a block.
In the code block below, I have attached a simple Rust program to get the CBF of a specified block.
Shell output
Full nodes construct compact block filters for each block by:
The GCS is the block filter for the given block.
Light clients can, upon receiving the block filter, check for an address match by:
There are a handful of benefits of using block filters in comparison with bloom filters. Here I highlight the most obvious.
CBFs have header
s which are functions of the hash of the previous filter's header and the hash of the current filter. These headers make it possible to form a chain of block filters as shown below
This chain helps light clients to compare filter headers from different full nodes, which light clients can use to monitor filter chain divergence, and tell if nodes are sending false information. Unlike with bloom filters where there are no deterministic artifacts, light clients retrieve block filter headers. This reduces the trust requirements on light clients because they do not worry about full nodes omitting transaction information with little risk of detection [3][1].
Light clients also do not have to send any probabilistic filter of their addresses to full nodes, and thus protect their privacy. They also do not have to download blocks without first confirming there is a match, and even in cases where the filters indicate a match when there is none, i.e. false positive, the probability of that happening is 1 in 784931[1][3].
Full nodes, on the other hand, no longer have to continuously scan each incoming transaction for each bloom filter a light client sends to them. We established in Part I
that this is neither a verification technique that rewards the CPU work done nor does it lend itself to scale, opening full nodes to a DOS attack vector. Given the compact size of compact block filters, full nodes can do a one-time calculation, for each block, and save it to disk, allocating small space for them.
Although suggestive transaction information could be gotten by checking the mempool with a bloom filter, CBFs are calculated for confirmed blocks. This means that light clients have no way to get information on scriptpubkeys in relevant but unconfirmed transactions [7].
It is expected that the bandwidth for both full nodes and light clients will go up. This is because of the request-response cycle between nodes (full and light) to get filters, filter headers, and filter headers at spaced intervals. Light clients in particular have to sync the block header before they can download any filters or filter headers.
Compact block filters are another technique for light clients to use in transaction filtering and payment verification. They are constructed by hashing, sorting, and compressing the scriptpubkeys into a set for each transaction in every block. Light clients match their hashed scriptpubkeys against the decompressed set, only downloading blocks with a likely match.
These filters offer the benefits of privacy, and less trust, addressing the incentive-misalignment of full nodes matching addresses to bloom filters. With compact block filters, the risk of Denial of Service (DoS) attacks on full odes is also reduced.
On the contrary, there is an increase in bandwidth for light clients.