Challenges with Estimating Transaction Fee Rates

Fee Estimation is the process of predicting the fee rate that a transaction should pay for it to be included in some number of blocks in the future.
The fee rate is the fee per transaction size.

Fee estimation is critical because it determines when a transaction is likely to be mined.
Transactions are prioritized based on their fee rate; higher rates ensure faster confirmations as miners favor them to maximize revenue.

Especially during congestion periods, low-fee rate transactions may linger, potentially never confirming.

Users must determine suitable fee rates, balancing confirmation speed and cost, aided by Fee Estimators.

Fee estimation is probabilistic and utilizes various data sources, including historical block transactions, historical block transactions the mempool has seen confirmed, and unconfirmed mempool transactions.

Challenges of Fee Estimation with Historical Block Transactions

• Gameable by Miners, Miners including high feerate transactions in their mined blocks can influence future fee estimates. Those high feerate transactions will impact future predictions.
• This method overlooks the current state of miners' mempool, which may be congested with high feerate transactions or shallow with low-fee rate transactions. Relying solely on previous block fee rates can lead to overpayment or underpayment.

Challenges of Mempool-Based Fee Estimation

• Accurate fee estimation depends on having a fully validating node with a synchronized mempool.
• Mempool Differences, The user's mempool may differ from the miner's mempool leading to inaccurate predictions.
• Relying on external fee estimation services might introduce the risk of service providers colluding with miners to artificially inflate fee estimates.

Challenges of Fee Estimation with Historical Blocks and Mempool Transactions Data

• This approach, based on transactions in the node's mempool that have been confirmed, reduces the risk of miners manipulating future predictions by publishing high feerate transactions. The threat of other miners picking up these high feerate transactions discourages manipulation. However, this method does not consider the current mempool size and is "package un-aware" which reduces data points and in some cases leads to inaccurate assumptions.

How Historical Blocks and Mempool Transactions Data Fee Estimation Falls Short

Bitcoin core uses this historical block and mempool transaction data to estimate fees, the component that handles estimating fees is called CBlockPolicyEstimator.

“Because it's solely based on historical data (looking at how long mempool transactions take to confirm), it cannot react quickly to changing conditions. ” ~sipa

CBlockPolicyEstimator observes how long mempool transactions historically take to confirm. It assumes that if some number of mempool transactions with a certain fee rate have been confirmed after a specific number of blocks in the past, using that same fee rate for your transaction will likely result in a similar confirmation time.

However, this assumption may not hold if the mempool's state changes. The historical data may not accurately reflect the current situation.

“Because it's aiming to match seen behaviour rather than a requirement, if some non-negligible fraction of users keeps paying a certain high feerate, it may try to match that, even if it is unnecessary
for confirmation.” ~sipa

If a node's mempool primarily contains high feerate transactions due to mempool differences, the mempool transactions that end up confirming will also be high feerate transactions. This can lead to the CBlockPolicyEstimator producing high fee rate estimates, transactions with such high fee rates will likely be confirmed in a short time but at the cost of paying more than necessary.

Similarly, if the node’s mempool predominantly comprises low-fee-rate transactions, and the mempool transactions that confirm also have low fee rates, the CBlockPolicyEstimator will tend to provide lower fee estimates, miners might not likely pick that transaction because they are not incentivised enough to include the transaction in block. This reflects the estimator’s tendency to align with the observed behaviour of transactions that have historically been confirmed.

“The new fee estimator is much better at reacting to large volatility in tx throughput, but ignores the mempool” ~kallewoof

The statement implies that the CBlockPolicyEstimatr is designed to react and adjust fee estimates based on historical data and trends related to how mempool transaction fee rates are confirmed over time. It takes into account the past behaviour of transactions in terms of their fee rates and confirmation times to predict how fee rates will be in the future. However, it may not consider real-time or current conditions in the mempool when making these estimations, which could lead to potential discrepancies or inaccuracies during times of sudden changes or fluctuations in transaction throughput

Analysis to Confirm that CBlockPolicyEstimator is unreliable in some situations

I have been watching the blockchain for a while to verify the behavior of the CBlockPolicyEstimator; one of the instances where the mempool experienced congestion was between blocks 812593 and 812596, this is an example of where the Bitcoin core fee estimator was not reliable.

Fee rate estimates from 812593 to 812596

Block Height	EstimateSmartFee (2 blocks)	Mempool Based Fee Estimate(High priority)	Average Fee Rate	Median Fee Rate
Immediately after 812593 confirmed	18 s/vb	22 s/vb	N/A	N/A
45 minutes after 812593	18 s/vb	33 s/vb	N/A	N/A
Block 812594 was confirmed	18 s/vb	33 s/vb	44 s/vb	33 s/vb
Block 812595 was confirmed	18 s/vb	22 s/vb	25 s/vb	22 s/vb
Block 812596 was confirmed	20 s/vb	22 s/vb	23 s/vb	19 s/vb

The estimates above demonstrate how the CBlockPolicyEstimator operates during congestion, completely independent of the mempool's condition.

Thanks to Ishanaam and other core developers of Bitcoin, I was able to classify the wallets used by the mempool transactions dumps between 812594 and 812596 with the wallet printing tool.

I identified Bitcoin core transactions by their wallet fingerprints and determined whether they use estimates from the CBlockPolicyEstimator and the outcomes of the transactions that use the CBlockPolicyEstimator and those that don't below.

After 812593 block was mined, a new block was not found until after more than one hour which made the mempool congested with transactions and some users started broadcasting high fee rate transactions/ fee bumping already published ones to increase the chances of their transaction being picked by miners due to the congestion which in turn raise the fee estimate to 33s/vb while CBlockPolicyEstimator is still feeding 18s/vb for high priority fee estimates.

The mempool state at that moment is below.

The high-priority target fee rate of 18s/vb by CBlockPolicyEstimator is significantly below the ancestor score of 38% of fingerprinted Bitcoin core transactions.

This demonstrates unequivocally that all those transactions get their fee estimates from somewhere else.
For example, they might be utilizing the fee_rate field of sendtoaddress RPC with mempool.space feerate estimate value.

There are 116 transactions whose ancestor score falls within the 18s/vb of the CBlockPolicyEstimator high priority estimate range. which accounts for 12.6% of all Bitcoin core transactions.

It is implied that the remaining 49.4% of Bitcoin core transactions may or may not be using CBlockPolicyEstimator as their Fee estimator, because their ancestor score was below that of CBlockPolicyEstimator.

After an hour and a few minutes, block 812594 was mined.

Below is the mempool state after 812594.

30% of fingerprinted Bitcoin core transactions after 812594 had an ancestor score above the CBlockPolicyEstimator high priority target fee rate.

Some of the transactions paying more than the CBlockPolicyEstimator high priority target fee rate were mined.

Transactions whose ancestor score are within the CBlockPolicyEstimator high priority estimate range are now: 128, increasing to 14% of
transactions.

None of the transactions with high priority estimate was mined and we have additional once broadcasted with the same 18s/vb estimate.
CBlockPolicyEstimator is still feeding 18s/vb estimate, while the mempool is still congested and the mempool based fee estimate is still 33s/vb.

Block 812595 was discovered within a short time.

Only 12% of fingerprinted Bitcoin core transactions after 812595 have an ancestor score higher than the high priority target fee rate set by CBlockPolicyEstimator.

Transactions whose ancestor score is within the CBlockPolicyEstimator high priority estimate range are: 139 rising to 18% of Bitcoin core transactions, CBlockPolicyEstimator high priority estimate assures us that a transaction using 18s/vb is going to confirm in the next two blocks.

It's been two blocks already and transactions using that estimate have not been confirmed.
CBlockPolicyEstimator is still 18s/vb, At that time mempool based fee estimator reduced its estimate to 22s/vb while CBlockPolicyEstimator was still feeding 18s/vb.

Block 812596 was also mined after a few minutes

The mempool contained only 0.5% of fingerprinted Bitcoin core transactions whose ancestor score was higher than the CblockPolicyEstimator high priority target fee rate.

Almost all Bitcoin core transactions with higher fees than CBlockPolicyEstimator which estimated their fee rate from a different source were mined.

However, even after three blocks, there are still about 77 transactions that are stuck in the mempool. This is most likely due to the ancestor score of 18s/vb, which makes up 14% of the fingerprinted core Bitcoin transactions.

During this period, CBlockPolicyEstimator started adjusting the estimate, going from 18s/vb to 20s/vb, due to the high fee rates on the mempool transactions that were confirmed.

Why Fee Estimation Is Challenging

Estimating fees can be quite a challenge for several reasons. One of the main hurdles is the unpredictable block confirmation times.
Sometimes, blocks are mined swiftly, while at other times, they can take over an hour.
When blocks are mined quickly, the miners' mempool is cleared, resulting in a shallow mempool with lower fee rate transactions because the higher fee rate transactions have likely already been mined.
This, in turn, reduces the fee rate required for a transaction to be included in the next block. If fee estimators do not consider the current state of the mempool, users might end up paying more than necessary.

On the other hand, during congested periods with longer block confirmation times, failing to consider the state and size of the mempool can result in transactions getting stuck in the mempool due to underpayment.

However, accounting for the state of the mempool is a complex task for users. Each user has their unique view of the mempool, which can vary based on their connectivity and the policy rules they follow. If a user's mempool is not closely synchronized with that of the miners, it can lead to biased fee estimates.

For instance, if a user's mempool mainly consists of low-fee-rate transactions while miners primarily have high feerate transactions, the user might create transactions that get stuck due to an insufficient fee rate.
Similarly, if the user's mempool predominantly contains high fee rate transactions while miners have mostly lower fee rate transactions, the user might end up overpaying, as their fee estimates would be skewed towards the higher end.

Failing to consider the current mempool state can lead to delayed or overpriced transactions. Users' unique mempool views further complicate the process. Mismatched mempool contents can result in biased fee estimates, with low fee rates mempool leading to stuck transactions and high fee rates mempool causing overpayment.

Improvement Proposal

A mempool-based fee estimator is presently being proposed in Bitcoin Core. It will be checked to make sure that nodes' mempools roughly align with the global mempool of miners.
If the node mempool policy differs from the global miners mempool, CBlockPolicyEstimator should be used.
Until it is observed that the node has a mempool that is nearly identical to the miners' mempool, Mempool based fee estimator will not be reliable.

These are heuristics, the proposal is still in its early phases, and more insights and reviews of the idea are required to make sure the checks are functioning as intended.

In summary

Because of the aforementioned constraints, no estimate can ever be completely accurate in predicting future events. Don't worry if your transaction is taking significantly longer to confirm than expected.
In most cases, you have got two tricks up your sleeve: 'replace by fee' and 'CPFP' – the fee bumping tag team.
While the details of replace by fee and CPFP, along with scenarios where they may not save a transaction from being stuck, are beyond the scope of this article, they make for an interesting read.
But in the case where a transaction overpaid, well, tough luck! There's no 'undo' button for that, as a valid transaction will remain valid and will be graciously picked up by a lucky miner – consider it your generous contribution to the miner's retirement fund.