# Challenges with Estimating Transaction Fee Rates
**Fee Estimation** is the process of predicting the fee rate that a transaction should pay for it to be included in some number of blocks in the future.
The fee rate is the fee per transaction size.
Fee estimation is critical because it determines when a transaction is likely to be mined.
Transactions are prioritized based on their fee rate; higher rates ensure faster confirmations as miners favor them to maximize revenue.
Especially during congestion periods, low-fee rate transactions may linger, potentially never confirming.
Users must determine suitable fee rates, balancing confirmation speed and cost, aided by Fee Estimators.
Fee estimation is probabilistic and utilizes various data sources, including **historical block transactions**, **historical block transactions the mempool has seen confirmed**, and **unconfirmed mempool transactions**.
## Challenges of Fee Estimation with Historical Block Transactions
- Gameable by Miners, Miners including high feerate transactions in their mined blocks can influence future fee estimates. Those high feerate transactions will impact future predictions.
- This method overlooks the current state of miners' mempool, which may be congested with high feerate transactions or shallow with low-fee rate transactions. Relying solely on previous block fee rates can lead to overpayment or underpayment.
## Challenges of Mempool-Based Fee Estimation
- Accurate fee estimation depends on having a fully validating node with a synchronized mempool.
- Mempool Differences, The user's mempool may differ from the miner's mempool leading to inaccurate predictions.
- Relying on external fee estimation services might introduce the risk of service providers colluding with miners to artificially inflate fee estimates.
## Challenges of Fee Estimation with Historical Blocks and Mempool Transactions Data
- This approach, based on transactions in the node's mempool that have been confirmed, reduces the risk of miners manipulating future predictions by publishing high feerate transactions. The threat of other miners picking up these high feerate transactions discourages manipulation. However, this method does not consider the current mempool size and is "package un-aware" which reduces data points and in some cases leads to inaccurate assumptions.
## How Historical Blocks and Mempool Transactions Data Fee Estimation Falls Short
Bitcoin core uses this historical block and mempool transaction data to estimate fees, the component that handles estimating fees is called `CBlockPolicyEstimator`.
“Because it's solely based on historical data (looking at how long mempool transactions take to confirm), it cannot react quickly to changing conditions. ” ~sipa
`CBlockPolicyEstimator` observes how long mempool transactions historically take to confirm. It assumes that if some number of mempool transactions with a certain fee rate have been confirmed after a specific number of blocks in the past, using that same fee rate for your transaction will likely result in a similar confirmation time.
However, this assumption may not hold if the mempool's state changes. The historical data may not accurately reflect the current situation.
“Because it's aiming to match seen behaviour rather than a requirement, if some non-negligible fraction of users keeps paying a certain high feerate, it may try to match that, even if it is unnecessary
for confirmation.” ~sipa
If a node's mempool primarily contains high feerate transactions due to mempool differences, the mempool transactions that end up confirming will also be high feerate transactions. This can lead to the `CBlockPolicyEstimator` producing high fee rate estimates, transactions with such high fee rates will likely be confirmed in a short time but at the cost of paying more than necessary.
Similarly, if the node’s mempool predominantly comprises low-fee-rate transactions, and the mempool transactions that confirm also have low fee rates, the `CBlockPolicyEstimator` will tend to provide lower fee estimates, miners might not likely pick that transaction because they are not incentivised enough to include the transaction in block. This reflects the estimator’s tendency to align with the observed behaviour of transactions that have historically been confirmed.
“The new fee estimator is much better at reacting to large volatility in tx throughput, but ignores the mempool” ~kallewoof
The statement implies that the `CBlockPolicyEstimatr` is designed to react and adjust fee estimates based on historical data and trends related to how mempool transaction fee rates are confirmed over time. It takes into account the past behaviour of transactions in terms of their fee rates and confirmation times to predict how fee rates will be in the future. However, it may not consider real-time or current conditions in the mempool when making these estimations, which could lead to potential discrepancies or inaccuracies during times of sudden changes or fluctuations in transaction throughput
## Analysis to Confirm that `CBlockPolicyEstimator` is unreliable in some situations
I have been watching the blockchain for a while to verify the behavior of the `CBlockPolicyEstimator`; one of the instances where the mempool experienced congestion was between blocks **812593** and **812596**, this is an example of where the Bitcoin core fee estimator was not reliable.
### Fee rate estimates from **812593** to **812596**
| Block Height | EstimateSmartFee (2 blocks) | Mempool Based Fee Estimate(High priority) | Average Fee Rate | Median Fee Rate |
|-------------------------------------------|-----------------------------|-----------------------------|------------------|-----------------|
| Immediately after 812593 confirmed | 18 s/vb | 22 s/vb | N/A | N/A |
| 45 minutes after 812593 | 18 s/vb | 33 s/vb | N/A | N/A |
| Block 812594 was confirmed | 18 s/vb | 33 s/vb | 44 s/vb | 33 s/vb |
| Block 812595 was confirmed | 18 s/vb | 22 s/vb | 25 s/vb | 22 s/vb |
| Block 812596 was confirmed | 20 s/vb | 22 s/vb | 23 s/vb | 19 s/vb |
](https://hackmd.io/_uploads/Bk3FYPRzT.png)
The estimates above demonstrate how the `CBlockPolicyEstimator` operates during congestion, completely independent of the mempool's condition.
Thanks to Ishanaam and other core developers of Bitcoin, I was able to classify the wallets used by the mempool transactions dumps between **812594** and **812596** with the [wallet printing](https://ishaana.com/blog/wallet_fingerprinting/
) tool.
I identified Bitcoin core transactions by their wallet fingerprints and determined whether they use estimates from the `CBlockPolicyEstimator` and the outcomes of the transactions that use the `CBlockPolicyEstimator` and those that don't below.
After **812593** block was mined, a new block was not found until after more than one hour which made the mempool congested with transactions and some users started broadcasting high fee rate transactions/ fee bumping already published ones to increase the chances of their transaction being picked by miners due to the congestion which in turn raise the fee estimate to **33s/vb** while `CBlockPolicyEstimator` is still feeding **18s/vb** for high priority fee estimates.
The mempool state at that moment is below.
<iframe width="600" height="371" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vQrOX3TK3AswjhfNfc9gOx93Z-ODDID2GQlapoW0YlwB_v4lkjx17_A-5CasONVr7kyhBne2x6qmXpm/pubchart?oid=1181013940&format=interactive"></iframe>
The high-priority target fee rate of **18s/vb** by `CBlockPolicyEstimator` is significantly below the ancestor score of **38%** of fingerprinted Bitcoin core transactions.
This demonstrates unequivocally that all those transactions get their fee estimates from somewhere else.
For example, they might be utilizing the `fee_rate` field of `sendtoaddress` RPC with [mempool.space](mempool.space) feerate estimate value.
<iframe width="600" height="371" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vQrOX3TK3AswjhfNfc9gOx93Z-ODDID2GQlapoW0YlwB_v4lkjx17_A-5CasONVr7kyhBne2x6qmXpm/pubchart?oid=1061368107&format=interactive"></iframe>
There are **116** transactions whose ancestor score falls within the **18s/vb** of the `CBlockPolicyEstimator` high priority estimate range. which accounts for **12.6%** of all Bitcoin core transactions.
It is implied that the remaining **49.4%** of Bitcoin core transactions may or may not be using `CBlockPolicyEstimator` as their Fee estimator, because their ancestor score was below that of `CBlockPolicyEstimator`.
After an hour and a few minutes, block **812594** was mined.
Below is the mempool state after **812594**.
<iframe width="600" height="371" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vQrOX3TK3AswjhfNfc9gOx93Z-ODDID2GQlapoW0YlwB_v4lkjx17_A-5CasONVr7kyhBne2x6qmXpm/pubchart?oid=1278002772&format=interactive"></iframe>
**30%** of fingerprinted Bitcoin core transactions after **812594** had an ancestor score above the `CBlockPolicyEstimator` high priority target fee rate.
Some of the transactions paying more than the `CBlockPolicyEstimator` high priority target fee rate were mined.
<iframe width="600" height="371" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vQrOX3TK3AswjhfNfc9gOx93Z-ODDID2GQlapoW0YlwB_v4lkjx17_A-5CasONVr7kyhBne2x6qmXpm/pubchart?oid=330620633&format=interactive"></iframe>
Transactions whose ancestor score are within the `CBlockPolicyEstimator` high priority estimate range are now: **128**, increasing to **14%** of
transactions.
None of the transactions with high priority estimate was mined and we have additional once broadcasted with the same **18s/vb** estimate.
`CBlockPolicyEstimator` is still feeding **18s/vb** estimate, while the mempool is still congested and the mempool based fee estimate is still **33s/vb**.
Block **812595** was discovered within a short time.
<iframe width="600" height="371" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vQrOX3TK3AswjhfNfc9gOx93Z-ODDID2GQlapoW0YlwB_v4lkjx17_A-5CasONVr7kyhBne2x6qmXpm/pubchart?oid=1405156454&format=interactive"></iframe>
Only **12%** of fingerprinted Bitcoin core transactions after **812595** have an ancestor score higher than the high priority target fee rate set by **CBlockPolicyEstimator**.
<iframe width="600" height="371" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vQrOX3TK3AswjhfNfc9gOx93Z-ODDID2GQlapoW0YlwB_v4lkjx17_A-5CasONVr7kyhBne2x6qmXpm/pubchart?oid=687911508&format=interactive"></iframe>
Transactions whose ancestor score is within the **CBlockPolicyEstimator** high priority estimate range are: **139** rising to **18%** of Bitcoin core transactions, `CBlockPolicyEstimator` high priority estimate assures us that a transaction using **18s/vb** is going to confirm in the next two blocks.
It's been two blocks already and transactions using that estimate have not been confirmed.
`CBlockPolicyEstimator` is still **18s/vb**, At that time mempool based fee estimator reduced its estimate to **22s/vb** while `CBlockPolicyEstimator` was still feeding **18s/vb**.
Block **812596** was also mined after a few minutes
<iframe width="600" height="371" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vQrOX3TK3AswjhfNfc9gOx93Z-ODDID2GQlapoW0YlwB_v4lkjx17_A-5CasONVr7kyhBne2x6qmXpm/pubchart?oid=1428828125&format=interactive"></iframe>
The mempool contained only **0.5%** of fingerprinted Bitcoin core transactions whose ancestor score was higher than the `CblockPolicyEstimator` high priority target fee rate.
Almost all Bitcoin core transactions with higher fees than `CBlockPolicyEstimator` which estimated their fee rate from a different source were mined.
<iframe width="600" height="371" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vQrOX3TK3AswjhfNfc9gOx93Z-ODDID2GQlapoW0YlwB_v4lkjx17_A-5CasONVr7kyhBne2x6qmXpm/pubchart?oid=470892768&format=interactive"></iframe>
However, even after three blocks, there are still about **77** transactions that are stuck in the mempool. This is most likely due to the ancestor score of **18s/vb**, which makes up **14%** of the fingerprinted core Bitcoin transactions.
During this period, `CBlockPolicyEstimator` started adjusting the estimate, going from **18s/vb** to **20s/vb**, due to the high fee rates on the mempool transactions that were confirmed.
## Why Fee Estimation Is Challenging
Estimating fees can be quite a challenge for several reasons. One of the main hurdles is the unpredictable block confirmation times.
Sometimes, blocks are mined swiftly, while at other times, they can take over an hour.
When blocks are mined quickly, the miners' mempool is cleared, resulting in a shallow mempool with lower fee rate transactions because the higher fee rate transactions have likely already been mined.
This, in turn, reduces the fee rate required for a transaction to be included in the next block. If fee estimators do not consider the current state of the mempool, users might end up paying more than necessary.
On the other hand, during congested periods with longer block confirmation times, failing to consider the state and size of the mempool can result in transactions getting stuck in the mempool due to underpayment.
However, accounting for the state of the mempool is a complex task for users. Each user has their unique view of the mempool, which can vary based on their connectivity and the policy rules they follow. If a user's mempool is not closely synchronized with that of the miners, it can lead to biased fee estimates.
For instance, if a user's mempool mainly consists of low-fee-rate transactions while miners primarily have high feerate transactions, the user might create transactions that get stuck due to an insufficient fee rate.
Similarly, if the user's mempool predominantly contains high fee rate transactions while miners have mostly lower fee rate transactions, the user might end up overpaying, as their fee estimates would be skewed towards the higher end.
Failing to consider the current mempool state can lead to delayed or overpriced transactions. Users' unique mempool views further complicate the process. Mismatched mempool contents can result in biased fee estimates, with low fee rates mempool leading to stuck transactions and high fee rates mempool causing overpayment.
### Improvement Proposal
A mempool-based fee estimator is presently being proposed in Bitcoin Core. It will be checked to make sure that nodes' mempools roughly align with the global mempool of miners.
If the node mempool policy differs from the global miners mempool, `CBlockPolicyEstimator` should be used.
Until it is observed that the node has a mempool that is nearly identical to the miners' mempool, Mempool based fee estimator will not be reliable.
These are heuristics, the proposal is still in its early phases, and more insights and reviews of the idea are required to make sure the checks are functioning as intended.
### In summary
Because of the aforementioned constraints, no estimate can ever be completely accurate in predicting future events. Don't worry if your transaction is taking significantly longer to confirm than expected.
In most cases, you have got two tricks up your sleeve: 'replace by fee' and 'CPFP' – the fee bumping tag team.
While the details of replace by fee and CPFP, along with scenarios where they may not save a transaction from being stuck, are beyond the scope of this article, they make for an interesting [read.](https://gist.github.com/glozow/3de85d7e4fc58c5723f8fcbf21e5e795)
But in the case where a transaction overpaid, well, tough luck! There's no 'undo' button for that, as a valid transaction will remain valid and will be graciously picked up by a lucky miner – consider it your generous contribution to the miner's retirement fund.
Sign in with Wallet
Connect another wallet