## Analysis of Bitcoind Policy Estimator Effectiveness for Long-Term Fee Estimation

### Objective
This analysis aims to evaluate the effectiveness of the Bitcoind Policy Estimator for confirmation targets greater than 2. A previous study assessed and reported on its effectiveness for confirmation targets of 1 and 2, as detailed in my post on [Mempool-based Fee Estimation](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/8?u=ismaelsadeeq).

### Methodology for collecting and analysing data
I modified the Bitcoin Core code to add a scheduler that logs fee estimates every minute and logs percentile fee rates upon receiving a new block. This modified [branch](https://github.com/ismaelsadeeq/bitcoin/tree/07-2024-log-estimates-longterm) has been running on my node for weeks. I then used a script to collect and clean the data from the logs, and finally used another [script](https://github.com/ismaelsadeeq/bitcoind-long-term-estimates-analysis/tree/master) to generate the summary.

### Confirmation Target Range
The estimator can provide fee estimates for confirmation targets ranging from 2 to 1008. This means it can estimate fees for a transaction to confirm within the next ~10 minutes to one week.

### Assumptions
If the Bitcoind Policy Estimator provides an estimate, for example, a $100$-block confirmation target at 10 $sat/vB$, it implies:
- Paying $10 sat/vB$ should result in the transaction being confirmed in any block from current height + $1$ to current height + $100$ interval.
- If the transaction is confirmed within the next $(1-100)$ blocks, the forecast is successful.
- If not, the forecast fails.

This assumption will be used in this analysis.

### Key Definitions for the Analysis

Given a fee rate forecast $d$ for a confirmation target $n$ at a certain block height $h$, we define the evaluation criteria as follows:

- **Overpaid Estimate**: There exists at least one block from $h + 1$ to $h + n$ where $d$ is greater than the 50th percentile fee rate of the block.

*Rationale*: This means that the forecasted fee rate $d$ is higher than the median fee rate in at least one block within the range from $h + 1$ to $h + n$, indicating that we pay more than necessary to get included.

- **Estimate Within the Range (Successful Forecast)**: There is no overpaid estimate and at least one block from $h + 1$ to $h + n$ has a 25th percentile fee rate less than $d$.
*Rationale*: This means that the forecasted fee rate $d$ is not overpaid in any block, and there is at least one block within the range from $h + 1$ to $h + n$ where the 25th percentile fee rate is less than $d$, indicating a successful forecast.

- **Underpaid Estimate**: All blocks from $h + 1$ to $h + n$ have a 25th percentile fee rate above $d$.

*Rationale*: This means that in every block within the range from $h + 1$ to $h + n$, the fee rate at the 25th percentile is greater than $d$, indicating that the forecasted fee rate $d$ is lower than the actual required fee rate in all these blocks.

### Confirmation Targets
I've used 12 confirmation targets in this analysis $(3, 5, 10, 20, 50, 100, 202, 500, 701, 800, 1000, 1008)$

### Summary of Results
A total of 198,936 estimates were made from Block 851695 to Block 853515.

Bitcoind Policy Estimator uses two modes for estimating fees: `conservative` and `economical`. These modes were briefly described in a recent post [Bitcoind Policy Estimator Modes Analysis](https://delvingbitcoin.org/t/bitcoind-policy-estimator-modes-analysis/964#definitions-2)

#### Conservative Mode

| Conf Target | Underpaid Estimates | Overpaid Estimates | Estimates Within the Range |
|-------------|----------------------|--------------------|----------------------------|
| 3 | 0 (0.0%) | 8369 (50.48%) | 8209 (49.52%) |
| 5 | 0 (0.0%) | 9492 (57.26%) | 7086 (42.74%) |
| 10 | 0 (0.0%) | 11035 (66.56%) | 5543 (33.44%) |
| 20 | 0 (0.0%) | 11984 (72.29%) | 4594 (27.71%) |
| 50 | 0 (0.0%) | 12535 (75.61%) | 4043 (24.39%) |
| 100 | 0 (0.0%) | 12633 (76.20%) | 3945 (23.80%) |
| 202 | 0 (0.0%) | 10831 (65.33%) | 5747 (34.67%) |
| 500 | 0 (0.0%) | 8844 (53.35%) | 7734 (46.65%) |
| 701 | 0 (0.0%) | 10256 (61.87%) | 6322 (38.13%) |
| 800 | 0 (0.0%) | 10256 (61.87%) | 6322 (38.13%) |
| 1000 | 0 (0.0%) | 10256 (61.87%) | 6322 (38.13%) |
| 1008 | 0 (0.0%) | 10256 (61.87%) | 6322 (38.13%) |

#### Economical Mode

| Conf Target | Underpaid Estimates | Overpaid Estimates | Estimates Within the Range |
|-------------|----------------------|--------------------|----------------------------|
| 3 | 0 (0.0%) | 5941 (35.84%) | 10637 (64.16%) |
| 5 | 0 (0.0%) | 5503 (33.19%) | 11075 (66.81%) |
| 10 | 0 (0.0%) | 5275 (31.82%) | 11303 (68.18%) |
| 20 | 0 (0.0%) | 5089 (30.70%) | 11489 (69.30%) |
| 50 | 0 (0.0%) | 5305 (32.00%) | 11273 (68.00%) |
| 100 | 0 (0.0%) | 6294 (37.97%) | 10284 (62.03%) |
| 202 | 0 (0.0%) | 7345 (44.31%) | 9233 (55.69%) |
| 500 | 0 (0.0%) | 8844 (53.35%) | 7734 (46.65%) |
| 701 | 0 (0.0%) | 10256 (61.87%) | 6322 (38.13%) |
| 800 | 0 (0.0%) | 10256 (61.87%) | 6322 (38.13%) |
| 1000 | 0 (0.0%) | 10256 (61.87%) | 6322 (38.13%) |
| 1008 | 0 (0.0%) | 10256 (61.87%) | 6322 (38.13%) |

### Key Findings

- The data shows that for long-term fee estimation in both modes, there are no instances of underpayment at all from block `851695` to block `853515`, we can imply from this that there is a tendency of having few to no underpaid estimates in long-term fee estimation.
- We can also observe that the `conservative` mode has a higher number of overpaid estimates.
- On the other hand, the `economical` mode has a higher chance of getting a transaction confirmed within the target with a lower likelihood of overestimating.
- This data also indicates that the `economical` mode is more efficient for ensuring transactions are confirmed within the target.
- The data also shows that long-term fee rate forecast in Bitcoind has a high chance of being successful, at a risk of overestimating sometimes.
- In `economical` mode, as the confirmation target gets higher, the likelihood of overestimating increases.

### Summary

The results indicate that the Bitcoind Policy estimator long-term feerate forecasts are effective in ensuring transactions confirm within the target, unlike Bitcoind Policy estimator ASAP forecast.

One aspect worth exploring further is the practical applications for long-term fee estimation. One possible use case is for users who prefer not to pay high fees and are willing to wait for some time. It would be interesting to identify the specific user demographics that benefit most from this strategy.

Another side question is should we consider using time as the primary input for fee estimation instead of blocks, like ASAP (As soon as possible), the next 1 hour, 4 hours, 2 days etc?

Thanks for reading

---

- [Mempool-based Fee Estimation](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/)
- [Modified Source Code for collecting data](https://github.com/ismaelsadeeq/bitcoin/tree/07-2024-log-estimates-longterm)
- [Bitcoind Policy Estimator Modes Analysis](https://delvingbitcoin.org/t/bitcoind-policy-estimator-modes-analysis/964)
- [Bitcoind Long-Term Estimates Analysis script](https://github.com/ismaelsadeeq/bitcoind-long-term-estimates-analysis/tree/master)