# Mempool Archive Quickstart 📚

The Mempool Archive is a historical dataset of Ethereum mempool transactions. The dataset encompases 15 million detection events starting from August 1, 2019. The Mempool Archive set is available for research and may be freely used for non-commercial purposes. To sign up go to [Blocknative Mempool Data Program](https://docs.blocknative.com/mempool-data-program)

## Private Transaction Volume Attribution
When a public transaction first enters the mempool, it's marked with a `pending` status. At block inclusion time, the public transaction status updates to `confirmed`. Transactions bypassing the public mempool are known as private transactions. In the Mempool archive, private transactions only have a `confirmed` status.

We can attribute private transaction volume into OFAs, MEV Bots and the unknown.

Let's focus on June 1, 2023!

## Downloading the Data

### Mempool Archive
The dataset is partitioned by day and hour. For a detailed schema see: [Blocknative Mempool Archive - Schema](https://docs.blocknative.com/mempool-data-program#data-schema)

### Searcher Dataset

Searchers are a class of actors that bundle a group of transactions together for personal incentives. Before the rise of special RPCs, searches comprised of nearly 70% of private transaction volume.

There's been lots of work to label various searchers such as [this](https://bitwise.notion.site/bitwise/Creating-an-Open-Source-Searcher-Dataset-for-Public-Research-b2a9f853e815430fbd22a64b6978ec16#1f44230fdc23422ab4f29f75eac466a5), [this](https://frontier.tech/builder-dominance-and-searcher-dependence) and [that](https://eigenphi.io/) We'll use a combination of all of them.

### OFA (Order Flow Auctions)
Order Flow Auctions are special RPCs (remote procedure call). A RPC is communication protocol used to publish transactions to the chain. There are loads of different entities providing OFAs offering users a better transaction experience.

Features range from fast transaction inclusion to providing rebates for users through MEV creation.

Examples include [Flashbots Protect](https://docs.flashbots.net/flashbots-protect/overview), [MEVBlocker](https://mevblocker.io/), (and us) [Transaction Boost](https://www.blocknative.com/mev-protection).

Data sources include
[Dune - MEVBlocker](https://dune.com/beetle/mev-blocker-stats)
[MEV-Share Historical Hint Stream](https://docs.flashbots.net/flashbots-mev-share/searchers/event-stream)
[Transaction Boost](https://docs.flashbots.net/flashbots-mev-share/searchers/event-stream)

⚠️⚠️⚠️
When an onchain transaction is routed through several OFAs, it is not possible to determine the true source. We label any private volume as `Seen on Special RPC`

Not all OFAs share their order stream publicly, so the created dataset will always be a lower estimation of OFAs market share.
⚠️⚠️⚠️

## Exploring the Data
### Filtering out Private Transactions
Private transactions are transactions without a pending event in the Mempool Archive. There exists transactions lurking in the mempool for days to years. The time spent in the mempool is in the `timepending` column. This is calculated by

$$firstConfirmation-firstDetection$$

with millisecond units.

All private transactions with have a `timepending = 0`, since there weren't any pending events. We will also exclude any transactions with a status of `cancel` or `speedup`.

CREATE TABLE PRIVATE_TRANSACTIONS AS 
(
SELECT
  *
FROM
  MEMPOOL_ARCHIVE_SAMPLE
WHERE
  timepending = 0
  AND status not in ('speedup', 'cancel')
 )

Doing a count of total confirmed/failed, we see that 97% of private transactions were confirmed and 3% failed.


### MEV Bots
We classify the transactions into being sent by a MEV Bots or not by using our labelled searcher dataset.

22% of our private transaction sample were sent by MEV Bots.


### OFAs
Similar to above, we classify transaction into being seen on an OFA or not.

Nearly 50% of our private transaction sample can be attributed to OFAs.


## Beyond the Data

The share of private transactions have steadily increased with the introduction of OFAs. Adoption of OFAs will likely continue as they promise users speed and malicious MEV protection. Searchers will continue to seek privacy-preserving and guaranteed order execution. For these reasons, private mempools will be a mainstay.

Continued research into private transactions are important to avoid centralizing forces in the ecosystem.

We welcome the community to join us! Feel free to reach out to us through the usual channels.