---
# System prepended metadata

title: Data Pipeline (section 5.1?)

---

# Data Pipeline (section 5.1?)

We collected on-chain data from two Uniswap v3, CoW, and Chainlink price subgraphs. To perform this task, we utilized the DataStreams python library, which is a subgraph query utility package. We performed merges using Polars, following the Apache Arrow standards. This standard is an emerging in-memory columnar analytics that can accelerate data load times, reduce memory usage, and speed up calculations.

We restricted the data to WETH/USDC orders on CoW, Univ3 USDC/WETH .05%, and .3% fee pools, and for ETH/USD Chainlink price data. To obtain the Univ3 data, we queried the swaps schema for the USDC/WETH .05% and .3% fee pools. We merged the CoW trades data and Univ3 Swaps data for transactions at the same timestamp, adding decimal places for the amounts and calculating marginal swap execution prices for comparison.


### Misc Notes, needs to be organized
- trade directionality - we label the trade direction WETH -> USDC when WETH is in the CoW buy column and the UniV3 amountOut column. Conversely, the USDC -> WETH direction is when USDC is in the CoW sell column and the Univ3 amountIn column.
- the gas-adjusted price - we adjusted the execution price by adding the wei cost at each block if it is a WETH buy and subtracting if it is a WETH sell since the CoW solver execution price reflects the gas cost, whereas Univ3 and Chainlink do not.
- query optimization - Univ3 pools have a lot more swaps so we filter based on CoW timestamps. However since Chain has a lot less frequency than CoW, we don't apply this filter or need it as an optimization step.
- Forward Fill - In order to construct the final price dataframe, we create an index ranging from the CoW first and last trades. Then we forward fill Univ3, Chain, and CoW execution prices to the index to propagate the last valid observations forward through time.



### Sources
DataStreams - https://github.com/Evan-Kim2028/DataStreams
Polars - https://github.com/pola-rs/polars
CoW Subgraph - https://thegraph.com/hosted-service/subgraph/cowprotocol/cow
Uniswap V3 Messari Subgraph - https://api.thegraph.com/subgraphs/name/messari/uniswap-v3-ethereum
Uniswap Labs V3 Subgraph - https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v3
Chainlink Price Subgraph - https://api.thegraph.com/subgraphs/name/openpredict/chainlink-prices-subgraph
Dune Solver Names Query link - https://dune.com/queries/1941061