## Post-mortem 10 Apr 2025 - EUR/USD

This thursday at 4.30pm CET, there was an outage on Pragma's EUR/USD mark price feed. This document will outline the course of events that led to this outage and the next steps taken to ensure it doesn't happen for this feed and future feeds being integrated.

### Timeline

1. In a discussion with our main data provider, we asked them to give us a test account so that we can test with new feeds without impacting the current API.
2. They updated the credentials of our account without any notice which in turns resulted in an outage of our publishing service, as the connection got closed and login was failing.
3. In the meantime, we used to have a secondary data provider as a fallback which also experienced some issue leading to incorrect decimals being published. This led to returning a mark price of `0.00011`.
4. This invalid mark price led to the liquidations of positions on Extended.
5. Thanksfully all of the funds could be retrieved and all the users refunded.

### Action Items

- Extended and Pragma will both add a "bounds" system to ensure a sudden move in raw data is not taken into account and invalid prices are not returned.
- Lmax was contacted to make sure they notice us ahead of time next time they do passwords change.
- The secondary data provider was removed from the publishers list.
- Upon the migration to the new Pragma API, the inclusion of other data sources could be done as a fallback mechanism. It is a tradeoff between resiliency and accuracy.