# Incident Postmortem: Dec-20-2022 ### Root Cause Analysis Sentiment uses a custom oracle to price [Curve Tricrypto LP Tokens](https://arbiscan.io/address/0x8e0b8c8bb9db49a46697f3a5bb8a308e744821d2) throughout the protocol. The oracle used [Curve's recommended method](https://github.com/curvefi/crypto_lp_pricing/blob/master/contracts/LPPrice_tricrypto_arbi.vy) to fetch the price of an LP Token: ```solidity! function getPrice(address) external view returns (uint) { return curveTriCryptoOracle.lp_price() * 1e18 / pool.price_oracle(1); } ``` `curveTriCryptoOracle.lp_price()` returns the USD price of the LP token. Since sentiment prices all tokens in terms of ETH we need to divide this USD price by the current ETH/USD price to compute the ETH-denomiated price of the token. We divide the price returned by the oracle by `pool.price_oracle(1)`, the price of ETH in the curve pool and this snippet forms the root cause of this bug. `pool.price_oracle(1)` reflects the internal price of ETH in the pool and is susceptible to large changes over time. In this case we saw the following price changes of `pool.price_oracle(1)` jump from $1216.55 to $1379.44 (+13.38%) in less than 1 sec between consecutive blocks `47637853` and `47637854`. Ahead of the liquidation in block `47637889`, the price of ETH was $1389.50 which represents a cumulative increase of 14.21% over the span of 12 seconds. There was no corresponding change in the market price of ETH during this duration. The large change in the ETH price returned by `pool.price_oracle(1)` caused a sudden drop in the price of the LP Token as returned by `getPrice` described above. The affected account had a significant staking position in the Convex Tricrypto pool that lost value as a result of this price change and pushed their health factor beyond the liquidation threshold which triggered a liquidation, as expected. The reason this price change only affected one account was because it was the only account operating at a high leverage (greater than 4x) with price exposure to the Tricrypto LP token. ### Impact This impact of the incident was limited to a single account. No other borrowers or LPs were affected. There was no protocol-wide insolvency or bad debt at any point in time. Before the sharp change in price, the account had 255.70 ETH worth of total assets in their account with 204.97 ETH worth of aggregate debt. Accounting for slippage during the liquidation, the account lost ~45 ETH when it was liquidated. ### Resolution Timeline ##### Dec-21-2022, 07:03 UTC The new oracle contract was deployed at `0x46447BbD557Cd433D2eadC5da6e31048D2684e86`, pending sign-off from auditors before it would be upgraded. ##### Dec-21-2022, 09:39 UTC We receieved confirmation about the fix being approved. ##### Dec-21-2022 12:00 UTC After performing final checks the oracle was [upgraded](https://arbiscan.io/tx/0x1310dbe4f209591b7b3a6eca0025abb938b24fcf4f8753001e3ce512d1f90c2a) and the issue was resolved. ### Corrective Measures While there have always been continuous realtime alerts and monitoring to make sure all our oracles are functioning as expected, this issue could have been mitigated if these processes were configured in a more granular manner to raise alerts when: * The price of an asset changes beyond a threshold within a particular time period * The price of an asset changes beyond a threshhold relative to the price of another asset within a particular time period Either of these alerts would have allowed us to preemptively detect the error in the oracle contract and fix it beforehand. ###### tags: `sentiment`