# Stake manager post mortem

Date: 2022-11-25

Authors: markus, ale

Status: Complete

### Summary:
On Youves, users receive YOU rewards for minting synthetic assets. A total of 15'000 YOU tokens are allocated per week for synthetic asset minters. Minters can claim these rewards at any time. The Stake Manager contact keeps track of each user's allocated YOU rewards represented in the `global_stakes` bigmap.

A bug in the referral feature of the Youves v3 engine contracts caused the `global_stake` of a referrer's vault to be set to the same value as the `global_stake` of a referee's vault once that referee minted synthetic assets. Once the referrer would interact (deposit, withdraw, mint, burn or touch) with the engine contract, the `global_stake` would be set back to the right value for that user.

A user tried to exploit this bug, by using some of the biggest vaults as referrers for new vaults in order to lower their `global_stake`. As the `global_stake` is used to determine the YOU rewards across all vaults of a user, this could have impacted the amount of YOU a user could have claimed during a limited period of time.

By replacing the Stake Manager contract and touching all vaults, the Youves team was able resolve this issue permanently. The impact on vault owners was limited.

### Impact:
Youves users who's addresses have been used as referrers had a wrong `global_stake` between the minting of a referee and the next interaction on any of the referrers vaults. Users who did claim with that wrong `global_stake` also received wrong amount of YOU tokens. Users with a wrong `global_stake` who didn't claim YOU during that period, were not affected at all.

No user deposited funds were at risk at any point in time.

### Detection:
On October 9, 2022 at 15:10, a Youves user reported an issue with the accrual of YOU rewards on Discord. This user observed much slower YOU accrual compared to the amount that was usually accrued on his account. The user also pointed out that this behaviour started once a referee using this users address as referral (introducer) minted a small amount of uUSD. He added, that an interaction with his vault, a burn operation, was resetting the accrual of YOU to the expected levels.

The Youves team started investigations on October 10, 2022 at 6:17 am, but it took until October 12, 2022 13:28 until we were sure that it must be a smart contract issue, while still searching for the root cause.

On Wednesday October 12, 2022 the root cause has been identified in a bug in the new v3 engines. It became clear that it was possible to exploit the bug in order to get an unfair share of YOU rewards. The team immediately started to search for solutions to mitigate the effects. As the new v3 engines were just deployed a few weeks ago, a solution that would not result in another engine migration was preferred.

While the team was working on the solution, a different user reported similar observations on Discord on. He stated that he was not actively sending out referral links, but still, his address has been used as a referrer. This raised concerns that somebody was already trying to exploit the bug and we started to look for transactions revealing a possible exploiter.

A user was found that tried to set back the stake value 7 of the biggest vaults over the course of 4 days, by using them as a referrer with small new vaults. All used referee addresses were funded by the same originating account.

### Root causes:
*Possibly an explanation, what the bug in the v3 engine was (flo or ale)*

### Trigger:
A bug in the referral feature of the Youves v3 engine contracts caused the `global_stake` of a referrer's vault to be set to the same value as the `global_stake` of a referee's vault once that referee minted synthetic assets, in most cases resulting in a temorary lower `global_stake` of the referrer. Once the referrer would interact (deposit, withdraw, mint, burn or touch) with the engine contract, the `global_stake` would be set back to the right value for that user.

### Resolution:
Affected vaults were touched to mitigate the impact of the exploiter while waiting for the new Stake Manager contract to be developed and deployed. A new Stake Manager contract (Stake Manager V2, `KT1Ge5usHwAAH12PDxwP8FFYMdbMbXSRXSz4`) was then deployed in order to correct the behaviour of the V3 engines. After deployment, all vaults have been touched in order to recalculate their users `global_stake`.

### Action Items:
| Action Item | Type | Owner | State |
| -------- | -------- | -------- |-------- |
| Identfiy reason for failure | mitigate | markus, florin |DONE |
| Develop and test new stake manager contract | prevent | florin, ale |DONE |
| Touch vaults to set correct stake | prevent | ale | DONE |
| Deploy new stake manager contract | prevent | florin |DONE |
| Rewire engines to use the new stake manager | prevent | multisig signers | DONE|
| Touch vaults to set correct stake on the new contract | mitigate | ale |DONE |

### Timeline:
(all times UTC)

2022-10-09 15:10 A Youves user reports slower hourly YOU acruals on Discord
2022-10-10 06:17 Youves team takes up investigations
2022-10-12 16:30 Problem is identified, research on how to solve is taken up
2022-10-15 to 2022-10-18 Exploiter sets down 7 user's `global_stake` by minting as referee
2022-10-17 08:11 Solution was found, development of Stake Manager V2 begins
2022-10-18 11:31 Vaults are touched to mitigate exploit impact
2022-10-18 01:12 Stake Manager V2 is sucessfully tested and [deployed](https://tzkt.io/oo37D2MJaNM7xQiCvBHBArYtEkcWTYavWkddHxksbV8g4yHLuML/47331609): `KT1Ge5usHwAAH12PDxwP8FFYMdbMbXSRXSz4`
2022-10-18 16:56 Multisig [executed](https://tzkt.io/onwjxN9PMts1DxXPC77fp4SpWke25iXxKje4TfQv7D35QcVwMFf/47331610) to wire all engines to the new stake manager
2022-10-18 16:58 All vaults are touched to set the correct stake

## Lessons Learned

### What went well
1. The multi contract architecture of Youves allowed for a solution without being forced to migrate the v3 engines again

### What went wrong
1. Not giving the problem the priority it would have deserved, because
1.1. we didn't find the root cause at first sight and didn't unterstand the impact immediately
1.2. any user interaction would set back the global stake to previous levels so the problem was not too visible
1.3. developers were under pressure to develop and ship the uUSD/uBTC savings pools contracts which were proposed in YIP-009
2. when testing the referral feature on the v3 engine, both, the referrers vault and the referees vaults/mints were small and around the same size, so it wasn't obvious that the referee's mint would change the stake of the referrer
3. A user was trying to exploit the issue (after the report on discord)

### Where we got lucky
1. Community members were reporting the issue
2. The wrong `global_stake` was set back to normal as soon as a user interacted with any of his/her vaults, lowering the amount of users affected
3. Only a handful of users were negatively affected in the first place

### Conclusion
Important to note: **No user deposited funds were at risk at any point in time**, but the YOU rewards were potentially wrong between a maximum time window between the deployment of the V3 Engines and the deplyoment of the Stake manager V3. While for some users the YOU rewards were lower during certain points in time, for most other users they were higher.

Due to the nature of the problem, it's close to **impossible to calculate the real impact per user** in retrospective, at least not without considerable amount of time being spent. While some users received less YOU, others were unknowingly receiving more. A fair aftermath redistribution of YOU tokens cannot be done.

The incident again shows the **importance of testing smart contracts** and it's effects when executed. It also shows that tests should be conducted **with realistic amounts of funds** in order to better identify possible negative effects.

Also, when users report issues with impact on user balances or reward balances, **all other works need to stop** until the root cause is found - independent on what other deliverables are waiting to be shipped.