Postmortem: 26 Jan, 2022 bETH Anchor integration upgrade incident

After the AnchorVault upgrade switching the bETH Anchor integration from using Shuttle to Wormhole bridge, two users have been able to execute transactions aimed for Shuttle-using code, effectively blocking their funds on unaccessible Terra addresses. The integration have been upgraded to prevent such events from happening. Users have been refunded, and the funds corresponding to the amount locked on Terra has been recovered from the AnchorVault.

What happened?

On Jan 26th, 2022 Anchor bETH integration had been migrated 3 from using the Shuttle bridge to the Wormhole bridge. The upgrade lacked smart contract API versioning, allowing two users to send the total of 443.56111857 webETH (Terra-side Wormhole bETH) to inaccessible Terra addresses. The affected users have been refunded on Jan 27th, 2022 from the dev team’s funds. The bETH Anchor integration has been down for about 6 hours from 12:16 PM UTC to 5:52PM UTC. Refunding the dev team by unlocking stETH from the AnchorVault contract has been performed with AnchorVault upgrade on Feb 10th, 2022.

Why did it happen?

The root cause has been that upgraded contracts retained backwards compatibility, allowing users to send txs from the old UI version to the new contracts without reverts.

How did we fix it?

To prevent such kind of incidents from happening, the team has implemented versioning into the AnchorVault, and formulated the policy for upgradable contracts across the Lido codebase. The policy has been published as LIP-10. On a lower level, all state-changing methods used in UI now include version number as the parameter, preventing the txs formed by old UI versions to ever pass into the contracts after the upgrade. Internal guidelines regarding contract upgrades have been tweaked as well.

The fix has been implemented so that no third party integrations are interrupted. On Ethereum, AnchorVault received a tweak to take the refunded stETHs into account in internal calculations. For the Wormhole, bETH tokens locked on the Ethereum Wormhole bridge address corresponded 1-1 to the webETHs on Terra. On the Anchor side, as webETH balances don't affect rewards distribution (only original bETH token balances do), so no impact as well.

Incident recap

Jan 26th, 2022

  1. During 26th Jan upgrade, the UI for converting from stETH/ETH to bETH that interacts with the AnchorVault contract was disabled.
  2. The AnchorVault contract was upgraded to use the new bridge. The upgrade changed the semantics of the AnchorVault.submit method used for conversion: the _terra_address: bytes32 argument started requiring left zero padding of the 20-byte address instead of right padding.
  3. The new UI implementing left zero padding of Terra addresses was deployed and enabled.
  4. Two users were able to send AnchorVault.submit transactions using the outdated UI, either because they had a browser tab with the old UI open or due to the browser caching issues. These transactions (0xc875f85f525d9bc47314eeb8dc13c288f0814cf06865fc70531241e21f5da09d, 0x7abe086dd5619a577f50f87660a03ea0a1934c4022cd432ddf00734771019951) contained Terra addresses encoded using right zero padding.
  5. The Wormhole bridge decoded the right-padded Terra addresses incorrectly (since it expects left padding), minting the wrapper webETH tokens to unreachable addresses, effectively burning the tokens.
  6. The users got in touch with the team reporting the issue with bETH not available on the Terra side.
  7. UI & AnchorVault contracts have been disabled
  8. The fix for AnchorVault & UI breaking the backward-compatibility implemented & applied, resuming the operation of the integration.

Jan 27th, 2022

  1. Users have been refunded from dev team's funds.

Feb 10th, 2022

  1. The upgrade for refunding the dev team from stETH funds locked on the AnchorVault is implemented & performed.
  1. Initial Shuttle -> Wormhole upgrade annoucement.
  2. Integration upgrade for refunding dev team announcement
  3. LIP-10 on Proxy initializations and LidoOracle upgrade
Select a repo