# Osmosis outage 2021-08-11 - 2021-08-12

## Rules of engagement
* There is no _one simple cause_ to these kind of issues. It's not one person, one technology or one act. (I'll stop with the self-deprecating humour for this, too.)
* This also means, there's no point in pointing fingers. Please, just don't.
* Conclusions come at the _end_. Don't jump to them unless more aspects were discussed/investigated.

## Description
Discord users reported stuck packets going from the Osmosis network to Cosmos Hub at around 6PM UTC.

The IBC Integrators Telegram group worked on resolving the issue which was eventually resolved by 4AM UTC, in a total of 10 hours.

(Note from Greg: Someone please fill in details between 6PM-1AM and remove this comment. I was late to the party.)
1AM UTC: around 400 stuck packets are (still) present from Osmosis to Cosmos Hub 4. Akash network packet issues seem to have resolved.
1:45AM UTC: The need for non-pruning nodes was identified: some stuck packets were old enough to fall out from regular node databases. (Note from Greg: this might have been earlier, feel free to update if that's the case.)
2:30AM UTC: Non-pruning nodes provided, relaying stuck packets started.
3AM UTC: the stuck packets were all pushed/timed out successfully.
4AM UTC: additional testing finished, number of packets are converging to zero.

## Observations
* If a packet is unrelayed long enough, a non-pruning node (or node with longer pruning settings) is required to be able to relay it or time it out. (This was not explicitly stated before.)
* Multiple old packets got stuck effectively blocking the Hermes relayer from pushing through packets. (Hermes aggregates multiple packets into a transaction, which then fails to transfer.)
* Throughout the afternoon, Cephalopod's Hermes relayer experienced transactions that didn't end up in a block for any amount of normal time. They got stuck in mempool and that whacked the relayer sequence number out of sync. This made all Hermes IBC transactions from Cephalopod on the Cosmos Hub invalid and not transacted. (Exact time when this started is pending as well as more monitoring around this.)
* Doubling the gas price in the Cephalopod Hermes relayer resolved any issues with packets stuck in the mempool.
(Greg's note: everyone, add any other useful observations.)

## Praise
Big thanks to these people to taking extra time and effort to help resolve the issue.

(Note from Greg: I was only involved after 1AM UTC, please add people who helped before and remove this comment.)

* Matt Harrop from Figment took time out of his evening to provide non-pruning nodes for the resolution.
* Chalabi from Chandra Station provided a non-pruning node for the Osmosis network.
* Andy Nogueira from Informal Systems woke up to start syncing his own non-pruning nodes.
* Jacob Gadikian from Notional and Paul Lovette from SkyNet Validator jumped in to sync a non-pruning node from backup.
* Dev and Sunny helped coordinate among people to get the necessary resources together.

## Conclusions
Greg's opinion: I'll refrain from making conclusions until we have more information. There are multiple relayers between Osmosis and Cosmos Hub, I'm interested to see how others experienced the event. I'd like to see incident management and problem management improved among the teams but we had some great people adding value throughout the incident.