# Postmortem: Testnet outage on 16.03.20

**16 March 12:18 UTC**
Period submission on height 227424 has failed by validator 2.
Reason is unclear — tx was dropped out of the mempool. We have a `checkBridge` mechanism to recover from such an error, it was expected to fire on a next period submission. Blocks were still produced at this point, only the period sumission was missing

**16 March 13:18 UTC**
New period proposal was created on height 227456. `checkBridge` successfully figured out a missing period submission and assigned validator 1 to resubmit it. Period was submitted with tx hash `0x13ebe8cf83c6657b0ee3087d3a6bd19948f6186d193e33156bba04250c131b42`.

## Problem
`checkBridge` failed to detect [period submission](https://rinkeby.etherscan.io/tx/0x13ebe8cf83c6657b0ee3087d3a6bd19948f6186d193e33156bba04250c131b42) was mined just right and kept waiting on it stopping block production.

## Resolution

Both validators were restarted. On restart both automatically caught up with submitted period and chain resumed normal operation.

## Impact

- Chain was not producing blocks for ~17 hours
- No data loss, next period was submitted
- Next period was submitted with a wrong previous period hash. It is referencing the same period as period at height 227424 thus making a "period chain split".

    ... - 227391 — 227424
               \
                 227456 — 227488 — ...