# Incident summary - Mainnet 30/05/2021 and 31/05/2021

This post serves as a summary of the IOTA Mainnet incident that occurred on May 30, 2021 and May 31, 2021.

## Incident summary

The incident occured at 01:09:59 UTC on May 30, 2021 and was resolved by 06:24:23 UTC on May 30, 2021.

The same issue occured again at 23:41:04 UTC on May 30, 2021 and was resolved by 05:45:38 UTC on May 31, 2021.

During these two periods, no more milesones were issued on the network.

## Leadup

The Coordinator plug-in attempted to establish network communication between the dispatcher and signer components of the Coordinator set-up. This led to the next milestone not getting signed. The operation timed out and this led to the Coordinator gracefully shutting down.

## Impact

Between 01:09:59 UTC on May 30, 2021 and 06:24:23 UTC on May 30, 2021 and between 23:41:04 UTC on May 30, 2021 and 05:45:38 UTC on May 31, 2021 users were unable to confirm transactions on the IOTA Mainnet network.

## Timeline

### Event 1
**30/05/2021**

1. 01:09:50 UTC - Milestone issued (271662)
2. 01:09:59 UTC - The coordinator failed to sign remotely (received RST_STREAM from the dispatcher service)
3. 01:09:59 UTC - The coordinator shut down gracefully
4. 01:11:00 UTC - The devops team has been alerted to begin investigation into the issue due to milestones not being issued anymore
5. 06:24:23 UTC - Milestones started being issued again
6. 08:30:00 UTC - Rest of the services, such as the public API services have fully resumed operation.

### Event 2

**30/05/2021**

1. 23:39:54 UTC - Milestone issued (277875)
2. 23:40:04 UTC - Next milestone was about to be issued, but the coordinator starts waiting on signer
3. 23:41:04 UTC - After TCP Timout (60 secs) the coordinator shut down gracefully
4. 23:43:20 UTC - The devops team has been alerted to begin investigation into the issue due to milestones not being issued anymore

**31/05/2021**

5. 05:45:38 UTC - Milestones started being issued again
6. 07:50:20 UTC - Rest of the services, such as the public API services have fully resumed operation.

## Root cause

Logs proven that there was a network connection timeout between Signer dispatcher component and Signer component and their host instances.

The coordinator is set up to perform this operation in cases where investigation needs to happen as to the cause of the issue before the operation is fully restored.

## Lessons learned

* The Cooridnator components need to be more tolerant to timeouts of remote milestone signing set-ups with more control over the timeout and retry mechanisms.
* The Coordinator component needs to act automatically in this particular scenario to resolve the situation on its own.

## Recurrence

Actions have been taken to address lessons learned. A fix has been created that allows the remote signer set-up to be more tolerant to timeouts and perform retries. We created an additional coo monitoring sidecard component to resolve this situation automatically.