Postmortem Analysis

Background

• Rust 1.52.0 added extra incr-comp validation

• Result: Exposed preexisting issues in rustc; ICEs for users.

• Release: Fri May 6th; mitigation deployed on Mon May 10th (disabling incr-comp).

• Since then, team resolving uncovered issues and on track to re-enable incremental in 1.54.

Moving forward

• Because was a high-profile issue affecting many users, we've conducted a postmortem/retrospective/correction of error.

• Felix has written a postmortem document and we've used three of our weekly steering meetings to hold postmortem discussions.

• We want to understand what and why:

  • Explain issue: error itself and user impact
  • Explain why it happened

• So that we can fix our processes

  • Prevent similar issues from occurring
  • Install guardrails to reduce severity of future incidents

• 9 section postmortem doc drives above efforts

Preventing repeat occurrence

Focus is on process, not people, so we fix processes.

Anti-pattern: finger-pointing

Shame is the path to the dark side.

Shame ⟹ Fear ⟹ Hiding Facts ⟹ Misunderstanding ⟹ Repeat Failures

(Also: Fear ⟹ Risk Aversion ⟹ Stagnation.)

image courtesy of u/noteblockiller

Template: Understand Issue

• Summary (<= 3 paragraphs)
• Metrics/Graphs (>=1 graphs/tables illustrating impact of event)
• User Impact: (1-2 paragraph summary of user-facing impact/experience during the event)

Template: Understand Response

• Incident Response; four questions:
  • How was the event detected?
  • How could time to detection be halved?
  • How did you reach the point where you knew how to mitigate the impact?
  • How could time to mitigation be halved?
• Timeline: Explain how incident was managed. Include event start and end times, not just team's perception of event.

Template: Why event happened

• Five Whys (wikipedia): dig down until root cause is identified.

  • Each answer is meant to spark one or more why questions, until you find the root cause.

  • Search may end up digging into events before the incident began.

Outcomes for our fingerprint postmortem

• "5 Whys" turned into 21 Whys + answers
• 6 Lessons Learned
• 9 Corrective Actions
• 11 Action Items
  • These are fairly large items so please get in touch if you want to help!

Conclusions

• Overall, we're very happy with the results of the process!

• We've had good discussions about the principles we use to make difficult decisions.

• We have some new strategies to avoid similar issues in the future.

References

• Document Template: https://hackmd.io/esAoSljqQtyJnDi4ecCg2g

• pnkfelix postmortem slide deck (longer): https://hackmd.io/SbubQQIlRGSlrOpKQf73Tg

• 1.52 incremental fingerprint postmortem: https://hackmd.io/DhKzaRUgTVGSmhW8Mj0c8A