pnkfelix original postmortem writeup 2021-07-15

Postmortem Analyses

Rust itself may be safe

(at least in its idealized form)

… but

process failures happen

(we are all only human, after all!)

Postmortem

A Postmortem (or "Correction of Error") is a structured process to

1. document a failure as an individual event,
2. drive collective discussion of the event, and
3. identify changes that should be made.

Goals

Why do this?

Primary: prevent future occurrences of similar events.

Secondary: reduce fallout from future failure events.

Finger-pointing = Antipattern!

Assigning blame to people is not a goal.

Goal is to identify processes that can be improved, not people to publicly shame.

• shaming motivates people to hide facts that we would need to understand failure events,
• shaming also motivates risk-aversion
  • (bad for innovation, and forward progress in general!)

Software development is a collective, social activity

In open source, we all have differing degrees of ownership.

So: Don't play the blame game. For example, there's no need to include names in the document.

Goals

Assumption: There is always room for improvement.

(A portmortem may conclude that no currently cost-effective change is known. But: 1. this is expected to be rare outcome, and 2. the exercise of determining that conclusion is still worthwhile.)

Desired Outputs

Primary: Concrete improvements to development and deployment processes (or plans to make such improvements)

Secondary: Documentation of failure event itself

Even though the document itself is secondary, writing and discussing the document is amazing tool to drive team towards the primary output.

The postmortem document
a document, suitable for general audience, that:

• describes event in neutral fact-based language
• identify root cause(s)
  • remember: blame is not a goal.
• suggest long-term corrective actions
• establish short-term action items

Note: Postmortem document has three potential audiences

• The team who managed the event, since they will use it to guide follow-up preventive actions
• Other teams, who may want to learn how they can adjust their own processes to prevent such failures
• Our users, who we all know have made Rust "most-beloved". Advertising public postmortem documents is way to earn their trust.

Focus is on:

• what happened,
• why it happened, and
• how can we prevent it from happening again.

Prevention vs Response

As Rust developers, we often focus on 100% prevention of bugs ahead of time, e.g. via static analysis or tricky type system encodings.

The Postmortem can end up discussing ways to achieve 100% prevention.

But: first focus of the document is on how the teams responded to the incident.

• Remember: process failures are fact of life
• Ideal world: prevents them entirely
• Real world: failures sometimes happen; need guardrails to prevent them from becoming disasters.
  • That is why post-event responses are always discussed
    • and (?) often given priority over pre-event leadup

Document Template

• Summary (<= 3 paragraphs)
• Metrics/Graphs (>=1 graphs/tables illustrating impact of event)
• User Impact: (1-2 paragraph summary of user-facing impact/experience during the event)
• Incident Response (Detection/Mitigation/Diagnosis/Resolution), driven by four questions:
  • Q: How was the event detected (e.g. an alarm? manual?)
  • Q: How could time to detection be improved? As a thought exercise, how would you have cut the time in half?
  • Q: How did you reach the point where you knew how to mitigate the impact?
  • Q: How could time to mitigation be improved? As a thought exercise, how would you have cut the time in half?
• Timeline: Explain how incident was managed. Include event start and end times, not just team's perception of event.
• Five Whys (wikipedia): dig down until root cause is identified
• Lessons Learned, Corrective Actions, Action Items: The Five Whys yield lessons, and lessons yield long- and short-term actions.

Example

Rust 1.52.0 fingerprint bug

Event started on Thursday May 6th, and mitigation was deployed on Monday May 10th.

Compiler team proposed applying the Postmortem process on June 2nd. We have since had three one-hour steering meetings about the event (on June 25th, July 2nd, and July 9th), all driven by one (evolving) Postmortem document.

Postmortem document itself (aka "Correction of Error", or COE): https://hackmd.io/DhKzaRUgTVGSmhW8Mj0c8A

Template revisited

• Summary, Metrics/Graphs, and User Impact: establish broad context for the reader to understand exactly what happened.

Template revisited

• Incident Response (Detection/Mitigation/Diagnosis/Resolution) and Timeline: drives brainstorming on what to fix

  • e.g.: Was the response time for the 1.52.0 fingerprint issue "good enough"? Subjective! But: Good exercise to assume that there is always room for improvement

• New section, "Leadup": Our release process naturally lends itself to discussion of what happened before the release of a buggy tool, since that is also relevant to prevention.

  • However, got its own section, to ensure incident response gets appropriate attention.

Template revisited

• Five Whys: iterative process to identify root causes, by branching through different causal paths.

  • One reason why compiler-team had three separate meetings is that the discussion about the first draft led to exploration of new branches

• Lessons Learned, Corrective Actions, Action Items: Again, this guides collective discussion.

  • "Five Why's" should yield root cause(s), and those should be reflected in the lessons learned.
  • Each lesson yields at least one long-term corrective activity
  • Each corrective activity yields at least one short-term action item.

• Action Items are meant to have deadlines attached – otherwise, they're not short term!

  • e.g. something within 90 days, ideally much less.
  • Why: To stress that there is risk of similar event re-occurring. (otherwise we wouldn't be talking about it).
  • Deadlines are a tricky matter given open source development process. Might need to rely on Rust-as-day-job people to take ownership of such tasks with deadlines.

Example Why Sequence

from fingerprint 1.52.0 postmortem

(see doc for elided answers; point is, each answer directly yields followup "Why" Q's)

• Why didn't the beta channel act as a canary about 1.52.0 causing ICE's for end developers?
  • Why didn't the beta channel act as a canary about 1.52.0 causing ICE's for CI services?
  • Why are end developers favoring nightly or stable over beta channel?
    • Q: Why don't you entice people to use Beta by providing some sort of carrot (i.e. reward) in return? For example, beta could provide access to certain features that are not available in Stable?

Example Lesson Learned, Corrective Action and Action Item

Lesson Learned (5/5): We need to be willing and able to communicate more quickly to our users in response to an event like this.

Corrective Action (5a): Define a playbook for how to communicate effectively to our users after a user-impacting event has been discovered.

Corrective Action (5b): Add ability to issue Public Service Announcements (PSAs) from rustup.

Action Item: (5a-i, due 2021-08-05): pnkfelix does a survey of the landscape for response playbooks during user-impacting events, especially when it comes to programming languages.

No concrete action item yet for (5b).

Feedback and Thoughts

Participants from compiler seemed to like the structured process.

However: writing this document was not trivial.

pnkfelix vastly underestimated the effort involved, even after being repeatedly warned that such documents can be multiple-week efforts.

Conclusion

Failures happen… but we don't have to panic! in response

Don't play the blame game

Investing effort in a well-written, acted upon, and widely-shared postmortem can drive positive organizational change and prevent repeat failure events.

References

https://medium.com/the-cloud-architect/incident-postmortem-template-7b0e0a04f7a8

https://medium.com/@josh_70523/postmortem-correction-of-error-coe-template-db69481da31d

https://www.youtube.com/watch?v=yQiRli2ZPxU

https://sre.google/workbook/postmortem-culture/