hackertube.io

Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017

GOTO Conferences · talk · Watch on YouTube

Cloud, DevOps & reliabilityTesting, debugging & developer tools

What it is

Bryan Cantrill's GOTO Chicago 2017 talk on debugging production systems under pressure, centered on the incident where an operator's mistyped command rebooted an entire Joyent datacenter. It covers human factors, automation and architecture.

Why Hacker News recommends it

A perennial HN recommendation for outage and postmortem threads, praised as funny, educational and one of the best postmortems anyone has seen.

  • 8 people recommended it
  • 11 different months
  • 3 strong recommendations
  • first 2018, last 2025

What HN commenters said

  1. “Bryan Cantrill, who gives my favorite talks, has an interesting and funny talk on one of the Joyent outages”

    Interesting, funny talk about a Joyent outage

    OmarAssadi, 2022-01 · Strong recommendation · in “Famous outages along with deep postmortems?”

  2. “Time to share one of my favorite talks (and speakers) ever”

    A favorite talk by a favorite speaker on handling outages

    int0x2e, 2023-01 · Strong recommendation · in “WAN router IP address change blamed for global Microsoft 365 outage”

  3. “The best postmortem I've seen was actually a conference talk”

    Detailed datacenter outage postmortem covering human factors, automation and architecture

    KronisLV, 2023-03 · Strong recommendation · in “Ask HN: What is the best postmortem you've seen?”

  4. “Bryan Cantrill has a great talk[0] about dealing with fires”

    Dev versus ops mindsets when handling live fires

    merlincorey, 2019-07 · Recommendation · in “Root cause analysis: significantly elevated error rates on 2019‑07‑10”

  5. “I highly recommend Bryan Cantrill's talk[1] about the time an operator accidentally rebooted an entire datacenter”

    Operator mistyped command rebooted an entire datacenter

    pdkl95, 2020-09 · Recommendation · in “They're deleting my channel, but they don't know why? [video]”

  6. “worth watching IMNHO”

    Cantrill talk on debugging production systems

    e12e, 2021-08 · Recommendation · in “Harbormaster: Anti-Kubernetes for your personal server”

  7. “Bryan Cantrill gives a great presentation about managing an outage”

    Managing a long outage, including sleep management for the team

    wingmanjd, 2022-03 · Recommendation · in “Hackers claim to have breached Okta systems”

  8. “Bryan Cantrill draws some excellent parallels between this incident and software systems”

    Parallels between Air Canada 143 and semi-automated software systems

    cls59, 2023-07 · Recommendation · in “40 years ago yesterday Air Canada Flight 143 ran out of fuel mid-flight”

Excerpts from Hacker News comments, linked to the originals. Summaries and labels were written by a language model from those comments.

Topics: bryan cantrill, outages, postmortem, debugging, joyent, devops

Also recommended by the same people