Reading incident reports: symptoms, root cause, contributing factors, mitigation, and follow-ups

Where we are in the track

Track A / Lesson 2: Coinbase outage concept series — Reading incident reports: symptoms, root cause, contributing factors, mitigation, and follow-ups.

Track A uses the May 2026 Coinbase multi-hour outage as a concrete anchor for several infrastructure and system-design ideas. Today is about how to read an incident report clearly before jumping to conclusions.

Previously

• Lesson 1 gave the high-level map of the Coinbase outage: customer impact, thermal/cooling failure, failed isolation, Kubernetes draining, matching-engine quorum, Kafka recovery, delayed balance streams, and phased market reopening.
• We learned that production incidents often cross layers: physical infrastructure, cloud, platform, product systems, and operational process.
• Today zooms into a practical skill: separating what users saw, what systems reported, what triggered the incident, what made it worse, how teams mitigated it, and what follow-ups should reduce future risk.

Beginner foundation first

An incident report is a written explanation of a production problem. Its job is not just to say “what broke.” A good report helps teams learn, improve the system, and coordinate follow-up work.

For a novice, five terms are especially important:

• Symptom: the visible sign that something is wrong. Example: users cannot place trades, quotes fail, balances are stale, API latency spikes, or error rates rise.
• Root cause: the underlying trigger or condition that started the incident. Example: a cooling failure affecting racks in a cloud datacenter building.
• Contributing factor: something that did not necessarily start the incident but made it worse, broader, harder to detect, or slower to recover from.
• Mitigation: an action taken during the incident to reduce customer impact or restore safe operation. Example: draining workloads, failing over partitions, entering cancel-only mode, or disabling a risky path.
• Follow-up: work after the incident meant to reduce future risk. Good follow-ups are specific, owned, prioritized, and connected to a failure mode.

The beginner trap is to read an incident report as a single-cause story: “X happened, so the site went down.” Real incidents are usually chains: one thing fails, dependencies react badly, assumptions turn out to be weaker than expected, operators mitigate under time pressure, and recovery requires validation.

Today's mental model

Think of an incident report like an airplane accident investigation.

The visible symptom might be “the flight was delayed.” The immediate cause might be “the plane could not depart.” But the useful investigation asks deeper questions: Was there a mechanical issue? A weather issue? A crew scheduling issue? A maintenance process gap? A communication problem? A backup plan that existed on paper but failed in practice?

In production systems, the same pattern applies. The goal is not blame. The goal is to understand the chain well enough to make the next incident smaller, shorter, or less likely.

Core concept explained from first principles

A production system is a set of components connected by dependencies. When something fails, different people see different pieces of the same event:

1. Customers see product impact: “I cannot trade,” “my balance is delayed,” “the page errors,” or “the app is slow.”
2. Monitoring sees signals: error rates, failed quotes, latency, consumer lag, unhealthy hosts, unavailable pods, failed health checks.
3. Engineers see component behavior: a Kafka partition is stuck, quorum cannot be reached, a zone is impaired, a deployment is failing, or a database replica is behind.
4. Incident command sees coordination problems: who owns the mitigation, what status to communicate, which runbook to follow, and when it is safe to reopen.

A useful incident report connects those views without mixing them up.

Symptoms vs root cause

A symptom is evidence of pain. A root cause is the event or condition that set the failure chain in motion.

If a customer says, “My balance is stale,” that is a symptom. It does not tell you whether the problem is in the web app, an API, a database, Kafka, a consumer, a cache, or an upstream trading system.

If monitoring says, “quote failures are cascading,” that is also a symptom. It tells you internal services are failing to produce quotes, but it may not yet explain why.

In the Coinbase case study, the reported root cause was a thermal event, meaning a cooling system failure, affecting a subset of racks in a single AWS us-east-1 building. But that root cause alone does not fully explain the customer impact. You still need to understand the path from physical failure to exchange disruption, Kafka recovery, balance delays, and market reopening.

Root cause vs contributing factors

A contributing factor is a condition that made the incident worse or longer.

Example: suppose a cooling failure affects one datacenter building. That may be the root trigger. But the outage becomes more severe if:

• critical systems share that fault domain,
• isolation boundaries do not hold,
• failover is manual or slow,
• the system has large state that cannot move quickly,
• runbooks are incomplete,
• dashboards show component health but not customer impact,
• teams are unclear about ownership during recovery.

Good incident reviews ask: “Even if the trigger happens again, what could make the impact smaller?” That question usually points to contributing factors and follow-ups.

Mitigation vs permanent fix

A mitigation is what you do to stop the bleeding. It may not be the final fix.

During an incident, teams often choose safe short-term actions:

• route traffic away from a bad zone,
• drain Kubernetes workloads,
• fail over to standby infrastructure,
• disable a feature,
• shed load,
• pause new writes,
• enter cancel-only mode,
• replay or catch up delayed streams,
• validate state before reopening.

A permanent fix or risk-reduction follow-up comes later: improve isolation, automate failover, add monitors, rehearse disaster recovery, reduce dependency coupling, change architecture, or clarify ownership.

Follow-ups should map to failure modes

Weak follow-up: “Improve reliability.”

Stronger follow-up: “Add a customer-impact monitor for balance-stream freshness, page the owning team when lag exceeds X minutes, and add a runbook step for verifying no data loss before reopening affected products.”

The stronger follow-up is better because it has a clear failure mode, owner, detection signal, and operational action.

Real-world example

From the local Coinbase outage case study, you can practice classifying facts:

• Customer symptoms: spot trading affected, Prime affected, International Exchange and Derivatives Exchange affected, balance stream delays.
• Internal symptoms/signals: cascading quote failures, multiple Sev1 incidents, Kafka partitions stuck or needing failover, matching-engine cluster unable to reach quorum.
• Reported root cause: a thermal/cooling failure affecting a subset of racks in a single AWS us-east-1 building.
• Contributing factors: isolation that was expected to contain the primary-zone failure did not fully contain it; the exchange depended on dedicated hardware/storage; Kafka recovery involved many TiBs of data; some systems could not be automatically drained.
• Mitigations/recovery actions: drain about 10 Kubernetes clusters of related workloads, execute disaster recovery procedures, recover Kafka partitions onto new brokers, restore matching-engine quorum safely, validate system health, reopen markets in phases.
• Safe reopening steps: cancel-only mode, product-state audit, auction mode, then normal trading.

Notice how much more useful this is than saying, “AWS had a cooling issue and Coinbase went down.” The deeper version shows where architecture, operations, automation, and correctness all mattered.

Case study connection

This Track A lesson is directly anchored in the Coinbase outage. It gives Nick a template for reading the rest of the series:

• Tomorrow’s topic, latency vs availability and fault domains, will explain why “single-zone primary” can be a rational low-latency design choice while still creating recovery tradeoffs.
• Later Track A lessons will apply the same incident-report reading pattern to observability, Kubernetes draining, matching-engine quorum, Kafka lag, and disaster recovery.
• The important leadership habit is to avoid stopping at the most dramatic detail. A thermal event matters, but the learning comes from the chain: detection, containment, dependencies, mitigation, validation, and follow-up.

Why this matters in Nick's Coinbase role

As a manager and technical leader inside established Coinbase systems, Nick will often consume incident reports written by specialized teams. He does not need to become the deepest Kafka, Kubernetes, networking, or matching-engine expert to add value. But he does need to read reports precisely.

This skill helps Nick:

• Avoid false certainty: “root cause identified” does not mean all risk is understood.
• Ask better incident-review questions: Were the symptoms detected by customer-impact monitors or only by internal component alerts?
• Separate fix types: Are follow-ups preventing the trigger, reducing blast radius, improving detection, speeding mitigation, or making recovery safer?
• Spot ownership gaps: Which team owns the dependency, the runbook, the alert, the customer communication, and the product decision to reopen?
• Evaluate operational readiness: Are mitigations rehearsed, measurable, and safe under pressure?
• Connect product and platform impact: A delayed balance stream is both a platform event and a user-trust/product event.

A sharp question Nick can ask in an incident review:

“If the same trigger happened again tomorrow, which part of the customer impact would be smaller because of these follow-ups?”

If the answer is vague, the follow-ups may not yet be strong enough.

Why this matters for prediction markets/trading/social apps

For trading and prediction markets, incident reports are especially important because correctness, fairness, and user trust matter as much as uptime.

A trading system might be technically “available” but unsafe if:

• order sequencing is uncertain,
• balances are stale,
• the matching engine cannot guarantee a single leader,
• market data is delayed,
• users can act on inconsistent information,
• recovery happens without auditing product state.

For social apps, the same incident-report structure applies even when money is not directly at stake. A notification outage, feed delay, media-processing backlog, or abuse-review queue failure should still be broken down into symptoms, root cause, contributing factors, mitigation, and follow-ups.

In both domains, the report should answer: what did users experience, what system signals detected it, why did it happen, why was impact not contained, what did we do, how did we verify recovery, and what will be different next time?

Common failure modes in production

• Teams confuse the first alert with the root cause.
• Reports focus on the technical trigger but under-explain customer impact.
• Reports identify root cause but skip contributing factors, so systemic risk remains.
• Follow-ups are too generic: “add monitoring,” “improve runbook,” or “make failover better.”
• Follow-ups are not tied to owners, deadlines, metrics, or specific failure modes.
• Mitigations restore service but leave correctness unverified.
• Dashboards show infrastructure health but not product-level user impact.
• Incident communication says “resolved” before delayed streams, stale caches, or downstream systems have caught up.
• The organization over-learns one lesson from an incident and ignores other contributing factors.
• Teams blame humans for slow recovery instead of improving tools, runbooks, rehearsals, and system design.

Technical-leader checklist

Questions Nick can ask in design reviews, planning, incident reviews, or operational readiness reviews:

1. Customer impact: What exactly did customers experience, and how many were affected?
2. Detection: Which alert fired first? Was it a symptom alert, a component alert, or a customer-impact alert?
3. Root trigger: What event started the failure chain?
4. Contributing factors: What made the incident broader, longer, harder to diagnose, or harder to recover from?
5. Fault isolation: Which boundary was expected to contain the failure, and why did it not fully work?
6. Mitigation: What action reduced customer impact during the incident?
7. Correctness: Did recovery require data validation, replay, reconciliation, quorum restoration, or product-state audit?
8. Follow-up quality: Does each follow-up map to a specific failure mode?
9. Ownership: Who owns the alert, runbook, automation, dependency, and customer-facing decision?
10. Proof of improvement: How will we know the system is safer after the follow-ups ship?

Mini design exercise

Imagine a prediction market app has an incident where users report stale balances for 25 minutes after trades settle.

Classify each statement:

1. “Users saw balances that did not reflect recent trades.”
2. “Kafka consumer lag rose above 20 minutes.”
3. “A broker disk failure caused several partitions to move leaders.”
4. “The balance service had no customer-impact alert for freshness.”
5. “The team paused withdrawals until reconciliation completed.”
6. “Add a balance-freshness SLO and page when lag exceeds 5 minutes.”

Try labeling each as symptom, internal signal, root cause, contributing factor, mitigation, or follow-up.

One possible answer:

1. Customer symptom.
2. Internal signal.
3. Possible root cause or trigger, depending on the full chain.
4. Contributing factor.
5. Mitigation.
6. Follow-up.

Terms to remember

• Incident report: a written account of what happened, why it happened, how it was mitigated, and what will improve.
• Symptom: visible evidence that something is wrong.
• Root cause: the underlying trigger or condition that started the incident chain.
• Contributing factor: something that made impact worse, broader, longer, or harder to resolve.
• Mitigation: an action taken during the incident to reduce impact or restore safe operation.
• Follow-up: post-incident work intended to reduce future risk.
• Customer-impact signal: a metric or alert tied to what users experience, not just internal component health.
• Operational readiness: evidence that a system has alerts, runbooks, owners, capacity, and recovery procedures before it fails.
• RCA: root cause analysis; a structured investigation into what happened and what should change.
• Postmortem: a learning document after an incident, ideally blameless and action-oriented.

Source links

• Local case study: case-studies/2026-05-07-coinbase-multi-hour-service-disruption.md
• Rob Witoff / Coinbase engineering initial read on X: https://x.com/rwitoff/status/2052863502424133949
• Local fallback metadata/API source: https://api.fxtwitter.com/status/2052863502424133949
• Google SRE Book — Managing Incidents: https://sre.google/sre-book/managing-incidents/
• Google SRE Book — Postmortem Culture: https://sre.google/sre-book/postmortem-culture/

Next time

Next time: Track A / Lesson 3 — Latency vs availability and fault domains: why single-zone primaries exist and how blast radius works.

Today taught how to read the incident report. Next time we will unpack one of its most important architecture tradeoffs: why a low-latency exchange might keep a primary in a single zone, what that means for availability, and how fault domains are supposed to limit blast radius.