Incident overview: what failed, who was impacted, and why this incident matters

Where we are in the track

Track A / Lesson 1: Coinbase outage concept series — Incident overview: what failed, customer impact, timeline, and why the incident matters.

This is the opening lesson in the course. Track A uses one real Coinbase outage as a concrete anchor, then later lessons will move into broader foundations: networking, Kubernetes, observability, Kafka, Redis, reliability, trading systems, and system design.

Previously

• This starts the short Coinbase outage concept series and the overall infrastructure/platform/system-design course.
• We are beginning with a real incident because production systems are easiest to understand when you can connect abstract concepts to something that actually broke.
• Today is the high-level map. Future lessons will zoom into specific concepts: root cause vs symptoms, fault domains, observability, Kubernetes draining, quorum, Kafka recovery, and disaster recovery.

Beginner foundation first

An incident is a period when a production system is not meeting expectations for users or the business. That could mean the app is fully down, only one feature is failing, data is delayed, or a critical workflow is unsafe to continue.

A few beginner terms:

• Service disruption: the system is partially or fully unavailable, degraded, or incorrect from the customer’s point of view.
• Customer impact: what users actually experienced: failed trades, delayed balances, unavailable pages, bad quotes, slow responses, or blocked access.
• Root cause: the underlying trigger or condition that started the incident. It is not always the same as the first visible symptom.
• Cascading failure: one failure causes other systems to fail because modern systems depend on each other.
• Sev1: a highest-severity incident category, usually meaning broad customer impact, high business risk, or urgent executive/engineering coordination.
• Recovery: restoring the system to a safe, usable state. In financial systems, recovery often includes validation steps before customers can fully use the product again.

For a novice, the key idea is: large platforms are not one big machine. They are networks of many smaller systems — apps, APIs, databases, queues, streams, caches, Kubernetes clusters, trading engines, observability tools, and operational runbooks. Incidents happen when a failure crosses boundaries that were expected to contain it.

Today's mental model

Think of a large platform like a modern airport.

A cooling failure in one terminal sounds local, but it can affect security lines, baggage systems, gates, flight boards, and runway scheduling if those systems are connected. The airport may not be “destroyed,” but normal operations become unsafe or impossible until teams isolate the problem, reroute traffic, validate systems, and reopen in phases.

Coinbase’s outage is similar: a physical infrastructure failure started in a limited place, but the impact spread through systems that trading and balances depend on.

Core concept explained from first principles

A production platform has layers:

1. Physical infrastructure — buildings, racks, servers, networking gear, power, and cooling.
2. Cloud infrastructure — regions, availability zones, managed services, storage, compute, load balancers.
3. Platform layer — Kubernetes clusters, Kafka clusters, deployment systems, observability, automation, security tooling.
4. Product systems — trading, balances, quote services, APIs, customer-facing apps.
5. Operational process — on-call, incident command, runbooks, dashboards, communication, postmortems.

Most product conversations happen at layer 4: “Can customers trade?” or “Are balances updated?” But outages often begin lower in the stack. A physical cooling issue, overloaded database, broken queue, or failed deployment can surface as product-level symptoms.

The Coinbase incident is useful because it shows several important system-design ideas at once:

• A physical failure can affect cloud-hosted systems.
• A “small” failure domain, like racks in a single building, can still produce large customer impact if important dependencies live there.
• Low-latency trading systems may make different architecture choices than ordinary web apps.
• Some workloads can be moved quickly; others are harder because they hold state, require quorum, or manage huge volumes of data.
• Recovery in financial systems is not just “restart everything.” Teams must restore correctness, validate state, and reopen safely.

A good incident overview separates four things:

1. What users saw — trading blocked, exchange access affected, balance streams delayed.
2. What systems failed — quote paths, matching-engine infrastructure, Kafka messaging, balance stream freshness.
3. What triggered it — a thermal/cooling failure affecting racks in a single AWS us-east-1 building.
4. Why recovery took time — isolation did not fully contain the problem; exchange quorum and Kafka partition recovery required careful disaster recovery work.

Real-world example

From the local Coinbase case study:

• Around 23:50 UTC on 2026-05-07, monitoring detected cascading quote failures from internal services.
• Multiple Sev1 incidents were triggered.
• Customer-facing impact included spot trading, Coinbase Prime, International Exchange, Derivatives Exchange, and balance stream delays.
• The reported root cause was a thermal event, meaning a cooling system failure, affecting a subset of racks in a single building in AWS us-east-1.
• Coinbase reported that its exchange infrastructure runs a primary replica in a single zone to reduce latency, with a distributed standby for failure scenarios.
• During this incident, failures expected to be isolated were not fully isolated, extending the outage.
• Automated tooling drained roughly 10 Kubernetes clusters of related workloads out of the affected zone, but the exchange and Kafka could not be drained as easily.
• The matching engine could not safely continue because it could not reach quorum.
• Kafka partitions required recovery onto new brokers involving many TiBs of data; balance streams were delayed, but Coinbase stated there was no data loss.
• Markets were reopened carefully: cancel-only mode, product-state audit, auction mode, then normal trading.

For a beginner, the most important lesson is that several different failures can be true at the same time: physical infrastructure failed, isolation boundaries failed, a matching-engine quorum could not be reached, Kafka recovery was slow, and customer-facing products were degraded.

Case study connection

This case study matters because it is not just “Coinbase was down.” It is a compact example of many platform concepts Nick will see repeatedly:

• Fault domains: racks, buildings, zones, and regions are boundaries where failures may or may not be contained.
• Blast radius: the amount of customer and system impact caused by one failure.
• Stateful vs stateless systems: some services can move quickly; systems with data, ordering, quorum, or specialized hardware are harder.
• Quorum and safety: financial systems may correctly stop serving traffic rather than risk inconsistent trading state.
• Durability vs freshness: “no data lost” can coexist with “balances were delayed.”
• Runbooks and phased recovery: reopening markets safely requires validation, not just restarting services.

We will unpack each of these in later Track A lessons.

Why this matters in Nick's Coinbase role

As a manager and technical leader inside established systems, Nick does not need to personally debug every Kafka partition or Kubernetes node. But he does need to understand enough to ask sharper questions and connect product impact to platform reality.

This lesson helps with:

• Reading incident reports: separating customer symptoms, technical causes, contributing factors, mitigations, and follow-ups.
• Evaluating reliability risk: asking whether a failure is truly isolated or only theoretically isolated.
• Understanding operational tradeoffs: recognizing why low-latency systems may choose single-zone primaries, and what risk that creates.
• Improving runbooks: asking whether recovery steps are rehearsed, automated, observable, and safe under pressure.
• Cross-team coordination: seeing when an incident spans exchange, cloud infrastructure, Kafka, Kubernetes, product, support, communications, and leadership.
• Prioritizing follow-ups: distinguishing “prevent the exact trigger” from “reduce blast radius,” “speed detection,” and “make recovery safer.”

A useful leadership move after an incident is to ask: “Which boundary did we believe would contain this, and what evidence now shows that boundary was weaker than expected?”

Why this matters for prediction markets/trading/social apps

For trading and prediction markets, outages can directly affect money, fairness, order execution, positions, settlement, and customer trust. If a system cannot guarantee correct ordering or balance state, stopping trading may be safer than continuing in a corrupted or unfair mode.

For social apps, the stakes are different but the patterns are similar. A physical or cloud-zone failure might delay notifications, break feeds, stall media processing, or make direct messages inconsistent. Users may not care which Kafka broker failed; they care that messages, posts, notifications, or likes are missing or delayed.

Across both domains, the core question is: which parts of the system can be degraded gracefully, and which parts must stop to protect correctness?

Common failure modes in production

• A low-level infrastructure issue creates high-level product symptoms.
• Teams detect internal technical errors but do not immediately know customer impact.
• A dependency believed to be redundant fails in a correlated way.
• A “single zone” or “single cluster” assumption creates a larger blast radius than expected.
• Stateless services recover quickly while stateful systems remain stuck.
• Messaging systems preserve data but fall behind, creating stale balances or delayed updates.
• Recovery automation works for common cases but fails for rare combinations of constraints.
• Teams reopen service too quickly without validating state, causing secondary incidents.
• Incident reports over-focus on root cause and under-explain contributing factors and missed isolation.

Technical-leader checklist

Questions Nick can ask in design reviews, planning, incident reviews, and operational readiness reviews:

1. Customer impact: If this component fails, what exactly does the customer see?
2. Critical path: Is this system on the hot path for trading, balances, login, quote generation, or order execution?
3. Fault domain: What rack, zone, region, cluster, or dependency boundary is supposed to contain this failure?
4. Blast radius: How many products, customers, or workflows are affected if that boundary fails?
5. Statefulness: Can this workload be moved automatically, or does it depend on local state, ordering, quorum, or specialized hardware?
6. Detection: Which monitor tells us about customer impact, not just host health?
7. Mitigation: What is the first safe action: drain, fail over, pause, enter cancel-only mode, shed load, or disable a feature?
8. Recovery validation: How do we prove the system is safe before reopening?
9. Runbook quality: Has the runbook been rehearsed under realistic constraints?
10. Follow-up framing: Are we fixing the trigger, reducing blast radius, improving detection, improving recovery, or all four?

Mini design exercise

Imagine a prediction market app has three core systems:

1. A web/API layer for placing orders.
2. A matching engine that orders trades fairly.
3. A balance stream that updates users after trades settle.

A cloud-zone issue causes the balance stream to lag by 20 minutes, but the matching engine is still healthy.

Think through:

• Should users still be allowed to place new orders?
• What warning, if any, should the product show?
• Which dashboards would you want before deciding?
• What condition would make you switch to cancel-only mode?
• What validation would you require before declaring full recovery?

There is no single perfect answer. The goal is to practice separating availability, correctness, freshness, and user trust.

Terms to remember

• Incident: a production event where the system fails to meet user or business expectations.
• Sev1: a top-severity incident requiring urgent coordination.
• Customer impact: what users actually experienced.
• Root cause: the underlying trigger or failure condition.
• Contributing factor: something that made the incident worse or longer.
• Cascading failure: one failure causing other failures through dependencies.
• Fault domain: a boundary where failures are expected to be contained.
• Blast radius: the size of the impact from a failure.
• Quorum: the minimum number of nodes that must agree before a distributed system can safely proceed.
• Kafka partition: a shard of an event log stored and replicated across brokers.
• Durability: data is not lost.
• Freshness: data is up to date.
• Runbook: a documented operational procedure for responding to known scenarios.

Source links

• Local case study: case-studies/2026-05-07-coinbase-multi-hour-service-disruption.md
• Rob Witoff / Coinbase engineering initial read on X: https://x.com/rwitoff/status/2052863502424133949
• Local fallback metadata/API source: https://api.fxtwitter.com/status/2052863502424133949
• Google SRE Book — Managing Incidents: https://sre.google/sre-book/managing-incidents/
• Google SRE Book — Addressing Cascading Failures: https://sre.google/sre-book/addressing-cascading-failures/

Next time

Next time: Track A / Lesson 2 — Reading incident reports: symptoms, root cause, contributing factors, mitigation, and follow-ups.

Today gave you the map of the outage. The next lesson will teach how to read an incident report without getting lost: what is a symptom, what is a root cause, what is a contributing factor, what counts as mitigation, and how to evaluate whether follow-up actions actually reduce future risk.