← HomeCase Studies
case-studies/2026-05-07-coinbase-multi-hour-service-disruption.md

Case Study: Coinbase multi-hour service disruption — 2026-05-07

Source: Rob Witoff / Coinbase engineering initial read on X

What happened, in plain English

Coinbase experienced a multi-hour service disruption affecting trading, exchange access, and balance updates. Monitoring detected cascading quote failures from internal services and multiple Sev1 incidents were triggered.

Customer-facing impact included:

The reported root cause was a thermal event — a cooling system failure — affecting a subset of racks inside a single AWS us-east-1 building.

Coinbase said their exchange infrastructure runs a primary replica in a single zone, consistent with industry standards to reduce latency. They also maintain a distributed standby. During this incident, failures in the primary zone that were designed to be isolated were not isolated, extending outage duration.

Main technical themes to incorporate into lessons

This case study should become a recurring concrete example across multiple learning tracks.

1. Single-zone primary systems and latency tradeoffs

Coinbase stated that the primary replica of exchange infrastructure runs in a single zone to reduce latency.

Teaching angle:

Use in lessons:

2. Failure isolation did not hold

The post says failures in the primary zone were designed to be isolated but were not, extending the outage.

Teaching angle:

Use in lessons:

3. Kubernetes was drainable; exchange and Kafka were not

Coinbase said automated tooling drained about 10 Kubernetes clusters worth of related workloads out of the affected zone. Most services were back to normal within about 30 minutes of diagnosis.

The two things they could not automatically drain were:

Teaching angle:

Use in lessons:

4. Matching engine quorum failure blocked trading

The exchange matching engine processes orders and maintains order books. It is a distributed cluster that requires quorum to safely elect a leader and continue processing trading.

During the incident, only a subset of nodes remained healthy, so the cluster could not reach quorum. Trading across Retail, Advanced, and Institutional exchanges was blocked.

Teaching angle:

Use in lessons:

5. Kafka partition failover at multi-terabyte scale

Coinbase said distributed Kafka clusters managing messaging across systems failed to remain available and required partition failovers to new hardware brokers with many TiBs of data.

Kafka recovery required disaster recovery procedures to move stuck partitions onto new brokers. Balance streams were delayed until replication caught up. Coinbase stated no data was lost.

Teaching angle:

Use in lessons:

6. Reopening markets carefully

After recovery, Coinbase reopened markets in phases:

  1. Cancel-only mode
  2. Audit product states
  3. Auction mode
  4. Restore normal trading on Coinbase Exchange

Teaching angle:

Use in lessons:

7. Observability and incident response

Monitoring detected cascading quote failures. Multiple Sev1s were opened. Engineering executed disaster recovery plans, used runbooks, secure automation tooling, and planned a full RCA.

Teaching angle:

Use in lessons:

Concepts to add or emphasize in the curriculum

Suggested lesson inserts

Add dedicated case-study lessons after foundational concepts:

  1. After Track 0 Lesson 2: “Case study preview: why Coinbase chose latency tradeoffs for exchange infrastructure.”
  2. After Track 1 Lesson 11: “Kubernetes case study: why draining 10 clusters worked for many services but not the exchange or Kafka.”
  3. After Track 2 Lesson 9: “Observability case study: detecting quote failures, Sev1s, dashboards, and incident response.”
  4. After Track 3 Lesson 6 or 12: “Kafka case study: broker/partition failover, TiBs of data, consumer lag, and delayed balance streams.”
  5. After Track 4 Lesson 2: “Matching engine case study: quorum failure, safe halting, cancel-only mode, auction reopen.”
  6. After Track 6 Lesson 8: “Disaster recovery case study: single-zone primary, distributed standby, and failure isolation gaps.”

Original post text captured for local reference

Yesterday @coinbase experienced a multi-hour service disruption affecting trading, exchange access, and balance updates. Here's our initial read from Coinbase engineering on what happened, how we recovered, and what we're addressing.

At approximately 23:50 UTC on 2026-05-07, our monitoring detected cascading quote failures from internal services that triggered multiple Sev1 incidents that engineering immediately began investigating. Customer-facing impacts included spot trading, Prime, International and derivative exchanges.

Root cause: a thermal event (cooling system failure) inside a subset of racks within a single building in AWS us-east-1. We run a primary replica of our exchange infrastructure in a single zone, consistent with industry standards to reduce latency. To prepare for failures like this, we maintain a distributed standby, but during this incident, failures in the primary zone that were designed to be isolated were not, extending the duration of our outage.

The failure cascaded down two paths:

1. Multiple hardware components beneath our exchange’s matching engine failed, requiring recovery and failover

2. Distributed Kafka clusters that manage messaging across Coinbase systems failed to remain available, also requiring partition failovers to new hardware brokers with many TiBs of data

After isolating the incident: automated tooling drained ~10 Kubernetes clusters worth of related workloads out of the affected zone to stabilize internal services. Most services were back to normal within ~30 minutes of diagnosis. The two things we couldn't automatically drain: the exchange (dedicated hardware and storage) and Kafka (managed service that was designed to be resilient to this, with unique problems).

The exchange matching engine is the core system responsible for processing orders and maintaining order books. It is a distributed cluster and requires quorum to safely elect a leader and continue processing trading activity. During the incident, infrastructure-level constraints in the affected datacenter left only a subset of nodes healthy, preventing the cluster from reaching quorum. As a result, trading across Retail, Advanced, and Institutional exchanges were blocked.

Recovery required our oncall and engineering teams to execute our disaster recovery plan, restore quorum safely, and validate system health under constrained infrastructure conditions. The team built, tested, deployed, and validated the fix while continuing to manage the broader incident.

Kafka recovery was a much larger scale operation. Our primary managed Kafka partitions process many terabytes of data daily and are designed with resiliency guarantees for uninterrupted operation during a datacenter failure just like this. In this case, those guarantees failed and required manual recovery.

We again relied on disaster recovery procedures to recover stuck partitions onto new hardware (brokers) that enabled us to safely bring x-service messaging back online across Coinbase. During the lag, customers saw delayed balance streams which resolved automatically once replication caught up. No data lost.

Once the engine came back up as part of our standard runbooks, we re-opened markets carefully: all products to cancel-only mode first, audited product states, then moved all markets to auction mode, before restoring trading on Coinbase Exchange.

What went right: the team. Incident response across the company came together within minutes, followed well-rehearsed playbooks and used secure automation tooling to recover all services. We have a strong, senior team at Coinbase that worked through rare failure modes to recover all services.

To our customers: losing access to your account, even temporarily, is unacceptable. We know that. We're sorry, and we’ll publish a full root cause analysis in the coming weeks 🙏

Built locally from Markdown on 2026-05-29 12:50:15