Daily Infrastructure / Platform / System Design Learning Track

Audience: Nick knows the basics of apps and product work, but is a novice in infrastructure/platform/system design. Each lesson should explain terms from first principles, avoid assuming prior infra knowledge, and explicitly connect back to previous lessons.

Role orientation: balance fundamentals with practical leadership relevance. Nick still wants to learn the basics of Kubernetes, observability/Datadog, Kafka, Redis/caching, databases, networking, and distributed systems from first principles. Do not skip the beginner explanations. The Coinbase manager/technical-leader angle should be an added lens: after teaching the concept clearly, explain how it helps with reading existing systems, asking better design/reliability questions, understanding incident reports, making tradeoffs explicit, improving runbooks/observability, coordinating across teams, and spotting operational risk before it becomes customer impact.

Cadence: one email per day. The order below is the default path. Start with Track A, a short Coinbase outage concept series, beginning tomorrow. After that focused series, continue into foundations and the other topic tracks. Do not make every future email about this one incident; reuse it only when it is a natural example. Do not repeat completed entries in sent-index.md or digests/.

How each digest should reference the sequence

Every lesson should include:

1. Where we are in the track — e.g. “Track A / Lesson 1: Coinbase outage overview.”
2. Previously — 2–4 bullets tying this lesson to prior lessons.
3. Beginner foundation first — explain the basic concept from first principles before discussing leadership/operational implications.
4. Why this matters in Nick’s Coinbase role — after the basics, explain how the concept helps a manager/technical leader in established systems: better questions, better risk assessment, better incident follow-up, better cross-team execution.
5. Today’s mental model — one simple analogy.
6. Core concept — novice-friendly explanation.
7. Real-world example — from the Coinbase outage, large-company incidents, popular system design books, SRE/DDIA concepts, or realistic Coinbase/trading/social-app scenarios.
8. Why this matters for prediction markets/trading/social apps — make it relevant to Nick’s interests.
9. Common failure modes — what breaks in production.
10. Technical-leader checklist — questions Nick could ask in design reviews, planning, incident reviews, or operational readiness reviews.
11. Mini design exercise — one small prompt to think through.
12. Terms to remember — glossary.
13. Next time — preview the next lesson and how it builds on today.

---

Track A — Start here: short Coinbase outage concept series

This is the first sequence to send, starting tomorrow. Use case-studies/2026-05-07-coinbase-multi-hour-service-disruption.md as the anchor source for a focused opening series only. The goal is to use one real Coinbase incident to introduce several high-value concepts, then move on to the broader fundamentals tracks. Do not turn the entire course into Coinbase-outage commentary.

1. Incident overview: what failed, customer impact, timeline, and why the incident matters
2. Reading incident reports: symptoms, root cause, contributing factors, mitigation, follow-ups
3. Latency vs availability and fault domains: why single-zone primaries exist and how blast radius works
4. Observability during the incident: quote failures, Sev1s, monitors, dashboards, and customer-impact signals
5. Kubernetes draining and stateful vs stateless systems: why many workloads moved quickly but exchange/Kafka did not
6. Matching engines and quorum: order books, leader election, safety, and why blocking trading can be correct
7. Kafka and balance streams: brokers, partitions, replication, lag, delayed freshness, and no-data-loss recovery
8. Disaster recovery and safe market reopening: runbooks, validation, cancel-only mode, product-state audit, auction mode

After Track A, continue to Track 0 and the topic-specific tracks. Later lessons may reference this incident only when it genuinely clarifies the concept being taught.

Track 0 — Foundations for thinking about production systems

These come before deeper tools. They establish the vocabulary used by Kubernetes, Kafka, observability, and large-scale app design.

1. Client/server basics, requests, responses, and APIs
2. Latency vs throughput vs availability
3. Processes, ports, hosts, and networking basics
4. DNS, load balancers, TLS, and reverse proxies
5. Stateless vs stateful services
6. Vertical vs horizontal scaling
7. Caches and why they make systems faster but trickier
8. Databases: primary keys, indexes, transactions, and replication
9. Queues vs synchronous requests
10. Failure as a normal design input: retries, timeouts, backoff, idempotency
11. Consistency basics: strong vs eventual consistency
12. Capacity planning and back-of-the-envelope estimates

Track 1 — Kubernetes from zero

Kubernetes is one major topic/path, not the whole curriculum. Nick knows almost nothing about it, so start with basic container and deployment concepts.

1. Why Kubernetes exists: running many services on many machines
2. Containers vs virtual machines: the packaging problem Kubernetes builds on
3. Images, registries, and immutable releases
4. Pods: the smallest deployable unit
5. Deployments and ReplicaSets: desired state and self-healing
6. Services: stable networking for changing pods
7. Ingress and gateways: exposing services to the internet
8. ConfigMaps and Secrets: configuration without rebuilding images
9. Resource requests/limits: CPU, memory, scheduling, and noisy neighbors
10. Health checks: liveness, readiness, startup probes
11. Rolling deployments, rollbacks, and canaries
12. Namespaces, labels, and selectors: organizing clusters
13. StatefulSets and persistent volumes: why databases are harder
14. Jobs and CronJobs: one-off and scheduled work
15. Autoscaling: HPA, cluster autoscaler, and scaling signals
16. Kubernetes failure modes: CrashLoopBackOff, pending pods, bad rollouts
17. Observability in Kubernetes: logs, metrics, traces, events
18. Security basics: RBAC, service accounts, network policies
19. Platform teams and internal developer platforms built on Kubernetes
20. Coinbase/trading/social-app example: deploying a market data API safely

Track 2 — Observability, Datadog, monitoring, and production debugging

Nick knows next to nothing about this. Teach conceptual foundations first, then Datadog-style products.

1. Why observability exists: knowing what production is doing
2. Monitoring vs observability vs alerting
3. Metrics: counters, gauges, histograms, percentiles
4. Logs: structured events, correlation IDs, and search
5. Traces: following one request across many services
6. The three pillars and why “pillars” are not enough
7. SLIs, SLOs, SLAs, and error budgets
8. Dashboards: useful views vs vanity graphs
9. Alerts: symptoms, thresholds, paging, fatigue, and runbooks
10. Datadog mental model: agents, integrations, tags, dashboards, monitors
11. Application Performance Monitoring (APM): spans, services, flame graphs
12. Infrastructure monitoring: hosts, containers, Kubernetes, databases
13. Log pipelines: ingestion, parsing, indexing, retention, cost
14. Distributed tracing for trading/social app request flows
15. Incident response: detect, triage, mitigate, postmortem
16. Observability for Kafka and queues: lag, throughput, consumer health
17. Observability for Kubernetes: pod health, node pressure, rollout debugging
18. Observability cost control: cardinality, sampling, retention, indexes
19. Coinbase/trading example: detecting stale prices or delayed settlement
20. Social app example: debugging notification delivery delays

Track 3 — Kafka, streams, queues, and event-driven systems

Kafka and related topics are another learning path. Start from “why async?” and build toward real-time trading/social systems.

1. Synchronous APIs vs asynchronous messaging
2. Queues vs logs: SQS-style queues vs Kafka-style event streams
3. Kafka mental model: topics, partitions, offsets, brokers, consumers
4. Producers and consumers: delivery contracts and retries
5. Consumer groups and parallelism
6. Partition keys: ordering, hot partitions, and scaling
7. Retention and replay: why event logs are powerful
8. At-least-once, at-most-once, exactly-once, and effectively-once processing
9. Idempotency and deduplication in event-driven systems
10. Schemas: Avro/Protobuf/JSON, compatibility, schema registries
11. Stream processing: Kafka Streams, Flink, materialized views
12. Backpressure and consumer lag
13. Dead-letter queues and poison messages
14. Event sourcing vs using Kafka as a pipe
15. CDC: change data capture from databases into streams
16. Market data pipelines: price updates, order book deltas, sequence numbers
17. Social app pipelines: feed events, notifications, ranking signals
18. Monitoring Kafka: broker health, ISR, lag, throughput, disk usage
19. Disaster recovery and multi-region Kafka tradeoffs
20. When not to use Kafka

Track 4 — Redis, caching, and fast data access

Redis and caching are another dedicated learning path. Start from why caches exist, then build toward production failure modes, Redis data structures, and use cases in trading/social apps.

1. Why caching exists: avoiding repeated expensive work
2. Cache-aside mental model: app, cache, database
3. Redis from zero: in-memory key-value store, commands, TTLs
4. Cache hits, misses, evictions, and expiration
5. Cache invalidation: why it is famously hard
6. Read-through, write-through, write-behind, and cache-aside patterns
7. Preventing cache stampedes: locks, jitter, request coalescing, stale-while-revalidate
8. Hot keys and uneven traffic distribution
9. Redis data structures: strings, hashes, lists, sets, sorted sets, streams
10. Redis for rate limiting: counters, token buckets, sliding windows
11. Redis for sessions, auth tokens, and ephemeral state
12. Redis for leaderboards, rankings, and social-app feeds
13. Redis Pub/Sub and streams: when Redis is messaging, and when Kafka is better
14. Persistence and durability: RDB, AOF, and why Redis is usually not your source of truth
15. Redis replication, Sentinel, Cluster, and failover basics
16. Consistency risks: stale data, lost writes, split-brain, and read-after-write surprises
17. Observability for caches: hit rate, latency, memory, evictions, key cardinality
18. Caching in trading/prediction markets: prices, order books, balances, and what not to cache
19. Multi-layer caching: browser, CDN, edge, app cache, database cache
20. When not to use Redis or a cache

Track 5 — Trading, prediction markets, and financial platform design

1. Order books, bids/asks, spreads, and matching
2. Matching engines: determinism, sequencing, fairness, and latency
3. Market data fanout: snapshots, deltas, sequence numbers
4. Idempotency keys in trading/payments APIs
5. Double-entry ledgers and balances as derived state
6. Holds, reservations, settlement, and reconciliation
7. Risk checks, limits, and kill switches
8. Circuit breakers and market halts
9. Consistency tradeoffs in balances, orders, and fills
10. Audit logs and compliance-driven system design
11. Hot/warm/cold wallet architecture as platform design
12. Prediction market architecture: markets, outcomes, resolution, settlement

Track 6 — Social apps and large-scale product systems

1. Feed architecture: fanout-on-write vs fanout-on-read
2. Ranking systems: candidate generation, scoring, serving
3. Notifications: preference fanout, dedupe, batching, quiet hours
4. Realtime chat/presence: websockets, pub/sub, connection state
5. Graph systems: follow, block, mute, recommendation edges
6. Media upload and delivery: object storage, CDN, transcoding
7. Search systems and indexing pipelines
8. Abuse prevention: rate limits, reputation, queues, human review
9. Feature flags and experimentation platforms
10. Privacy and data deletion at scale

Track 7 — Reliability and platform engineering

1. SRE basics: reliability as a product feature
2. Load shedding, backpressure, and overload protection
3. Rate limiting: token bucket and leaky bucket
4. Retries, timeouts, exponential backoff, jitter
5. Bulkheads and circuit breakers
6. Deployments: blue/green, canary, progressive delivery
7. Zero-downtime migrations
8. Multi-region architectures: active-active vs active-passive
9. Disaster recovery: RPO, RTO, backups, restores
10. Internal developer platforms and golden paths

Suggested source pool

Local case studies to reuse

• case-studies/2026-05-07-coinbase-multi-hour-service-disruption.md — Coinbase outage involving thermal event in AWS us-east-1, Kubernetes draining, matching-engine quorum, Kafka partition failover with TiB-scale data, delayed balance streams, cancel-only/auction reopening, Sev1 incident response, and disaster recovery tradeoffs.

Use reputable sources when possible, including:

• Designing Data-Intensive Applications by Martin Kleppmann
• Google SRE books
• System Design Interview / Alex Xu books
• Kubernetes official docs
• Kafka official docs and Confluent engineering material
• Datadog documentation and engineering blog
• Coinbase, Stripe, Uber, Meta, Twitter/X, Netflix, Discord, Slack, Cloudflare, AWS, Google engineering blogs