Scale Lab

10 missions and 23 lessons written. A naive Java ordering service, one defect a day: measure it, fix it, prove it.

1. 01Open your eyes: a dashboard tourRED, p50/p95/p99, why the average liesready
2. 0209:00 rush: the breaking pointLittle's Law, connection pool, saturationready
3. 03The dealer's order historyEXPLAIN, seq scan, index, Flyway migrationready
4. 04One request, many queriesN+1, Hibernate lazy loading, batch fetchingready
5. 05Slow payments slow down everythingtransaction boundary, pool exhaustionready
6. 06The last 10 itemsrace condition, lost update, atomic UPDATE, optimistic lockready
7. 07The provider stops answeringtimeouts (connect/read), fail fastready
8. 08Retry, but wiselyretry, exponential backoff, jitter (Resilience4j)ready
9. 09Open the circuitcircuit breaker, bulkhead, fallbackready
10. 10The same order twiceidempotency key, unique constraintready
11. 11Price list cacheRedis, cache-aside, TTLlesson only
12. 12Cache expired, everyone rushed the DBthundering herd, request coalescinglesson only
13. 13One dealer slows down everyonerate limiting, token bucket, noisy neighborlesson only
14. 143 instances + a load balancerstateless service, horizontal scaling, nginxlesson only
15. 15Virtual threadsJava 21+ Loom, thread-per-request, pinninglesson only
16. 16Don't lose the order eventdual write, outbox pattern, Kafkalesson only
17. 17The consumer crashedat-least-once, consumer idempotency, DLQlesson only
18. 18SLOs and alertsSLI/SLO, error budget, burn rate alertlesson only
19. 19Why is it slow: flame graphsprofiling (Pyroscope), allocation, GClesson only
20. 20Reads go to a replicaread replica, replication lag, read-your-writeslesson only
21. 21400k → 40M orderspartitioning, archiving, vacuumlesson only
22. 22Chaos daykilling instances, DB restart, graceful shutdownlesson only
23. 23Capacity reportcapacity planning, bottleneck chainlesson only