Reliability at Scale
Advanced SRE for Staff Engineers
Built on questions I carried through 16 years of running mission-critical telecom platforms. Most serious reliability problems do not come from missing tools. They come from broken feedback loops.
Technical loops
inside the system
They grow an outage and hold it in place. Retry storms that backoff and jitter cannot fix. Outages that outlive their trigger, where you roll the bad deploy back, traffic returns to normal, and the system is still on fire.
Organizational loops
around the system
Measurement that never turns into a decision. Error budgets that run out while the launch ships anyway. Postmortems that find the root cause, close every action item, and still let a similar incident happen again.
A technical loop keeps an outage from ending. An organizational loop makes it happen again.
Fourteen chapters covering blast radius through cells and shuffle sharding, telemetry cost and gray failure, capacity by queueing theory, safe config delivery, incident command, and a side-by-side reading of 16 public postmortems.