In December 2024, a single config change at Meta took 50+ engineers and 28 hours to recover from. What if an AI agent had detected the cascade in seconds and proposed the rollback before it propagated?
Over the past year, we’ve been building exactly that. This talk introduces the reliability flywheel: a system that encodes the best on-call engineer’s investigation methods into an always-on copilot that gets sharper with every incident. Our investigation agent has handled 1,000+ incidents across Meta’s recommendation systems, cutting detection-to-mitigation time by 60% and matching senior on-call engineers ~80% of the time. It correlates signals across time-series metrics, deployment logs, configuration changes, and infrastructure health through a context-engineered architecture: structured tool servers and reusable workflows.
We’ll trace the full progression: (1) pattern identification across hundreds of incidents, (2) autonomous investigation that matches human accuracy, (3) supervised mitigation for low-risk reversible actions, and (4) self-healing as the long-term outcome.
You’ll leave with what worked, what didn’t, and how we think about the trust boundary between autonomous agents and human engineers in production.
