Meta engineers Suja and Sophie share how they’ve rebuilt software testing for the AI era, moving beyond traditional “landed tests” that only catch known bugs to a new system of disposable, AI-generated tests that target specific code changes.
When a risky change appears, their system scores diffs for risk factors like blast radius and intent mismatch, then generates property-based or mutation-based tests designed to fail only when that specific change introduces defects [6:09] [6:45]. Context is critical: providing the generator with code change details and author intent roughly doubled catch rates at each step.
A key innovation is the two-stage architecture: an aggressive generator proposes many hypotheses, while an LLM judge filters out intended behavioral changes to protect engineer attention, discarding the vast majority of generated tests before they reach humans.This reduced engineer time spent chasing bugs by 70% despite not finding fewer actual bugs.
The system has prevented 74 documented safety incidents and bent privacy incident growth to one-fifth of forecasted levels through counterfactual measurement .
The talk closes with three ongoing shifts: tests becoming disposable, solving the long-term trust challenge, and moving testing left from review to co-authorship during code writing.
