@Scale 2018: Regression testing against real traffic on a Big Data reporting system

Adobe Analytics’ underlying query API handles thousands of complex reporting queries per second across a user base of hundreds of thousands. In his @Scale talk, Trenton Davies, Principal Scientist at Adobe, discusses how the Adobe engineering team has significantly increased the quality and frequency of releases over the past few years by mirroring production API traffic to its test builds and running regression analysis that compares the API responses of its production and test builds. Adobe’s novel approach allows its engineers to test millions of API permutations using real traffic, without mocking any portion of the system and without affecting the availability of Adobe’s production systems.