Daniel Greenia continued the theme of improving the triage process to better leverage human analysis. Advertisers are the lifeblood of Google’s AdWords business, but at Google’s scale, it can be difficult to determine which advertisers are real and which are fraudsters. Daniel described the system they use to detect fraud, showing how machine learning could catch many, but not all, malicious accounts in their system. For the accounts in the “gray zone,” trained human analysts are still the last line of defense. Still, despite the scale of their automatic classifier, analysts still had more work to do than time in which to do it. After this, they were able to rank each suspicious account according to how much “good money” or “bad money” was at stake. From this a pattern emerged, producing an extremely simple formula that analysts could use to determine which accounts were worth reviewing manually, and what order to review them in. With this work, the team was able to improve their performance by several multiples with just a fraction of the effort.