AI @Scale 2020: Netflix’s Human-Centric Approach to ML Infrastructure

Netflix’s unique culture affords it’s data scientists extraordinary freedom of choice in ML tools and libraries. At the same time, they are responsible for building, deploying, and operating complex ML workflows autonomously without the need to be significantly experienced with systems or data engineering. Metaflow, our ML framework (now open-source at metaflow.org), provides them with delightful abstractions to manage their project’s lifecycle end-to-end, leveraging the strengths of the cloud: elastic compute and high-throughput storage. In this talk, we present our human-centric design principles that enable the autonomy our users enjoy.