Netflix's unique culture affords it's data scientists extraordinary freedom of choice in ML tools and libraries. At the same time, they are responsible for building, deploying, and operating complex ML workflows autonomously without the need to be significantly experienced with systems or data engineering.
Metaflow, our ML framework (now open-source at metaflow.org), provides them with delightful abstractions to manage their project's lifecycle end-to-end, leveraging the strengths of the cloud: elastic compute and high-throughput storage.

In this talk, we present our human-centric design principles that enable the autonomy our users enjoy.

Google BigQuery is a petabyte-scale serverless cloud data warehouse that enables scalable machine learning using SQL. In this talk, we take a look at how enabling data analysts and other SQL users to perform machine learning tasks can accelerate business decision-making and intelligence. We also present challenges in democratizing ML in large scale data warehouses such as BigQuery. We describe how a combination of general purpose SQL query engine and dedicated machine learning infrastructure can create a robust infrastructure for performing machine learning tasks.

We will discuss a novel model development process and tools we introduced to ads ranking machine learning teams, where a single model can be concurrently developed by dozens of engineers, whose changes to the model are centralized collected, combined, tested and launched.

Google BigQuery is a petabyte-scale serverless cloud data warehouse that enables scalable machine learning using SQL. In this talk, we take a look at how enabling data analysts and other SQL users to perform machine learning tasks can accelerate business decision-making and intelligence. We also present challenges in democratizing ML in large scale data warehouses such as BigQuery. We describe how a combination of general purpose SQL query engine and dedicated machine learning infrastructure can create a robust infrastructure for performing machine learning tasks.

Flyte is the backbone for large-scale Machine Learning and Data Processing (ETL) pipelines at Lyft. It is used across business critical applications ranging from ETA, Pricing, Mapping, Autonomous etc. At its core it is a Kubernetes native workflow engine that executes 1M+ pipelines and 40M+ containers per month.

Flyte abstracts complex infrastructure management from its users and provides a declarative fabric to connect disparate compute technologies. This increases productivity and thus product velocity by enabling them to focus on business logic. Flyte has made it possible to build higher-level platforms at Lyft, further reducing the barriers to entry for non-infrastructure engineers.

The talk will focus on:

Motivation and tenets for building Flyte, and parts of the Data Stack tackled by it.
Architecture of Flyte and its specification language to orchestrate compute and manage data flow across disparate systems like Spark, Flink, Tensorflow, Hive etc.
Use-cases where Flyte can be leveraged
Extensibility of the Flyte and the burgeoning ecosystem.