NOVEMBER 23, 2020

AI @Scale: Flyte: Making MLOps and DataOps a reality

Flyte is the backbone for large-scale Machine Learning and Data Processing (ETL) pipelines at Lyft. It is used across business critical applications ranging from ETA, Pricing, Mapping, Autonomous, etc. At its core it is a Kubernetes native workflow engine that executes 1M+ pipelines and 40M+ containers per month. Flyte abstracts complex infrastructure management from its users and provides a declarative fabric to connect disparate compute technologies. This increases productivity and thus product velocity by enabling them to focus on business logic. Flyte has made it possible to build higher-level platforms at Lyft, further reducing the barriers to entry for non-infrastructure engineers. The talk will focus on: Motivation and tenets for building Flyte, and parts of the Data Stack tackled by it. Architecture of Flyte and its specification language to orchestrate compute and manage data flow across disparate systems like Spark, Flink, Tensorflow, Hive etc. Use-cases where Flyte can be leveraged Extensibility of the Flyte and the burgeoning ecosystem.

To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookies Policy