AI @Scale: Flyte: Making MLOps and DataOps a reality
Flyte is the backbone for large-scale Machine Learning and Data Processing (ETL) pipelines at Lyft. It is used across business critical applications ranging from ETA, Pricing, Mapping, Autonomous, etc. At its core it is a Kubernetes native workflow engine that executes 1M+ pipelines and 40M+ containers per month. Flyte abstracts complex infrastructure management from its users and provides a declarative fabric to connect disparate compute technologies. This increases productivity and thus product velocity by enabling them to focus on business logic. Flyte has made it possible to build higher-level platforms at Lyft, further reducing the barriers to entry for non-infrastructure engineers. The talk will focus on: Motivation and tenets for building Flyte, and parts of the Data Stack tackled by it. Architecture of Flyte and its specification language to orchestrate compute and manage data flow across disparate systems like Spark, Flink, Tensorflow, Hive etc. Use-cases where Flyte can be leveraged Extensibility of the Flyte and the burgeoning ecosystem.
Join the @Scale Mailing List and Get the Latest News & Event Info
Code of Conduct