The recent revolution of LLMs and Generative AI is triggering a sea change in virtually every industry. Building new AI applications or incorporating AI in existing applications require developers to stitch together and scale a plethora of workloads from data ingestions, pre-processing, training, tuning/finetuning and serving. This is a very challenging task as different workloads require different systems, each of these systems coming with its own APIs, semantics, and constraints. Ray can dramatically simplify building these applications by providing a unified framework that can support and scale all these workloads. As a result, Ray has been increasingly being used by companies across industries to build scalable ML infrastructures, platforms, and applications. Examples include Uber, Spotify, Instacart, Netflix, Cruise, Ant Group, ByteDance, and OpenAI (to train ChatGPT and other large models). In this talk, I will present the design considerations behind Ray, our experience with using Ray, and the lessons we learned in the process

Serving multi-terabyte models at planet scale is not an easy challenge. It takes everything engineers have in their arsenal to achieve the scale. The problem is further complicated by scientists' ambitions for complex and larger models. Adding in security requirements only adds more dimension to already challenging space. This talk is a collection of some of the lessons we have learned during the OpenAI model launch journey.

Today, machine learning plays a key role in Uber’s business, being used to make business critical decisions across the board from marketplace pricing, Eats search and discovery, maps ETA, fraud detection etc. Michelangelo is an end-to-end ML platform that democratizes machine learning and enables ML practitioners to seamlessly build, deploy, and operate machine learning solutions at Uber’s scale.

In this talk, we will discuss how Michelangelo addresses the challenges of streamlining ML developer experience with seamless UI and code driven model iteration, supporting large-scale deep learning with a declarative ML application framework, and consolidating fragmented ecosystems with a unified API framework inspired by Kubernetes CRD design pattern.

While Michelangelo had some industry leading features like Horovod, Palette feature store etc, the ML industry is rapidly evolving. We believe that the future of Michelangelo is an open ML platform that leverages the best-of-class 3rd party or in-house ML components. We will share some early experience on the plug-and-play of ML components in Michelangelo by using our unified API framework.

We built IPnext as a modular control plane that comprises multiple loosely coupled model management domains like deployment, update, and synchronization. We believe that our design promotes simplicity, flexibility, and reliability -- all necessary components to meet the demanding and dynamic requirements of ML inference systems.

Flux is a system that uses global routing patterns to accurately size backend services based on frontend traffic demand shifts

This report presents the practice and evolution of high-performance networking in Alibaba Cloud Storage services. High-performance networking plays a crucial role in storage systems providing not only communication channels for users and storage services but also achieving high-speed interconnect between storage components. It is the cornerstone of highly available, performant cloud storage services.

Datacenter applications are often structured as many interconnected microservices, and service mesh has become a popular approach to route RPC traffic among services. We present, ServiceRouter (SR), perhaps one of the world’s largest service meshes, which has been in production since 2012. SR differs from publicly known service meshes in several important ways. First, SR is designed for hyperscale and currently uses O(10^6 ) of L7 routers to route O(10^10) of requests per second across O(10^4 ) services. Second, in contrast to the common approach of using sidecar or remote proxies, SR employs an embedded routing library, which reduces the hardware cost of our hyperscale service mesh by O(10^5 ) machines. Third, SR provides built-in support for sharded services, which account for 92% of the total RPC requests in our fleet, whereas existing general-purpose service meshes do not support sharded services. Finally, SR introduces the concept of locality rings to simultaneously minimize RPC latency and balance load across geo-distributed regions, which to our knowledge has not been attempted before.