News & Ideas | At Scale Conferences

Deep learning models may deliver personalized content—from short videos to friend posts—to people on apps. As these models have grown in complexity, so has the importance of the compute that trains them, and the network that connects those accelerators. Training recommendation models is a unique infrastructure challenge. Unlike large language models, which need enormous floating-point […]

By: Rajiv Krishnamurthy ...

Systems and Networking

MTIA 300: Meta’s First Training Chip with Built-in NICs and Communication-Offloading Engines

Deep learning models may deliver personalized content—from short videos to friend posts—to people on apps. As these models have grown in complexity, so has the importance of the compute that trains them, and the network that connects those accelerators. Training recommendation models is a unique infrastructure challenge. Unlike large language models, which need enormous floating-point […]

MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet

At Meta, we’ve been a strong driver behind the industry’s growing consensus that Ethernet should be the fabric of choice for AI infrastructure. We’ve already shown that RoCE can power distributed AI training at scale. Now, we’re building on that work with MetaRoCE, protocol designed from the ground up for Ethernet at million-GPU scale. We’ve scaled up clusters […]

Building the Backbone of AI: Inside the NAO Mega Cluster’s Fiber Revolution

When we talk about the future of artificial intelligence, the conversation often centers on models, algorithms, and GPUs. But beneath every breakthrough in AI lies something far more tangible—the physical infrastructure that makes it all possible. In New Albany, Ohio, a first-of-its-kind engineering project is quietly redefining what’s possible in data center connectivity: the NAO […]

Keynote from Meta

Peter will share the latest progress from Meta’s mission to build the future of human connection and the technology that makes it possible, including infrastructure for AI by leveraging agentic innovation across the stack.

WATCH VIDEO

Systems and Networking

LIVE PANEL WITH AURORA, GOOGLE, META AND MICROSOFT

Saad Syed (Google) Mark Jen (Meta) Viral Mehta (Aurora) Jenny Lawrance (Microsoft)

WATCH VIDEO

Systems and Networking

AI Storage Blueprint

The rapid, exponential growth in model capabilities and training dataset sizes over the last few years has accelerated AI innovation. New frontier models are now being released in a matter of weeks, down from months just a year ago. This pace makes reliable, consistent fast access to storage essential for managing both the speed and […]

WATCH VIDEO

Systems and Networking

Taming AI Infrastructure Failures with Agentic Debugging

NCCL watchdog timeouts are a common failure mode in distributed AI model training. They impact not only Meta, but broadly affect anyone running PyTorch distributed training—and they’re notoriously hard to debug: even experts can spend hours triaging a single incident, and non-experts may be unable to root-cause them at all. Over the past year, we […]

WATCH VIDEO

Systems and Networking

Securing Production Debugging at Hyperscale

This presentation will go over how Microsoft uses SSH to debug across our global fleet of K8s clusters at hyperscale. We’ll cover how we integrate that with JIT elevations and Task-Based Execution in a secure and auditable way. We will alsogo over how we use these building blocks to engage AI and the Semantic Kernel […]

WATCH VIDEO

Systems and Networking

[Almost] Fail & Tell: Stop The World

In March 2025, Meta avoided an major outage that would have resulted in prolonged sitewide unavailability. Instead, we were able to quickly contain the issue in a matter of minutes and avoid the worst case scenario. This is a story of how we re-bootstrapped the infrastructure control plane in an entire region, on the fly, […]

WATCH VIDEO

Systems and Networking

Building Privacy Aware Infrastructure in the AI-Native Era

Building privacy-aware infrastructure in the AI-native era requires systems that can learn, evaluate, and operationalize improvements continuously—at the scale of millions of data assets. This talk presents an AI-native approach through a case study of AI-native asset classification: a hybrid deterministic rules engine with LLM fallback that transforms messy context (metadata, lineage, code references, scan […]

WATCH VIDEO

Systems and Networking

Experience Productionizing and Operating GB200 clusters

The NVIDIA GB200 NVL72 functions as an exascale computer within a single rack. Maximizing the utilization of its NVLink fabrics through proper scheduling allows AI training jobs to significantly leverage the substantial networking bandwidth this system offers. Notably, recent MLPerf training results (>2.6x improvement with recent MLPerf training) confirm that GB200 NVL72 provides substantial performance […]

WATCH VIDEO

Systems and Networking

Teaching AI to Fight Fires: Autonomous Reliability Agents at Meta Scale

In December 2024, a single config change at Meta took 50+ engineers and 28 hours to recover from. What if an AI agent had detected the cascade in seconds and proposed the rollback before it propagated? Over the past year, we’ve been building exactly that. This talk introduces the reliability flywheel: a system that encodes […]

WATCH VIDEO

LATEST ON @SCALE

MTIA 300: Meta’s First Training Chip with Built-in NICs and Communication-Offloading Engines

MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet

Building the Backbone of AI: Inside the NAO Mega Cluster’s Fiber Revolution

Keynote from Meta

LIVE PANEL WITH AURORA, GOOGLE, META AND MICROSOFT

AI Storage Blueprint

Taming AI Infrastructure Failures with Agentic Debugging

Securing Production Debugging at Hyperscale

[Almost] Fail & Tell: Stop The World

Building Privacy Aware Infrastructure in the AI-Native Era

Experience Productionizing and Operating GB200 clusters

Teaching AI to Fight Fires: Autonomous Reliability Agents at Meta Scale