Videos & Articles
Clear
Data @Scale 2017
Bulk data movement serving Facebook’s global data storage and processing
Steve Stroiney describes Facebook’s system for the movement of bulk data across its global system for storage and processing, balancing business needs with being a “good citizen,” at Data @Scale.
Data @Scale 2017
Architectures for the new era of cloud specialization
Increasing computational needs are forcing us to change how we think about large-scale systems, with new stacks and new interfaces. Doug Burger describes the deployment of programmable hardware in the Microsoft cloud, at Data @Scale.
Data @Scale 2017
Spanner’s SQL evolution
Google is in the process of turning its globally distributed Spanner data management system into an SQL DBMS. At Data @Scale, Sergey Melnik details the evolution of Spanner and its technical aspects.
Data @Scale 2017
How reporting and experimentation fuel product innovation at LinkedIn
The deeply data-driven LinkedIn uses metrics measurement and experimentation in every product decision. Kapil Surlaker introduces the Data @Scale audience to UMP and XLNT, two platforms the company developed to efficiently process ...
Data @Scale 2017
Cadence: Microservice architecture beyond request/reply
Maxim Fateev talks about Cadence, the open source solution that Uber uses for building and running microservices that expose asynchronous, long-running operations in a scalable and resilient way, at Data @Scale.
Data @Scale 2017
Evolution of storage and serving at Pinterest
Yongsheng Wu talks to the Data @Scale crowd about the evolution of storage and serving at scale at Pinterest as the company has grown, enabling rapid product innovations and confronting new challenges efficiently.
Data @Scale 2017
Yandex ClickHouse: A DBMS for interactive analytics at scale
At Data @Scale, Alexey Milovidov walks through Yandex’s development of ClickHouse and how its iterative approach to organizing data storage has resulted in a powerful and extremely fast open source system.
Data @Scale 2017
Next generation of globally distributed databases in Azure
Rimma Nehme describes for Data @Scale the new databases at Microsoft that are distributed worldwide and can run on millions of nodes across hundreds of data centers, handling up to trillions of data objects.
Data @Scale 2017
Accelerating machine learning for computer vision
Pieter Noordhuis of Facebook shares some recent insights in distributed systems design for training machine learning models at scale, at Data @Scale.
Networking @Scale 2017
How Facebook designs its own hardware and network
Hans-Juergen Schmidtke and Jimmy Williams take the Networking @Scale audience on a tour of data center networks and the hardware that allows Facebook to build them while meeting demand.
Networking @Scale 2017
Solving large-scale data center and cloud interconnection problems
Equinix has used open disaggregated network hardware and software to transform data center interconnection and cloud connectivity services. Ihab Tarazi tells us about the company’s approach at Networking @Scale.
Networking @Scale 2017
High-altitude platforms for rural backhaul
In his talk at Networking @Scale, Adit Sahasrabudhe focuses on Facebook’s work to deliver backhaul connectivity links from high-altitude platforms (HAPs), and details three link technologies that make up the network architecture.
Networking @Scale 2017
Microsoft IT IPv6 only: The path ahead
As Microsoft IT works toward a v6-only corporate network, Todd Thorson explains the strategy the team is taking, along with its plans to overcome the challenges along the way, at Networking @Scale.
Networking @Scale 2017
Delivering IPv6 on Amazon Virtual Private Cloud
Alan Halachmi covers the process of delivering IPv6 for Amazon Virtual Private Cloud (VPC), including decisions made along the way and the creation of an egress-only internet gateway, at Networking @Scale.
Networking @Scale 2017
Internet-scale virtual networking using IPv6 ILA
At Networking @Scale, Petr Lapukhov sheds light on what virtual networking is and how it works, and talks about the ways ILA (identifier locator addressing) has helped Facebook implement it.
Networking @Scale 2017
The very best of Certificate Transparency (2011-present)
Katrina Joyce and Al Cutter give a tour of Google’s Certificate Transparency project, which allows the detection of mistakenly issued TLS certificates, at Networking @Scale.
Networking @Scale 2017
Facebook CDN: Minimizing latency and jitter for content delivery
Sumanth Sukumar discusses the work of Facebook’s traffic infrastructure team and talks about proactive CDN for media delivery, at Networking @Scale.
Networking @Scale 2017
Zero Protocol and TLS 1.3: Low-latency transport security at Facebook
Subodh Iyengar talks about Zero Protocol, Facebook’s custom 0-RTT security protocol based on QUIC Crypto, and the reasons behind the current transition to TLS 1.3, at Networking @Scale.
Networking @Scale 2017
Delivering terabits of content: External considerations
Aaron Klink of Netflix talks to the Networking @Scale crowd about the various external aspects involved in delivering dozens of terabits of video traffic to nearly 100 million streaming subscribers.
Networking @Scale 2017
A close look at Alibaba’s high-performance packet processing platform
Alibaba built a packet processing platform that powers all its gateway-class services and is essential to the company’s day-to-day global operations. Haiyong Wang gives us the details at Networking @Scale.
Networking @Scale 2017
Automating and scaling the Edge
Facebook’s Xiaozheng Tie talks about how engineers adapted Robotron to automate and scale the new Edge network, from design to deployment and beyond, at Networking @Scale.
Networking @Scale 2017
Scaling for exabytes
Jason Wilson covers the challenges Dropbox confronted in building its own in-house multi-terabit storage system, and talks about the lessons learned, at Networking @Scale.
Networking @Scale 2017
Moving fast with Facebook’s long-haul network
Henry Kwok introduces a new private long-haul network for efficiently interconnecting Facebook’s data centers. He also discusses design aspects and touches on future goals, at Networking @Scale.
Dev Tools @Scale 2017
Bazel: Google’s extensible, multilingual, scalable build system
Dmitry Lomov walks through the architecture and extensibility model of Bazel, Google’s open source build system, and describes how it scales for different codebases.
Dev Tools @Scale 2017
Searching through code at scale
Jeroen Vaelen explores some of the custom code search tools Facebook is developing for large repositories, allowing engineers to efficiently find files and commits that match certain patterns.
Dev Tools @Scale 2017
Cross-platform dev tools for million-core applications
Through two practical use cases, Chris January of ARM demonstrates the challenges and architectural choices involved in creating dev tools for supercomputers that approach 1 million cores.
Dev Tools @Scale 2017
Scaling Android application testing at Spotify
Sean Kenny of Spotify delves into new testing tools that engineers have created to help teams keep up with the rapid pace of Android development.
Dev Tools @Scale 2017
Git Virtual File System
Edward Thomson discusses the challenges of migrating Microsoft’s Windows team to Git, and the changes in Visual Studio Team Services and in the Git Virtual File System that made it possible.
Dev Tools @Scale 2017
Scaling up job scheduling with a matchmaking service
Adriana Libório introduces Jupiter, a matching service that has helped Facebook scale its infrastructure by balancing the different requirements of jobs with the different configurations and dependencies of servers.
Dev Tools @Scale 2017
Hosting large-scale repositories on GitHub
Carlos Nieto walks through how GitHub has developed projects that have made it better able to reliably host Git repositories in an array of wildly different use cases.
Dev Tools @Scale 2017
Facebook Infer: Advanced static analysis at speed and scale
Dulma Churchill and Jules Villard describe the Infer Static Analyzer tool, which helps Facebook quickly catch and fix complex bugs before code gets committed, and reveal a domain-specific language for writing simple static analysis checks.
Dev Tools @Scale 2017
A tale of two pizzas: Accelerating software delivery with AWS developer tools
Amazon’s Paul Maddox shares the best practices and tools used by Amazon’s “two pizza” engineering teams to release software quickly. Implementing continuous integration and delivery workflows has helped ...
Dev Tools @Scale 2017
Resource management at scale
Phyllipe Medeiros talks about how Facebook scales end-to-end testing at Facebook, providing signal as quickly and reliably as possible so that its engineers can move fast.
Video @Scale 2017
Video understanding at scale
At Video @Scale, Manohar Paluri explores some of the challenges of video understanding, highlighting Facebook’s recent efforts as well as some exciting results.
Video @Scale 2017
Any device, any network: DIRECTV API
Jonathan Tronson recounts AT&T Entertainment’s endeavor to build a cloud-native video API to support growing demand, at Video @Scale.
Video @Scale 2017
More efficient encoding for mobile video
At Video @Scale, Anne Aaron describes how Netflix improved video quality over low-bandwidth, unreliable networks for its global audience.
Video @Scale 2017
Scaling low-latency live streams (2 of 2)
In this talk from Video @Scale, Federico Larumbe covers end-to-end considerations for delivering live streams on Facebook Live.
Video @Scale 2017
Scaling low-latency live streams (1 of 2)
In this talk from Video @Scale, Saral Shodhan talks about lower latencies for real-time interactions on Facebook Live.
Video @Scale 2017
Transitioning codecs on mobile
At Video @Scale, Christopher Blay and James Zern talk about Google’s VP9 format and its uses and benefits.
Video @Scale 2017
Spatial audio at scale
Jen Dolson and Hans Fugal explore the technical details behind Facebook’s use of high-quality spatial audio for large-scale consumption, at Video @Scale.
Video @Scale 2017
Streaming 360 video
At Video @Scale, Evgeny Kuzyakov talks about some of the improvements to Facebook’s Dynamic Streaming system.
Video @Scale 2017
Next-generation transport for live, glitch-free, high-quality video delivery over commodity internet
Aspera CEO Michelle Munson discusses the core principles, performance characteristics, and use cases of the FASPStream software at Video @Scale.
Machine Learning @Scale 2017
Building AI for everyone on the planet
Clarifai CEO Matthew Zeiler takes viewers through a suite of product demos for identifying, classifying, and searching digital images using the company’s machine learning technology.
Machine Learning @Scale 2017
Medical specialty triage using machine learning
Michelle Ye uses her perspective as a data scientist to talk through the challenge of scaling engineering ideas through an organization to gain buy-in and bring benefits to both the company and the end user.
Machine Learning @Scale 2017
Detecting place visits at scale
Jan and Danielle offer a deep dive into a system capable of interpreting location signals coming from mobile devices at scale. The case study they present exposes challenges their team faced while designing and productionizing a system ...
Machine Learning @Scale 2017
Measurement and analysis of predictive feed ranking models on Instagram
Thomas uses the launch of Instagram’s feed ranking as a working example to talk through issues in quantifying network effects, while exploring unusual A/B testing techniques such as country-level tests, testing on balanced graph ...
Machine Learning @Scale 2017
Learning in auctions
Andres describes the challenges of learning in repeated auctions for revenue maximization.
Machine Learning @Scale 2017
Matching publications and patents to LinkedIn members
Xiaoqiang presents a recent project in which papers, patents, and other professional content created by LinkedIn members are pulled from the web and matched to their creators automatically. The matched content is sent to LinkedIn ...
Machine Learning @Scale 2017
Search and ranking at Bloomberg
Parth discusses the challenge of providing effective search for financial markets, balancing the need for accuracy and speed, the diversity of the data, and the difficulty of gaining an accurate picture of markets from moment to ...
Machine Learning @Scale 2017
Designing AI at scale to power everyday life
Joaquin shares insight into how Facebook is conducting and applying industry-leading research to help drive advancements in AI disciplines like computer vision and language understanding.
Spam Fighting 2016
Welcome to Spam Fighting @Scale 2016
Vlad Gorelik, an engineering manager at Facebook, kicked off the event describing his work supporting teams that work on fighting spam and related issues. He discussed how spam-fighting techniques and approaches are similar across ...
Spam Fighting 2016
How to Utilize User-Generated Feedback to Fight Spam
When used in aggregate, user reporting can provide valuable indicators that can supplement automated systems. Algorithmic-based systems rely on classes of signals that have previously been shown to correspond to spam attacks. However, ...
Spam Fighting 2016
Defending Netflix and its Members from Abuse
Netflix is the world’s leading Internet television network with over 83 million members in over 190 countries enjoying more than 125 million hours of TV shows and movies per day. We’ll discuss the range of unique abuse-related ...
Spam Fighting 2016
Using Weighted Sampling to Understand the Prevalence of Spam
To effectively fight spam, we need an unbiased estimate of how much bad content there is in the ecosystem and where it resides. In this presentation we discuss sampling schemes to identify the small percentage of bad content viewed ...
Spam Fighting 2016
The Economics of Modern Spam
For most people, spam is a nuisance. Behind this nuisance, however, is a profitable business operation that continues to thrive despite the billions of dollars spent trying to disrupt it. This talk will cover the technical and economic ...
Spam Fighting 2016
Measuring Performance of Fake Account Detection and Remediation
Effective fake account defense systems are important to preventing spam without impacting product growth. This presentation will discuss some of the methods Facebook uses to understand the performance of fake account detection and ...
Spam Fighting 2016
Finding Clusters of Fake Accounts at Scale
Fake accounts are a preferred means for malicious users of online social networks to send spam, commit fraud, or otherwise abuse the system. In order to scale, a single malicious actor may create dozens to thousands of fake accounts; ...
Spam Fighting 2016
How WhatsApp Reduced Spam while Launching End-to-End Encryption
End-to-end encryption, which protects message content so that only the sender and recipient can access it, is gaining popularity in messaging applications. At the same time, there is some concern about its deleterious effects on spam ...
Spam Fighting 2016
Detecting fake inventory on Airbnb
Millions of hosts & guests confidently list & book on the Airbnb platform. Our trust and safety features give them the comfort and confidence in our marketplace. Bad actors can be incentivized to take advantage of the good will ...
Spam Fighting 2016
Anti-Spam by Design: Building a Company of Spam-Fighters
Spam fighting isn’t just about writing policies, training classifiers, and combating attacks. Leveraging the “secure by design” principal in the context of spam helps create better products with built-in features for preventing ...
Boston Networking @Scale 2016
Networking @ Google
This talk will present a high level view of Network efforts at Google including datacenters, cloud, and content delivery. It will then move into more detail on the issue of content delivery and how Google builds and operates its CDN ...
Boston Networking @Scale 2016
Open/R: The joy of packet routing
Open/R is our way of building extensible control planes for wide variety of network types: datacenter, backbone and wireless meshes. Starting as a simple routing protocol for the Terragraph mesh network it grew into a framework for ...
Boston Networking @Scale 2016
Introducing Akamai’s Cloud Networking
Akamai is a leading CDN provider with a mission to make Internet fast, reliable, and secure. In addition to it’s traditional CDN offering, Akamai provides many other products to improve all types of Internet services. This talk will ...
Boston Networking @Scale 2016
Browser networking: Hopscotching your way to great performance
Understanding how browsers interact with the world through their network layer can lead to some major improvements in application performance. The days of two connections per host are long gone but performance problems still exist. ...
Boston Networking @Scale 2016
Modeling operational complexity
Understanding the precise impacts that new services have on the cost to operate infrastructure can often be an afterthought in product development and sales cycles. In order to build and expand upon service offerings successfully, it ...
Boston Networking @Scale 2016
Scaling Facebook Live
Facebook Live enables people to share their experiences and perspectives in real time with those who matter to them — whether they’re someone who wants to broadcast to friends and family, or a public figure who wants to ...
Boston Networking @Scale 2016
@Scale network traffic visibility and anomaly detection
Many network operators still lack systems or rely on outdated non-distributed technology from the 90’s to ingest, store, process, and visualize network traffic data. This results in sparse network visibility, delayed data, painfully ...
Boston Networking @Scale 2016
Scaling traffic configurations
A sufficiently complex configuration can rot much like a codebase does. How can this happen, and how can you approach the process of fixing it? This talk explores the evolution of Facebook’s traffic configuration over the years.
Boston Networking @Scale 2016
Securing our pipes
With thousands of services communicating with each other across the globe within the Facebook network, encryption becomes a necessity. The infra should make this transparent to the service owner with minimal impacts to performance ...
Boston Networking @Scale 2016
Terragraph: A gigabit wireless network for dense urban environments
Terragraph is a gigabit wireless network for fiber distribution replacement in dense urban environments. This talk provides a deep dive into the underlying motivations, science, and challenges that come with providing fiber-like speeds ...
Boston Networking @Scale 2016
QUIC @ Google
QUIC is a multiplexed transport protocol running over UDP. It builds on the success of SPDY and HTTP/2 to make the web faster (e.g. improving page load latency, reducing video playback buffering), and to provide a platform for ...
The @Scale Conference 2016
@Scale 2016 Keynote – Jay Parikh, Facebook
Jay Parikh, head of Engineering and Infrastructure at Facebook, kicks off the third annual event with a welcome keynote.
The @Scale Conference 2016
@Scale 2016 Keynote – Himagiri Mukkamala, GE
Himagiri (Hima) Mukkamala is in charge of building the platform that connects software and sensors to machines and the machines to the cloud, collecting as much as 50 million data variables from 10 million sensors. It’s an open, ...
The @Scale Conference 2016
Uber’s data visualization stack
When creating graphics in the browser, there’s no consensus on what language or API should be used. From markup languages like SVG to OpenGL based APIs like WebGL, the browser provides several different ways for creating ...
The @Scale Conference 2016
All apps shall crash. Deal with it.
No matter how smart you are, your app will crash. At Square, our customers trust us to move money, so we take crashes very seriously. This talk presents our systematic approach to ensuring crashes are rare.
The @Scale Conference 2016
DevOps and containerization at scale
Since the split from eBay in 2015 PayPal has turbo charged DevOps. A key example of this in action is adopting Docker as containerization technology to enhance our developer experience, reduce drift in different environments like test ...
The @Scale Conference 2016
360 video stabilization: A new algorithm for smoother 360 video viewing
360 video is rapidly becoming more widespread. There are dozens of devices for capturing 360 video, from professional rigs to consumer handheld cameras, all with different specs and quality outputs. As these types of cameras become ...
The @Scale Conference 2016
Real-time data compression at scale
The year is 1993, and Phil Katz releases its hugely successful compression algorithm, deflate, commonly known as Zip. Thanks to openness and efficiency, served by an excellent industry-grade reference implementation, it is deployed and ...
The @Scale Conference 2016
Shipping mobile apps smart and fast at scale
In the mobile world, developers aspire to get new features out to people as fast as possible. But how do you speed up your development cycle and drastically cut down on stabilization time without sacrificing quality? In this talk, the ...
The @Scale Conference 2016
Addressing continuous deployment’s resource creep at scale
Resource creep is one disadvantage of continuous deployment. Small changes, on average, have smaller incremental resource demands (CPU, memory, network, disk IOPS, etc.). The additional resource demands may seem reasonable or ...
The @Scale Conference 2016
GPUs and deep learning deployments at scale
GPUs are already important compute engines in the HPC top 500, but GPU evolution has also made them a great fit for scale workloads like deep learning and inference. Recent increasing complexity of neural nets has created an ...
The @Scale Conference 2016
Unifying big data workloads in Apache Spark
In contrast to previous big data systems, Apache Spark was designed to offer a unified engine across diverse workloads, such as SQL, streaming, and batch analytics. While this approach may seem counterintuitive, it has some key ...
The @Scale Conference 2016
Building the Quip editor for iOS
Quip is available across eight platforms, all of which share a single HTML5/JavaScript editor. This talk will go over the architecture behind Quip’s editor, its evolution as the company adopted React, and how it connects to its ...
The @Scale Conference 2016
Netflix: Engineering the cloud
Decisions Netflix made early on facilitated later choices. In this talk, we will discuss how decisions have built upon one another and led to a highly resilient service. But the architecture isn’t perfect. We’ll talk about ...
The @Scale Conference 2016
Amazon Aurora: An under-the-hood view of a cloud-scale relational database service
This talk will be a deep-dive into Amazon Aurora — a relational database service reimagined for the cloud. Amazon Aurora is a disruptive technology in the database space, bringing a new architectural model and distributed systems ...
The @Scale Conference 2016
GraphQL in native applications at scale
GraphQL is a query language that is powering a lot of Facebook applications. Mobile developers at Facebook use GraphQL to describe the data they need to render their view. Server developers use GraphQL to expose the server’s data ...
The @Scale Conference 2016
BrowserLab: Fully automated regression detection for the web
Client-side rendering is gaining in popularity which presents new challenges for measuring and optimizing end-to-end performance on the web. BrowserLab provides a controlled, consistent environment for collecting timing metrics which ...
The @Scale Conference 2016
Deep learning in production at Facebook
Facebook is powered by machine learning. From advertising relevance, news feed and search ranking to computer vision, face recognition, and speech recognition, Facebook runs ML models at massive scale, computing trillions of ...
The @Scale Conference 2016
MyRocks: Space and write optimized OLTP database at Facebook
Facebook created and open-sourced a next generation OLTP SQL database on modern Flash storage called MyRocks — a RocksDB storage engine for MySQL. Facebook has used MySQL (InnoDB) for many years, and InnoDB is a great general purpose ...
The @Scale Conference 2016
A practical guide to writing solid UI tests on Android
Developer-driven testing is a great way to scale quality control as your team grows and code churn rises. While most developers are sold on unit testing their code, they are often intimidated by UI testing. Is it worth the effort? Can ...
The @Scale Conference 2016
Blazing fast: Scaling iOS at Uber
In the six years since it began, Uber’s business has definitely grown, and the same can be said for the size of its engineering team. Every week, Uber ships thousands of code changes to millions of users of its mobile apps. What began ...
The @Scale Conference 2016
How VP9 is saving the world
Steve Robertson (YouTube) makes the case for next generation video delivery today, including an explanation of the fundamentals of video compression, a recipe for deployment in your own service, and ways you can help shape VP9’s ...
The @Scale Conference 2016
No shard left behind: APIs for massive parallel efficiency
Apache Beam (incubating) is a unified batch and streaming data processing programming model that is efficient and portable. Beam evolved from a decade of system-building at Google, and Beam pipelines run today on both open source ...
The @Scale Conference 2016
Benchmarking the cloud to build applications that work
In this talk, Google will cover its pursue of a fair and meaningful Cloud benchmarking framework, PerfKit Benchmarker, from one of its performance engineers’ perspective. The talk will cover the challenges and pitfalls the team ...
The @Scale Conference 2016
Creating and scaling Spotify’s Discover Weekly playlist
Discover Weekly is a personalized mixtape of 30 highly personalized songs that’s curated and delivered to Spotify’s 100M active users every Monday. It’s received high acclaim in the press and reached over 5B streams ...
The @Scale Conference 2016
Building with a mobile-first approach in India
Why is mobile big for India? Flipkart will be sharing some of the challenges this throws at e-commerce businesses and how the company transformed itself and its technology to make things happen for its customers. The company will ...
The @Scale Conference 2016
Machine learning at scale: FBLearner Flow
Efficient use of large-scale data for Machine Learning (ML) research is a challenge. Training and distributing hundreds of models, monitoring performances, and sharing algorithms in a production environment requires tools to simplify ...
The @Scale Conference 2016
HTTP2 server push: Lower latencies around the world
With HTTP2 push, Facebook has built out a new client/server interaction model, which now makes it possible for the company’s Edge/FBCDN servers to ‘push’ required images and Live streams from the server for a News ...
The @Scale Conference 2016
Active fault finding in networks
Most network monitoring relies in the individual network devices themselves telling you that they are healthy or unhealthy via syslog messages, SNMP data, etc. In a Facebook scale network we just can’t trust the network devices ...
The @Scale Conference 2016
Dropbox Infinite: A different kind of distributed file system
Project Infinite is one of the biggest evolutions in the Dropbox product since the shared folder, enabling users to see all of the content across their entire team on their local computer without syncing it first (and instead syncing ...
