In November 2023, Meta introduced the Ray-Ban Meta, AI-powered wearable glasses that aimed to seamlessly integrate multi-modal AI technology into everyday life. We knew that for an AI-powered wearable to be truly useful, it needed to engage with users in their context, sensing what they sensed. But enabling the technology wasn’t enough; to have the kind of usefulness people expect from a device they carry, it needed to be fast and reliable. This article aims to highlight some of the behind-the-scenes stories and challenges the team had to overcome to bring Ray-Ban Meta to life.
The importance of design in wearables
In the development of wearables, design is very important and highly personal. Meta’s partnership with Ray-Ban exemplified the pivotal role design plays in creating devices that users would choose to wear every day. The collaboration aimed to address the challenges associated with shape and weight, two critical factors that influence user comfort and device adoption. Striking the right balance between aesthetics and functionality was essential to create a product that seamlessly integrates into the user’s lifestyle without compromising advanced technological capabilities.
Aesthetics and functionality: Meeting fashion expectations
In the world of wearable technology, aesthetics are crucial. The Ray-Ban Meta needed to appeal to a diverse audience with eclectic fashion and personal appearance preferences. With its sleek and stylish design, it could transition smoothly across different settings—whether a user wore it in the office, while spending time with family, or when lounging at the beach. The product needed to be reliable in different real-world conditions, including in cold weather or rain, or on a sweaty run. Crafting a device that not only performed flawlessly under these conditions but also complemented a wide range of sartorial choices was vital for its widespread acceptance and daily use. The designers worked tirelessly to integrate sophisticated technology into a fashion-forward frame, aligning perfectly with the fashion expectations of today’s consumers.
Weight and balance
The total weight of the Ray-Ban Meta glasses plays a crucial role in user comfort, making the distribution of that weight a primary design consideration. It was imperative that the glasses feel balanced on the user’s face, avoiding pressure points that could cause discomfort during prolonged use. This required strategic placement of certain components within the frames. For instance, heavier elements such as batteries and microcontrollers were deliberately positioned to maintain even weight distribution, preventing the glasses from slipping down the nose or exerting uneven pressure on the ears. Such meticulous considerations ensured that the wearer could enjoy both the utility and the comfort of the device throughout the day.
Miniaturization
To fit within Ray-Ban Meta’s sleek frames and provide unobtrusive wearability, the hardware required miniaturization. Essential components such as cameras, processing units, wireless chips, batteries, speakers, and microphones had to be minutely reduced in size. Each element was designed with precision to maintain the balance between minimal weight and optimal power usage, ensuring comfort without sacrificing performance.
While miniaturization allowed for the sleek, seamless integration of technology in the Ray-Ban Meta, it also brought new constraints. With smaller components—particularly batteries—the device faced challenges in maintaining adequate power usage and supply. The reduction in battery size inevitably translated into less power availability, emphasizing the necessity for efficient-energy management.
Thermal considerations
Additionally, the compact design necessitated considerations regarding heat dissipation. Smaller devices can be subject to increased temperatures, potentially impacting performance and safety. The smaller surface area in reduced-sized devices poses challenges for heat dissipation, especially in high-performance modes or under strenuous usage conditions. This can lead to overheating issues that might compromise performance or even safety. When external temperatures rise, as in hot climates, the ambient heat exacerbates these challenges by reducing heat dissipation capabilities, making thermal management even more crucial. To combat such issues, smart hardware and software design is vital. This includes ensuring the device controls and dynamically adjusts the system load in response to external conditions, making strategic use of thermal sensors and algorithms to prevent overheating and to maintain efficient operation. By doing so, the Ray-Ban Meta ensures that it performs safely and effectively, regardless of environmental conditions.
For example, when the device temperature reaches temperatures that are too high, and the user tries to perform a power-hungry action (such as taking a video or importing media), we can either slow down the operation, by underclocking components, or prevent the user from performing the action (while informing them of why).
Managing peak power draw
Another critical aspect of developing the Ray-Ban Meta was managing peak power draw. Miniaturized batteries affect not only battery life but also active power draw. Pulling too much power from the battery at once can lead to a brownout, where the device temporarily loses power or functionality. Therefore, it’s crucial to consider peak power usage for all supported scenarios to ensure reliability and performance. Thus, we selected hardware based not just on total power consumption but also on effectively managing the power draw. This sometimes requires underclocking components to allow multiple parts of the device to function concurrently without overloading the system.
Ambient temperature also impacts our ability to pull power from the battery, as the temperature changes the battery chemistry and reduces the peak power we can draw. As anyone who lives in a cold weather area knows, batteries don’t work as well in freezing temperatures, possibly causing your car not to start or your electric vehicle getting a lower-than-usual mileage range.
So we have to monitor ambient and device temperatures, not only for safety considerations but also for power-draw considerations.
System architecture: Designing for speed, reliability, and privacy
We designed Ray-Ban Meta’s system architecture with the goals of speed, reliability, privacy, and advanced AI capabilities. This architecture is composed of three main components:
- On-frame components: Ray-Ban Meta frames are equipped to handle raw input and perform on-device computation. This is because we’ve integrated the necessary sensors, compute power, and AI processing capabilities directly into the frames, which allows the Ray-Ban Meta to capture and process data swiftly and reliably as well as support user privacy by reducing the need to send data externally. The frames include embedded microphones, cameras, and processing units, ensuring that AI computation can occur in real time, in line with the user’s immediate surroundings.
- Meta AI app: The Meta AI app, installed on the user’s smartphone, plays a pivotal role in providing connectivity and integration with communication and third-party applications. It serves as a bridge between the on-frame components and Meta’s wider network, facilitating seamless integration and communication across platforms, enabling enhanced connectivity and functionality.
- Meta servers: These servers host advanced AI functionalities that require more intensive computation than the on-frame or on-phone processes can handle. The Meta servers ensure that the most sophisticated AI tasks are executed efficiently, further improving the Ray-Ban Meta’s ability to deliver advanced AI capabilities in real time.
On-frame compute
On-frame compute in the Ray-Ban Meta is supported by a dual-system mechanism:
- A low-power microcontroller dedicated to handling the wake word.
- A more advanced system on a chip (SoC) tasked with essential compute and acceleration needs for various machine learning models. These models include, but are not limited to, speech to text (ASR), natural language understanding (NLU), text to speech (TTS), and optical character recognition (OCR).
While on-device computation offers fast AI responses and high reliability, thus providing significant privacy benefits by minimizing data transmission, it also presents unique challenges. These include power and hardware constraints, requiring innovative engineering solutions to maintain performance without compromising the device’s wearability or operational efficiency.
The Meta-AI app
The Meta AI app goes beyond simple connectivity; it also manages interactions with sensitive information such as users’ contacts, messages, and media. It offers enhanced functionalities, including sending encrypted WhatsApp messages through voice commands and supporting user privacy by keeping the content within the phone. The app can read incoming message notifications, allowing users to remain updated without having to take out their phone. Additionally, it links the Meta Ray-Ban glasses with third-party apps, including Spotify, Apple Music, Amazon Music, Shazam, and Be My Eyes, facilitating a seamless integration into users’ digital lifestyles and expanding the device’s utility.
The role of Meta servers
The Meta servers play a crucial role in Ray-Ban Meta’s system architecture by providing advanced multi-modal AI capabilities that are seamlessly connected with various plugins, including search, user memory, and agents. This advanced setup enables users to perform a range of tasks that greatly enhance their interaction with Meta AI. The capabilities facilitated by the Meta Servers allow users to:
- Identify objects: Users can point the Ray-Ban Meta glasses at various objects to identify them instantaneously, receiving information about their surroundings in real time.
- Understand scenes: Beyond mere object recognition, the glasses can provide context about the user’s environment that delivers intelligent feedback and assistance.
- Access real-time information: Users can obtain immediate access to current sports scores, news updates, weather conditions, and more, allowing them to stay informed without needing to look away from their immediate activities.
- Provide long-term memory: Meta AI can recall past user interactions, preferences, and frequently accessed information, thus enhancing the personalization and usefulness of the wearable.
By integrating these functionalities through the Meta servers, the Ray-Ban Meta is equipped to offer a more context-aware, intelligent, and seamless user experience, bridging the gap between AI’s potential and the needs of the modern world.
The need for speed
When users think about something they want accomplished—whether taking a photo of a special moment, sending a message to their mom, or looking up a piece of trivia—they usually want it to happen right away.. While a product can’t read minds, it can try to ensure it responds as quickly as possible after the user has informed it what they want. Unfortunately, when there are many steps in that process, the response may be delayed. On our mission for speed, we’ve had to optimize every part of the Ray-Ban Meta system and squeeze every few 10s of milliseconds from our pipeline, but even then it’s often not enough.
Meta implemented several technical optimizations to enhance the Ray-Ban Meta’s speed, making it an AI that processes information and acts in sync with the user’s experiences.
User-perceived latency
With speed in mind, one way we’ve improved that user experience is by understanding how a user perceives interaction latency. Typically we measure latency from the end of the user’s speech to when the action happens. We use multiple techniques to minimize this perceived latency.
- Speculative processing: This involves predicting the most likely actions the user might want to take based on partial data inputs. For instance, if a user starts saying “take a photo,” the system can preemptively prepare to capture an image.
- Parallelization: Speculative processing enables us to parallelize expensive operations. For example we can start sending data to the phone while still processing other requests. Or if we detect that we might need to send this photo for AI usage, we might start doing OCR or resizing the image.
- Leveraging accelerators: Utilizing accelerators for specific workloads, such as machine learning, enhances performance and efficiency by handling computationally intensive tasks faster. For example, some types of workloads are better on DSPs, some on HTPs, and others on GPUs.
Trade-offs in optimizations
As advanced as the Ray-Ban Meta is, realizing its full potential involves balancing numerous optimization trade-offs. These challenges predominantly revolve around:
- Parallelization: While parallelizing tasks can achieve greater efficiency, it also poses risks, such as excessive power consumption. Balancing workloads becomes critical to preventing secondary operations from interfering with primary needs. For example, if speech recognition requires the bulk of CPU resources to process swiftly, simultaneously running image processing may lead to performance degradation.
- Model-based limitations: The speculative-model approach, although effective, is not foolproof. The system may falter if a user articulates a command in an unexpected manner or the predictive model inaccurately anticipates user intent. Despite extensive training, these systems occasionally mispredict, leading to delayed or incorrect outputs.
Collaboration and innovation
The development of Ray-Ban Meta necessitated collaborative efforts among diverse engineering teams within Meta, including hardware engineers, software developers, and AI specialists. Their combined expertise ensured that AI was not a distant, abstract idea but a close companion that shared in the user’s daily experiences. This project not only showcased Meta’s technological prowess but also served as a beacon for attracting new talent eager to work on cutting-edge AI projects .
Looking toward the future
Meta plans to expand the capabilities of Ray-Ban Meta, introducing new features and supporting additional languages. This will further our goal of having AI accompany users wherever they go, accommodating different cultural and linguistic contexts in real time. The integration of more complex agents will enhance functionality, but it will also lead to new performance and reliability challenges—problems the team is looking forward to solving.