In my talk, I shared how AI voice translations make content on Facebook and Instagram accessible across languages, so creators can reach new audiences in their own voice. Over half a billion people on each app now watch AI-translated videos every week. Delivering that depends on high-quality speech translation, and my colleagues on Meta’s speech research team are pushing what’s possible. Two of their papers go deeper on the audio generation ideas I covered:
• Transcribing and Translating, Fast and Slow: Joint Speech Translation and Recognition (Moritz et al.): introduces JSTAR, a streaming model that transcribes and translates speech at the same time using fast and slow encoders. It was applied to bilingual conversations on smart glasses, where it can distinguish the wearer from the person they’re talking to, and it performs better than a cascaded baseline on both translation quality and latency.
• Textless Streaming Speech-to-Speech Translation using Semantic Speech Tokens (Zhao et al.): translates speech directly into speech, skipping the usual chain of transcribe → translate → text-to-speech. Removing those steps cuts latency and the errors that build up between them, and it reaches state-of-the-art streaming results across multiple language pairs.
