Inside Waymo’s New Foundation Model Powering Self-Driving Cars [Drago Anguelov] - 725

Today, we're joined by Drago Anguelov, head of AI foundations at Waymo, for a deep dive into the role of foundation models in autonomous driving. Drago shares how Waymo is leveraging large-scale machine learning, including vision-language models and generative AI techniques to improve perception, planning, and simulation for its self-driving vehicles. The conversation explores the evolution of Waymo’s research stack, their custom “Waymo Foundation Model,” and how they’re incorporating multimodal sensor data like lidar, radar, and camera into advanced AI systems. Drago also discusses how Waymo ensures safety at scale with rigorous validation frameworks, predictive world models, and realistic simulation environments. Finally, we touch on the challenges of generalization across cities, freeway driving, end-to-end learning vs. modular architectures, and the future of AV testing through ML-powered simulation. 🎧 / 🎥 Listen or watch the full episode on our page: https://twimlai.com/go/725. 🔔 Subscribe to our channel for more great content just like this: https://youtube.com/twimlai?sub_confirmation=1 🗣️ CONNECT WITH US! =============================== Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/ Follow us on Twitter: https://twitter.com/twimlai Follow us on LinkedIn: https://www.linkedin.com/company/twimlai/ Join our Slack Community: https://twimlai.com/community/ Subscribe to our newsletter: https://twimlai.com/newsletter/ Want to get in touch? Send us a message: https://twimlai.com/contact/ 📖 CHAPTERS =============================== 00:00 - Introduction 3:56 - Waymo One 7:03 - Safety Impact 12:10 - VLMs and generative AI technologies 16:15 - EMMA 19:17 - VLMs 24:00 - Challenges 29:37 - Foundation model 30:52 - Sensor data integration and foundation models 36:18 - Relationship of sensor prediction and vehicle behavior 42:28 - End-to-end training vs. modular architectures 45:58 - Simulation, testing, and validation 49:22 - Explainability and controllability 52:25 - Testing process 55:20 - Challenges in defining rules for autonomous vehicles 58:57 - Simulation technologies 1:04:10 - Waymo Open Dataset Challenges 🔗 LINKS & RESOURCES =============================== Waymo - https://waymo.com/ Waymo Safety Impact - https://waymo.com/safety/impact/ NVIDIA GTC Keynote: Advancing AI to Build the Most-Trusted Driver - https://register.nvidia.com/flow/nvidia/gtcs25/vap/page/vsessioncatalog/session/1727887107684001i7zG System Design for Autonomous Vehicles with Drago Angelov - 454 - https://twimlai.com/podcast/twimlai/system-design-autonomous-vehicles-drago-angelov/ 📸 Camera: https://amzn.to/3TQ3zsg 🎙️Microphone: https://amzn.to/3t5zXeV 🚦Lights: https://amzn.to/3TQlX49 🎛️ Audio Interface: https://amzn.to/3TVFAIq 🎚️ Stream Deck: https://amzn.to/3zzm7F5