Google’s Genie 3 world model has thus far been used for creating video game scenarios and wondering if we’re ourselves living in a simulation, but it seems to have some very practical use-cases as well.
Waymo has unveiled the Waymo World Model—a specialized adaptation of Google DeepMind’s Genie 3 that simulates edge-cases to train Waymo’s self-driving cars. “By simulating the “impossible”, we proactively prepare the Waymo Driver for some of the most rare and complex scenarios—from tornadoes to planes landing on freeways—long before it encounters them in the real world,” Waymo said on X.
Waymo World Model
The Waymo World Model represents a convergence of Google’s diverse technological initiatives. By building upon Genie 3, DeepMind’s general-purpose world model for generating photorealistic and interactive 3D environments, Waymo has created a system capable of simulating scenarios that would be nearly impossible to capture through traditional fleet data collection alone. While the Waymo Driver has logged nearly 200 million fully autonomous miles on public roads, the vehicle has traveled billions more in virtual environments. The new model takes this simulation capability to unprecedented levels, generating hyper-realistic scenarios across both camera and lidar sensor modalities—a critical distinction from purely visual simulation systems.
Simulating the Impossible
The system’s most compelling capability lies in its ability to conjure rare, potentially dangerous scenarios that autonomous vehicles might encounter only once in millions of miles—or perhaps never at all during typical testing. Waymo demonstrated simulations ranging from tornadoes and flooding to encounters with elephants, lions, and even pedestrians dressed as dinosaurs.
These aren’t merely visual curiosities. The model can simulate safety-critical events like vehicles driving the wrong way, reckless drivers veering off-road, and precariously loaded trucks—scenarios that would be unethical or impossible to stage in real-world testing but essential for comprehensive safety validation.
Perhaps most impressively, the system inherits Genie 3’s vast world knowledge from its pre-training on an extensive video dataset, then transfers that understanding into the 3D lidar outputs specific to Waymo’s hardware suite. This allows the model to generate realistic physics and visual details for situations the Waymo fleet has never directly observed.
Three Dimensions of Control
The Waymo World Model offers engineers three primary mechanisms for crafting specific test scenarios. Driving action control enables “what if” counterfactual simulations, allowing engineers to test whether the Waymo Driver could have navigated a situation more confidently or efficiently. Unlike purely reconstructive methods that break down when simulated routes diverge from recorded drives, the generative approach maintains consistency across novel trajectories. Scene layout control, meanwhile, permits customization of road configurations, traffic signal states, and the behavior of other road users, enabling the creation of bespoke test scenarios through strategic placement of vehicles and pedestrians. Language control provides the most flexible interface, allowing engineers to adjust environmental conditions like time of day and weather, or generate entirely synthetic scenarios using simple text prompts.
Bridging Google’s Ecosystem
The collaboration between DeepMind and Waymo exemplifies how Google’s organizational structure can yield unexpected benefits when different divisions combine their expertise. DeepMind’s fundamental research in world modeling and generative AI has found immediate application in one of the most challenging real-world AI problems: making autonomous vehicles safe enough for widespread deployment.
The system can even convert standard dashcam footage or mobile phone videos into multimodal simulations, showing how the Waymo Driver would perceive and navigate through scenes captured in locations like Norway’s scenic roads or Utah’s Arches National Park. This capability creates a bridge between real-world observations and controlled testing environments.
Scaling for the Long Haul
Recognizing that some scenarios require extended simulation time, Waymo has developed an efficient variant of the model capable of maintaining quality over longer rollouts while dramatically reducing computational requirements. This efficiency enables the large-scale simulation necessary for comprehensive safety testing across Waymo’s expanding service areas.
The Waymo World Model is a glimpse into how different branches of a tech giant’s AI research can converge to address tangible safety and scaling challenges. As Waymo continues expanding its autonomous ride-hailing services across U.S. cities, the ability to proactively prepare for rare edge cases through impossibly realistic simulation may prove as crucial as the millions of real-world miles already logged. And having access to a top tier AI lab could stand Waymo in good stead as it prepares to battle Tesla in the self-driving space in the coming years.