NVIDIA Revolutionizes Autonomous Vehicle Development with Cosmos Predict-2
The autonomous vehicle (AV) landscape is undergoing a significant transformation, shifting from multiple distinct models to a unified, end-to-end architecture that executes driving actions directly from sensor data. To support this transition, high-quality, physically based sensor data is in high demand for training, testing, and validation.
Introducing Cosmos Predict-2
NVIDIA has released Cosmos Predict-2, a groundbreaking world foundation model that enhances future world state prediction capabilities for high-quality synthetic data generation. This innovation is part of the NVIDIA Cosmos platform, which equips developers with the tools to tackle complex challenges in end-to-end AV development. Industry leaders such as Oxa, Plus, and Uber are already leveraging Cosmos models to rapidly scale synthetic data generation for AV development.
Accelerating AV Training with Cosmos Predict-2
Building on the success of Cosmos Predict-1, which predicted and generated future world states using text, image, and video prompts, Cosmos Predict-2 demonstrates a deeper understanding of context from text and visual inputs. This results in fewer hallucinations and richer details in generated videos. The latest optimization techniques have also significantly sped up synthetic data generation on NVIDIA GB200 NVL72 systems and NVIDIA DGX Cloud.
Unlocking New Training Data Sources with Post-Training Cosmos
By post-training Cosmos models on AV data, developers can generate videos that accurately match existing physical environments and vehicle trajectories. This capability also enables the generation of multi-view videos from single-view videos, such as dashcam footage, providing access to new troves of data for AV training. The NVIDIA Research team has successfully post-trained Cosmos models on 20,000 hours of real-world driving data, improving model performance in challenging conditions like fog and rain.
AV Ecosystem Adoption and Advancements
AV companies have already integrated Cosmos Predict to accelerate vehicle development. Autonomous trucking leader Plus is post-training Cosmos Predict on trucking data to generate highly realistic synthetic driving scenarios, while AV software company Oxa is using Cosmos Predict to support the generation of multi-camera videos with high fidelity and temporal consistency.
New NVIDIA Models and NIM Microservices
In addition to Cosmos Predict-2, NVIDIA has announced Cosmos Transfer as an NVIDIA NIM microservice preview for easy deployment on data center GPUs. The Cosmos Transfer NIM microservice preview augments datasets and generates photorealistic videos using structured input or ground-truth simulations from the NVIDIA Omniverse platform. The NuRec Fixer model helps inpaint and resolve gaps in reconstructed AV data.
CARLA Integration and Open-Source Data
CARLA, the world’s leading open-source AV simulator, will integrate Cosmos Transfer and NVIDIA NuRec into its latest release, enabling its user base of over 150,000 AV developers to render synthetic simulation scenes and viewpoints with high fidelity. Developers can try out this pipeline using open-source data available on the NVIDIA Physical AI Dataset.
Advancing End-to-End AV Safety with NVIDIA Halos
To bolster the operational safety of AV systems, NVIDIA has introduced NVIDIA Halos, a comprehensive safety platform that integrates the company’s full automotive hardware and software safety stack with state-of-the-art AI research focused on AV safety. Bosch, Easyrain, and Nuro have joined the NVIDIA Halos AI Systems Inspection Lab to verify the safe integration of their products with NVIDIA technologies and advance AV safety.
Explore More
Watch the NVIDIA GTC Paris keynote from NVIDIA founder and CEO Jensen Huang at VivaTech, and explore GTC Paris sessions.
