NVIDIA open-sources four Cosmos 3 world models for robot and driving simulation

On November 5, NVIDIA published four open-weight checkpoints in its Cosmos 3 family on the Hugging Face blog, all post-trained from Cosmos 3 Nano. Rather than a single demo reel, the release is a downloadable set of interactive world-simulation models, shipped with a technical report, inference notebooks and the full post-training recipe.

The checkpoints split the work. Cosmos3-Nano-Transfer-Auto generates offline synthetic driving data, producing synchronized video across up to 11 camera views including fisheyes and supporting joint RGB-LiDAR generation. Cosmos3-Nano-Sim-Auto is its causally distilled streaming counterpart, emitting the next chunk of camera and LiDAR observations in a few denoising steps so a simulation runtime can close the loop over long rollouts.

The other two target simulation and robotics. Cosmos3-Nano-Sim-Transfer turns a physics simulator's spatial controls — depth, segmentation, edges and blur — plus a text prompt into photorealistic RGB that a policy can consume, while physics and ground-truth state stay with the simulator. Cosmos3-Nano-Sim-Bimanual predicts the next observation directly from a two-armed robot's actions, supporting the YAM and AgiBot embodiments and driven by a policy or human joysticks.

According to the post, all four models carry 16B trainable parameters and use the OpenMDW 1.1 license, which permits commercial and non-commercial use under its terms. Inference requires Linux, an NVIDIA GPU, Docker and the NVIDIA Container Toolkit; the published vLLM-Omni setup was tested on a single GB200, which the authors note is not a minimum-memory specification.

Under the hood, Cosmos 3 uses a Mixture-of-Transformers backbone that links a text-processing network and a visual-generation network through attention. The generator predicts each short segment from the prompt, available controls and generated history; temporal-causal attention reads history while blocking future target observations, and a key/value cache carries attention state into the next prediction. Each latent time step covers four RGB frames.

The four-stage recipe is the more interesting artifact: bidirectional adaptation on complete recorded clips, teacher forcing on chunk prediction, consistency distillation into a student that generates each chunk in four denoising steps, then self-forcing on the student's own generated history. The post is blunt about why the final stage matters — a slightly mispredicted object becomes context for later frames, so errors compound.

NVIDIA frames the weights as a shared starting point rather than a finished answer, and the post is candid that hard problems remain: measuring where the models faithfully support policy learning and evaluation, understanding their limits, and improving where they fall short. For robotics and autonomous-driving teams, the near-term questions are what it costs to post-train these checkpoints on their own sensor rigs, and whether the streaming variants can sustain real closed-loop interaction rates.