NVIDIA opens Cosmos to outside models, ships NIM containers for world models

NVIDIA is cracking Cosmos wide open. In a developer blog posted Monday, Nov. 2, the chip giant laid out Cosmos Framework — an open, modular sandbox for training, tuning and running world models — plus Cosmos NIM, a containerized package meant to push those models into production without the usual deployment headaches.

The framework is no longer a Cosmos-only club. According to the post, it now reaches beyond Cosmos 3 to support NVIDIA Alpamayo, GR00T, Qwen, pi0.5 and other Hugging Face-compatible models. Training accepts text, images, video, audio and actions, with LoRA for light-touch adaptation or full supervised fine-tuning when developers want to move every parameter. Data can arrive as JSONL, WebDataset, LeRobot or custom adapters, and runs span multiple GPUs with checkpoints saved along the way.

Reinforcement learning gets both online and offline lanes, covering DPO, GRPO, PPO, OPD and DiffusionNFT. Teams can split rollout, reward and training roles across shared GPUs or separate pools, then wire in supported simulators to harvest experience. The recipe book leans on Cosmos 3 Nano checkpoints: Cosmos3-Nano-Transfer-Auto for offline synthetic driving data, Cosmos3-Nano-Sim-Auto for closed-loop scenarios that update as a policy or human drives, and robotics entries like Cosmos3-Nano-Sim-Transfer and Cosmos3-Nano-Sim-Bimanual for two-armed simulation. Surgical work gets Cosmos3-H-Embodied-Nano and the distilled, real-time Cosmos3-H-Dreams.

The simulation numbers are chunky. NVIDIA says Transfer-Auto handles 30 fps video in and out, up to 400 frames at 480p or 720p, with autoregressive chunking for longer clips. A reference run spits out seven 720p videos — 297 frames at 30 FPS — plus 99 LiDAR sweeps.

On the deployment side, Cosmos NIM bundles supported models into optimized containers and leans on quantization, speculative decoding, caching, token pruning and KV-cache-aware routing. Speculative decoding measured mean speedups of roughly 2 to 3 times on H200, per the post, while streaming VLM inference can drive sub-second alerts with 10 times lower latency. Memory demands vary sharply: Cosmos3-Edge needs 16 GiB in BF16 for reasoning, Cosmos3-Nano 23.1 GiB, and Cosmos3-Super 135 GiB in BF16 versus 67 GiB in FP8. For generation, Cosmos3-Nano wants 58 GiB in BF16, 44 GiB in FP8, or 31 GiB with FP8 plus offloading.

NVIDIA says the certified NIM builds run on more than 20 of its GPUs, with automatic configuration selection, validated performance and quality, licensing support, and quarterly feature, optimization and security updates. Developers who want a walkthrough can catch a livestream on Thursday, Nov. 5.