What changed
NVIDIA has introduced Cosmos3-H-Dreams, a generative simulation model that brings interactive real-time surgical simulation to clinical procedures. Built on Cosmos-Predict2.5, this model is trained on video and robot kinematics data from clinical cholecystectomy procedures using the CMR Surgical Versius platform. It is powered by Cosmos 3 and accelerated with FlashDreams, allowing it to run interactively on a single NVIDIA RTX PRO 6000 GPU.
What this means in practice
Cosmos3-H-Dreams enables the simulation of complex surgical scenarios with high visual realism, capturing tissue deformation, reflections, and occlusion in real time. The model is specialized for clinical cholecystectomy and distilled for real-time generation, ensuring that instrument actions and anatomical responses remain consistent. It uses 28 action dimensions, including pose and gripper information for both arms, energy controls, and thumbstick inputs. The model includes four-step and two-step checkpoints, allowing developers to balance visual quality and inference cost. It also supports streaming through FlashDreams, with an adapted FadeMem module to manage attention history during long rollouts.
Limitations and availability
The model is currently available for cholecystectomy procedures and can be adapted to other datasets with synchronized video and compatible action representations. Developers can use a step-by-step guide to train a real-time student model on their own dataset. Controller integrations must use the same action conventions and normalization as training. The supplied announcement does not specify the exact release date or availability for other surgical procedures.
