What changed
Hugging Face announced the release of Mundo, a model that converts photos of a physical space into a 3D world users can explore and simulate. The model allows users to interact with objects, such as opening drawers or moving items, and generates video output of these interactions. Mundo is designed to reconstruct scenes from multi-view images or videos and represent them as a 3D proxy.
What this means in practice
Mundo works by first reconstructing the scene from input images, then representing the layout and objects as a 3D proxy. A code world maintains and updates this proxy state during simulation. The simulated state is rendered into proxy frames, which guide a video model in generating the scene's appearance and motion. Users can explore the environment from new viewpoints, modify objects, and simulate robot tasks within it. The model uses a combination of 3D detectors, vision-language models, and numerical optimization to refine the proxy world, ensuring alignment with the input images.
Limitations and availability
The supplied announcement does not specify the exact hardware requirements or licensing conditions for Mundo. The model is available for use through the project page, where users can try interactions. Mundo is designed to work with MuJoCo for simulations, though other engines or custom scripts could also be used. The model's performance and capabilities are based on the input images and the accuracy of the 3D reconstruction. The supplied announcement does not specify the total or active parameter counts for Mundo.
