A new paper argues that current world models, such as Sora, Genie 3, JEPA, and Marble, fail to account for human mental states, which significantly impact behavior. These systems only model the physical world, ignoring beliefs, intentions, and social norms. The authors claim this omission leads to incorrect action predictions, especially in scenarios involving human interaction.
To address this, the researchers developed 'Mental World Modeling' (MWM), a framework that combines physical and mental states. The framework, published on GitHub, introduces mental variables like beliefs, goals, and emotions. It simulates how actions affect both physical and mental states, providing a first-person view to the target agent. The team emphasized that MWM does not simulate consciousness but rather uses behavior and context to form hypotheses about mental states.
The researchers tested their framework using MENTIS, a modular pipeline that evaluates action predictions based on physical plausibility, mental consistency, and social appropriateness. They created Menti-Bench, a dataset of 448 decision scenes, including text descriptions, picture stories, and sound-video clips. The dataset includes human-generated reference solutions that document both correct actions and underlying mental and physical states.
Source: thedecoder
