Moonshot AI has introduced PerceptionBench, a benchmark that evaluates the visual perception of multimodal models independently of logical reasoning and external knowledge. The test assesses basic visual abilities based on real-world errors, with all leading models, including GPT-5.6 Sol, Kimi K3, and Claude Fable 5, showing significant weaknesses. No frontier model reaches 60 percent accuracy, and GPT-5.6 Sol leads by a narrow margin. Many supposed reasoning errors actually happen as early as the image-reading stage.
The authors of PerceptionBench explain that existing benchmarks capture only narrow slices of perception errors. The 42 open-source benchmarks analyzed show little overlap in their error profiles, so each one covers a different subset of visual weaknesses. Instead of defining categories up front, the team built their taxonomy from actual model errors and traced each one back to the earliest failed step in existing benchmarks. The result is ten 'skill domains': Visual Relation, Counting, Attributes, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination.
The researchers argue that many multimodal model failures typically attributed to 'reasoning errors' actually occur at the perception level. When a model botches a multi-step task, the first step, correctly reading the image, has often already gone wrong. PerceptionBench breaks those questions into perception-only sub-questions, making it possible to pinpoint which specific visual ability is failing. The dataset and evaluation code are available on GitHub at MoonshotAI/PerceptionBench.
Source: thedecoder
