A new benchmark test, RadLE 2.0, evaluates AI chatbots' ability to read X-rays and recognize when they are uncertain. The test, developed by the CRASH Lab at Ashoka University, measures accuracy, confidence, and the ability to admit when a model is unsure. Human radiologists scored 988.7 out of 2,000 points, while the best AI model reached 758. The test involved 200 cases across 16 models and compared them against a panel of radiologists. The scoring system rewards honesty and penalizes overconfidence, with models that guess confidently dropping in rankings even if their raw hit rate looks decent. The study highlights a critical issue: confident misdiagnoses in medicine are far more dangerous than honest admissions of uncertainty. | Image: Crash Lab
The test reveals that no single model performs best across all metrics. Anthropic's Claude Fable 5 led in reliable and safe answers, while Google's Gemini 3 Pro had the highest raw accuracy. Meta's Muse Spark 1.1 excelled at knowing when to hand a case off to a human, having recently reduced its hallucination rate by nearly half. In contrast, models like Grok 4.5 show increased hallucination due to higher confidence in wrong answers. The handover index, which measures a model's ability to recognize when it should pass a case to a human, is crucial for patient safety. | Image: Crash Lab
According to the research team, several models could have scored better by staying quiet more often instead of guessing. This was especially true for open-weight models and those trained specifically for medical use. These models attempted nearly every case but were often wrong, usually with medium to high confidence. The study warns that confidence levels in top commercial models are not reliable indicators of accuracy, as they frequently produce highly confident misdiagnoses. | Image: Crash Lab
Source: thedecoder
