U.S. military officials discovered that an AI chatbot had hallucinated critical intelligence, leading to the near-activation of a military operation against a Chinese vessel. The operation was halted just in time, averting a potential conflict with China, CNN reported.
The intelligence report, which circulated during the war with Iran, falsely identified the vessel as carrying components for a nuclear weapons program.
The false information originated with a Special Operations Command analyst who used an AI chatbot to synthesize open-source data with classified signals intelligence.
The chatbot misidentified the ship’s cargo manifest, and the analyst then used the tool a second time to format the erroneous findings into an official-looking summary, which was circulated across command channels.
The incident highlights the risks of relying on AI for critical decision-making in high-stakes environments.
Jake Steckler, a research scholar at GovAI and former U.S. Army officer, emphasized the importance of understanding the uncertainty inherent to large language models (LLMs) in such contexts.
"It’s important for service members to understand the uncertainty inherent to LLMs," said Steckler. "But it’s especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning.
There are life and death consequences for those decisions." Steckler argued that the incident should serve as a call to add more safeguards to AI, not a reason to avoid it.
The near-miss comes as the U.S. military races to integrate AI to accelerate decision-making and maintain its edge over China. The Pentagon has described AI as delivering a significant advantage in speeding up its kill chain so commanders can respond in the right time. However, the same speed that makes AI attractive may also allow hallucinations with insufficient human oversight.
The U.S. military did not specify how the AI hallucination occurred or what steps are being taken to prevent similar incidents. The incident raises questions about the reliability of AI in military operations and the need for additional safeguards. The military is now considering how to better integrate AI while ensuring human oversight.
Source: techcrunch
