The US military narrowly avoided boarding a Chinese ship after an AI-generated intelligence report falsely claimed it was carrying nuclear arms components through the Middle East.

The erroneous report, created by a US Special Operations Command analyst, led to preparations for an interception with air support, according to CNN. Officials later discovered the AI chatbot used in generating the report had 'inaccurately identified the material the ship was carrying.'

The report was based on a chatbot that 'fused together open-source intelligence with secret signals intelligence in government holdings,' according to CNN. This information was packaged into an intelligence report that almost set off a disastrous chain of events. The incident highlights the risks of AI hallucination in critical decision-making processes.

Since 'hallucinating' became the Cambridge Dictionary’s word of the year in 2023, numerous cases have emerged of AI tools fabricating information. Despite attempts at 'do not hallucinate' prompts, researchers suggest it may be impossible to prevent large language models from hallucinating altogether. The incident underscores the need for caution when relying on AI for intelligence analysis.

The US Department of Defense has been actively integrating AI into its operations, including the use of Google’s Gemini for Government and Anthropic’s Claude for spy work. In June, a Pentagon representative told Congress that generative AI is used to create congressionally mandated reports, with 1.5 million active DoD personnel using the tools.

The incident comes amid growing concerns over AI safety, including calls for regulation and coordinated research from leading AI labs. The Department of Defense has also faced criticism for its use of AI in autonomous weapons systems, with Anthropic recently blacklisted over its opposition to such applications.

The Department of Defense did not comment on how it will address the risks of AI hallucination in intelligence operations, and the incident raises questions about the reliability of AI in high-stakes scenarios. The source notes that the incident underscores the ongoing challenges of balancing AI capabilities with the need for human oversight.

Source: arstechnica