Researchers discovered that 120 misconfigured llms.txt files across 6,214 live domains, including Fortune 500 companies, were directing AI agents to unregistered packages and domains. These files, used to provide machine-readable summaries of website content, were found to contain instructions for installing non-existent packages or visiting unclaimed domain names. The researchers registered some of the unclaimed names and hosted malicious packages, resulting in a phone-home response from a Fortune 500 company within an hour. Over time, they received responses from several more companies, including some Fortune 500 firms and startups. The researchers also identified the chain of parent processes leading to the installations, revealing that coding agents like Claude, OpenAI’s Codex, and Nous Research’s Hermes were involved. Source: arstechnica
The files were misconfigured because they listed non-existent packages from PyPI, npm, and other registries, along with installation instructions. For example, one file contained the prompt 'Installation: pip install [redacted at researchers’ request].' Another included 'npm install [redacted].' Because the package names were unregistered, an attacker could register one and host ransomware or other harmful packages. The vulnerability arises when a coding agent with permission to run shell commands treats the file as authoritative setup documentation. Some AI agents will then download and execute the package. In other cases, the LLM files pointed to non-existent domain names, allowing attackers to register the site and plant malicious instructions. The researchers demonstrated this by registering a domain and hosting live malware, which was later discovered by Clerk.com. Source: arstechnica
The researchers found that the 120 misconfigured files contained 227 commands to install non-existent packages or view unclaimed domains. It is unclear how these faulty entries got there, but some predate the AI era and were manually generated by humans. Others may have been created by AI that either hallucinated or couldn’t distinguish between legitimate and illegitimate instructions. The researchers suspect that the collapsing boundary between data and code is a fundamental limitation of LLMs, which cannot reliably differentiate between user instructions and content from untrusted sources. This leads to prompt injections and makes the entire corpus of published data an execution surface. Source: arstechnica
