What changed
Researchers discovered that sensitive data entrusted to AI services is leaking across the supply chain. By planting 2 million digital tripwires in prompts sent through AI services, they found that secrets were exposed at multiple points in the network. The findings revealed security breaches, prompt theft, and widespread failures to protect customer data.
What this means in practice
The experiment involved sending prompts with unique, trackable secrets to AI providers and monitoring where those secrets surfaced. The results showed that at least three AI providers were compromised, with attackers reusing AWS keys and accessing S3 buckets, secrets managers, and parameter stores. Resellers in the inference marketplace also logged prompts and passed them along, often offering low prices to lure users. Some providers were found to be scraping secret URLs from prompts, violating privacy terms. A Chinese AI lab was identified as scraping URLs from over half of the prompts sent to its models, including those marked as 'no data collection' or 'Zero Data Retention'.
Limitations and availability
The study highlights that prompts are encrypted in transit but decrypted at each hop in the inference reseller network, increasing the risk of exposure. The most effective way to prevent leakage, according to the researcher, is to host models on-premises, though this is expensive. For those who cannot afford it, using a first-tier provider with a clear privacy policy is recommended. However, there is no guarantee that data will not leak. The findings suggest that the AI ecosystem lacks robust privacy and security measures, making it difficult for users to ensure their secrets remain private.
