Researchers at IIT Bombay and Adobe Research have developed a method to reverse-engineer prompts used with large language models (LLMs) by analyzing only the output text. The technique, called 'Previous-Token Prediction' (PTP), reconstructs the original prompt with high accuracy, even without access to the model's internal weights. This approach could have significant implications for security and privacy, as it allows for the extraction of sensitive information from LLM responses. The method works by training an inverse language model that predicts the previous tokens instead of the next ones, using synthetically generated data from the target LLM. A single response can yield the exact prompt and multiple semantically similar variations. The reconstructed prompts, when fed back into the original model, produce responses that closely match the original output. This demonstrates the potential for recovering sensitive or proprietary prompts from LLM-generated text. | Image: Suhail et al.

Tests with real user prompts showed accurate reconstructions as well. When the researchers fed the reconstructed prompts back into the forward model, the responses closely matched the originals. For real user prompts, the reconstructed phrasing differs in wording but captures the semantic core of the original input. | Image: Suhail et al. Attackers don't even need to know which model produced the text. An inverse model trained on the small Qwen-3-0.6B chatbot was also able to reconstruct prompts from GPT-4o's responses. The reconstructed prompts weren't identical to the originals, but according to the paper, they captured the meaning and intent. A potential attacker wouldn't even need to know which model generated a given output. This creates a broad security problem. Companies risk exposing proprietary system prompts containing trade secrets, moderation rules, or specialized instructions. Individual users face a similar threat, since personal or sensitive queries could also be extracted from the output. A small, open inversion model could be enough to pull this off. The paper itself doesn't make explicit claims about attacks on commercial systems. But if the method works on current production models, AI labs will need to address it quickly and patch the issue.

Source: thedecoder