Anthropic announced it will integrate SynthID-Text, an AI watermarking technique developed by Google, into its upcoming Claude models. The approach uses a secret key to subtly alter how the model selects words during text generation. This key allows users with knowledge of it to determine if the content was generated by the platform.

The method works by embedding a signal that identifies AI-generated text, known as provenance. SynthID modifies the standard sampling process by introducing a random seed generator, a sampling algorithm, and a scoring function. Instead of using an arbitrary random number generator, the watermarking process uses a secret key to influence word selection.

"Watermarking changes refusal behavior on bare harmful requests, but the effect is more pronounced when the same requests are paired with the prompt-injection technique," said Andrea Siposova, an AI security researcher at Lasso Security. The researcher found that watermarking can make models more likely to answer harmful requests that they would otherwise refuse.

The findings highlight the need for developers to thoroughly test how their large language models and agents behave when watermarking is in place.

Watermarking can influence not only the responses of AI models but also the actions of AI agents that rely on them. This effect, called sampling drift, can change safety behavior, including whether a model refuses a harmful request.

The research also showed that model responses varied depending on which secret key was used. Watermarking changed which individual tool calls were correct, sometimes more than the overall accuracy score suggested. The study tested a half-dozen open-weight models, not the specific implementation used by Claude, but the results indicate that some forms of watermarking may affect model and agent safety.

There are limitations to the research, as it does not test how Claude model responses change under watermarking. The experiments focused on the Hugging Face implementation of SynthID-Text tournament sampling.

Still, the results show that at least some forms of watermarking may impact model and agent safety. It will be important for red-team hacking exercises to stress-test platforms to ensure they perform as expected when SynthID is deployed.

Source: arstechnica