What changed
OpenAI announced it will honor a common signal in robots.txt, allowing website operators to express their AI training preferences to multiple AI providers at once. This voluntary effort simplifies the process for publishers and website operators by eliminating the need to maintain separate rules for each provider.
What this means in practice
Previously, website operators had to identify individual crawlers and set separate rules for each AI provider. With the new signal, operators can communicate their preferences more efficiently. OpenAI will implement the signal in the coming months and will share further details about its implementation. The new signal will not replace existing crawler controls, which will continue to apply and be respected.
Limitations and availability
The implementation of the signal is planned for the coming months, and specific details will be shared in a future update. OpenAI is joining other participants in the IETF’s AI Preferences working group to support this shared signal. The company will continue to adapt its practices as discussions and the landscape evolve. The signal is part of ongoing efforts to create an open web where website owners can clearly communicate their choices about content usage.
