Four OpenAI user-agents, four purposes
Treating every OpenAI crawler as one bot produces inconsistent rules. OpenAI's documentation separates four public uses.
| User-agent | Documented purpose | Policy note |
|---|---|---|
| OAI-SearchBot | Surface pages in ChatGPT search features | Blocking it may prevent a page from being shown in ChatGPT search answers |
| GPTBot | Collect content that may be used to train foundation models | Blocking it opts out of this declared use without controlling the other purposes |
| ChatGPT-User | Perform certain actions requested by a user in ChatGPT or a custom GPT | robots.txt may not apply in the same way as it does to automatic crawling |
| OAI-AdsBot | Review the safety and relevance of pages submitted as ChatGPT ads | It visits submitted ad pages and its data is not used to train foundation models |
A minimal example to adapt
This configuration fits a site that wants to remain accessible to ChatGPT search, declines the training use declared for GPTBot, and does not use ChatGPT ads.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: OAI-AdsBot
Disallow: /
This is not a universal recommendation. An advertiser that submits landing pages should review
the OAI-AdsBot rule for those paths. An application should protect actions and private data
with authentication and authorization, not with robots.txt alone.
Extend the policy to other categories
The same reasoning applies to other operators. Cloudflare distinguishes categories including Search, Agent, Training, Transact, Data Collection, SEO, and Ads Verification. A declared category helps structure a decision but does not prove the actual use of one request.
Use the AI bot decision framework to connect each purpose with its value, risk, and appropriate control.
Verify the result after a change
- Read the public file through a direct HTTP request.
- Check that the CDN or WAF does not serve another version.
- Test sensitive paths separately from public pages.
- Inspect logs with the user-agent, source address, URL, status, and timestamp.
- Classify the result as allowed by robots.txt, blocked at the network, observed in logs, or unmeasurable.
A 200 status establishes that one request received a response. It does not prove indexation or citation. An absence of logs also tells you little when the period, sample, or technical identity of the bot is unknown.
FAQ
Does a Disallow on GPTBot block ChatGPT Search?
No. OpenAI documents GPTBot and OAI-SearchBot for different purposes. Use a separate rule for each purpose you want to control.
Does allowing OAI-AdsBot improve organic visibility?
No organic advantage is documented. OpenAI describes the bot as a reviewer of pages submitted as ads and says advertising is separate from answers.
Does an HTTP 200 status prove indexation?
No. It proves only that a particular request received a response. Indexation and citation require separate evidence.