Separate identity, declared intent and observation
Identity answers who sent a request. Published IP ranges, reverse DNS or Web Bot Auth can support that answer. Declared intent says what the operator reports the bot will do. Server logs show which URL, status and time your own system recorded.
These are separate evidence layers. A verified identity or declared category does not prove that one request trained a model, produced a citation or completed a transaction.
Use BotBase as a directory, not a guarantee
Cloudflare classifies verified bots by behaviors including Search, Agent, Training, Transact, Data Collection, SEO and Ads Verification. One bot can declare several behaviors.
BotBase for Operators lets an operator submit a bot, follow its submission status, correct information and declare behaviors, content use and direct or intermediary operation. This improves transparency. It does not provide a general authorization or prove how every request will use content.
Set and test policy by purpose
- Inventory observed agents before changing policy and verify the operator where possible.
- Decide separately whether the site wants search discovery, training reuse, user-triggered retrieval, transactions, data collection, SEO tools or advertising review.
- Protect private data and transactional actions with authentication and authorization rather than relying on robots.txt.
- Test the published rule, monitor traffic after deployment and keep a dated decision record.
- Describe an allowed request, an observed crawl, a citation and a conversion as different outcomes.