Specific allow/disallow per AI bot
We set up pointed rules for each AI bot separately: e.g. allow AI search engines (for visibility in AI answers) but disallow model-training crawlers — or vice versa. Granular control for your strategy, not 'all or nothing'. Honestly upfront: like robots.txt overall (912), these are rules on VOLUNTARY compliance — well-behaved bots respect them, malicious ones may ignore them; the bot list and their purpose change, so maintenance is needed; it is access-policy management, not a technical guarantee.
Specific allow/disallow per AI bot — overview
Specific allow/disallow per AI bot is granular access configuration separately for each AI bot for your strategy. Different AI bots do different things: some collect data for model TRAINING (e.g. GPTBot, CCBot, Google-Extended), others for AI-SEARCH OUTPUT and answers (e.g. OAI-SearchBot, PerplexityBot). This lets you, for example, allow those that give you visibility in AI answers (which can bring traffic/mentions — see GEO 889/AI search 887) but disallow content use for training others' models — or choose any other combination for your goals. Honestly about the nature of the rules, this is key: like all of robots.txt (912), these are directives on voluntary compliance, NOT a technical block. Well-behaved bots respect pointed rules, malicious ones may ignore them. Granularity gives a finer POLICY but does not change that it is 'a request', not 'a fence'. For real technical separation server-side detection (914) is needed. Honestly about volatility, this is especially important here: bots appear, get renamed, change purpose, and companies launch new crawlers. Pointed configuration requires maintenance and periodic updating, otherwise rules become outdated (a new bot will not fall under your rules). Honestly about the strategy nuance: the 'training vs search' split is not always clear and can change — we honestly configure by the current understanding of bot purposes, but it is a moving target. Honestly about the effect: it gives a deliberate, fine access policy respected by large bots, but not absolute control. Honestly about access: access to robots.txt is needed. An important boundary: this is pointed rules (a recommendation); general robots.txt — 912; real blocking — server-side 914; monetization — pay-per-crawl 915. Picture this: instead of 'all bots or none' — a fine 'yes to these, no to those' setting for your strategy. The base price starts from 15,000 ₽ (depends on the number of bots and strategy).
Problems we solve
- The 'all or nothing' approach does not fit — fine tuning is needed.
- You want AI-search visibility but not give content for training.
- It is unclear which bot is for what (training vs search).
- Rules become outdated: new AI bots are not accounted for.
What's included in the Specific allow/disallow per AI bot service
- Granular allow/disallow for each AI bot
- Separation by purpose (training vs AI-search/output)
- An access strategy for your goals (visibility vs content protection)
- An honest explanation (voluntary compliance, not blocking)
- An update plan (bots appear/change)
- A link with robots.txt (912), server-side (914), GEO/AI search (889/887)
- Rule correctness check
- Handover and review with you
What you get
- A fine access policy per AI bot
- A balance of AI-search visibility and content protection
- A deliberate strategy instead of 'all or nothing'
- Honest boundaries (voluntary compliance; maintenance needed)
How the work goes: steps
- We define the strategy (whom to allow and why); collect access
- We configure pointed rules by bot purpose
- We build in updating, honestly set boundaries with you
Why PDV Expert
- Fixed price and timeline — no surprises on the invoice.
- Report and recommendations in plain language — clear without a technical background.
- In touch at every step and answering questions about the result.
FAQ
Can I allow AI search but disallow training on my content?
Yes, that is the point of pointed rules — different bots do different things (some for training, others for AI-search output), and we configure them separately for your strategy. But honestly: it works on voluntary compliance — well-behaved bots respect the rules, malicious ones may ignore them. Plus the 'training vs search' split is not always clear and can change. It is a fine policy, not a technical guarantee.
Do these rules technically block bots?
No, honestly: like all of robots.txt (912), these are directives on voluntary compliance, not a block. Granularity gives a finer policy (who may do what) but does not change that it is 'a request'. Well-behaved bots respect it, malicious ones may ignore it. For real technical separation server-side detection (914) is needed.
Set up the rules — can we forget them?
No, honestly, and here it is especially important: AI bots constantly appear, get renamed and change purpose. Pointed rules without updating become outdated — a new bot simply will not fall under your setting. We configure by the current understanding and build in periodic updating, because it is a moving target, not a one-off task.
About the provider
The «Specific allow/disallow per AI bot» service is provided by PDV Expert — a team specialising in «Site quality». We work under contract and deliver a written report with recommendations.