Robots.txt for AI bots

Robots.txt for AI bots is directives in robots.txt addressed to AI companies' crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini training), CCBot (Common Crawl), Amazonbot, FacebookBot etc. You can allow or forbid them to crawl the site — i.e. manage whether your content is used for model training and generative AI answers. This is a new practice (relevant from 2024-2026). Honestly about 'depends on a bot's good faith', this matters: robots.txt is an instruction, not a technical wall. Rule-respecting bots (GPTBot, ClaudeBot, Google-Extended etc. from major companies) usually follow the directives. But there is no guarantee that ABSOLUTELY all crawlers (especially unscrupulous or unknown ones) will respect them — robots.txt does not block access technically, it asks not to enter. For hard protection other measures are needed (at the server/firewall level). We honestly warn about this. Honestly about the main trade-off, this is the essence of the decision: blocking AI bots protects your content from use in training but has a flip side — if AI systems do not see your content, you may NOT appear in their answers (ChatGPT, Perplexity, Google AI overviews). And presence in AI answers is a growing channel of visibility and traffic (see AEO/GEO — 410/411). So the choice is not 'protect or not' but a balance: content protection VERSUS presence in AI results. We honestly help weigh it for your situation rather than impose one side. Honestly about 'not ordinary SEO': this is about AI access to content, not positions in ordinary Google search. These directives do not affect classic SEO ranking (Googlebot for search and Google-Extended for training are different). Do not confuse them. Honestly about 'practices are forming': this is a young field, the list of bots and approaches change; the setup must be periodically updated as new AI crawlers appear. Honestly about the effect: it gives conscious control of AI access to your content accounting for the 'protection vs presence in AI answers' trade-off, but within bots' good faith and without a hard technical block. Honestly about access: a strategy decision is needed (protect content or be present in AI). An important boundary: this is robots for AI bots; ordinary robots.txt — 382; AEO/GEO (presence in AI) — 410/411; AI-search optimization — 409. The base price starts from 40,000 ₽ (depends on the strategy and the number of bots).

• It is unclear whether your content is used by AI for training and answers.
• You need to decide: protect content from AI or be present in AI answers.
• An expectation that robots.txt hard-blocks all AI bots (it does not).
• The trade-off is not accounted for: blocking AI reduces presence in growing AI results.

• Setting up robots.txt directives for AI crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot etc.)
• An honest strategy analysis: content protection vs presence in AI answers
• Accounting for the current list of AI bots (the field changes fast)
• Explaining: this is about AI access, not ordinary SEO ranking
• Honest boundaries (compliance depends on a bot's good faith — not a hard block; the protection vs AI-presence trade-off; does not affect ordinary search; practices are forming — updating needed)
• A link with AEO/GEO (410/411) and AI-search (409)
• Documentation and handover
• Review with you

• Conscious control of AI bots' access to your content
• A weighed decision 'content protection vs presence in AI answers'
• An honest assessment: what blocking gives and what you pay for it
• Honest boundaries (not a hard block; trade-off with AI visibility; not ordinary SEO; updating needed)

• We discuss the strategy: protect content from AI or be present in AI results
• We set up directives for the chosen strategy and the current bot list
• We honestly set boundaries (good faith, trade-off) and hand over to you

• Will robots.txt reliably block all AI bots from my content?

  Not hard, honestly: robots.txt is an instruction, not a technical wall. Rule-respecting bots (GPTBot, ClaudeBot, Google-Extended from major companies) usually follow the directives, but there is no guarantee that ABSOLUTELY all crawlers (especially unscrupulous or unknown ones) will respect them. For hard protection server/firewall-level measures are needed. We honestly warn about this rather than promise impenetrable blocking via robots.txt.

• If I block AI bots, do I lose nothing?

  You lose possible presence in AI answers, honestly — this is the main trade-off: blocking protects content from training, but if AI systems do not see your content, you may NOT appear in their answers (ChatGPT, Perplexity, AI overviews). And this is a growing channel of visibility and traffic (AEO/GEO — 410/411). The choice is not 'protect or not' but a balance: content protection versus presence in AI results. We honestly help weigh it for your situation.

• Will this affect my positions in ordinary Google?

  No, honestly: this is about AI access to content (training and AI answers), not ranking in classic search. Googlebot (for search) and Google-Extended (for AI training) are different crawlers; managing the second does not affect ordinary positions. Do not confuse them. Plus this is a young field — the bot list and practices change, the setup must be periodically updated. We honestly distinguish and warn.