robots.txt Examples: Control Search and AI Crawler Access
Copyable robots.txt examples, a dated table of AI crawler user agents (GPTBot, OAI-SearchBot, ClaudeBot, CCBot), WordPress robots.txt steps and a test matrix.
Useful robots.txt examples begin with a policy decision: which automated clients should be allowed to fetch which public paths? For AI crawlers, decide separately for search, training and user-requested visits, because operators such as OpenAI, Anthropic, Perplexity, Google and Apple use different user agents for each. Write those choices explicitly, test representative URLs, and verify the file at the root of the correct host. Crawling, indexing and private access are separate controls.
Updated November 20, 2026: you now get Google's newer user-triggered agents (including the renamed Gemini Notebook fetcher), the user-agent strings and IP lists to check in your logs, and Apple's separate opt-out for AI answers in Siri and Search, all read from the operators' pages that day.
This guide gives website owners four adaptable configurations, a dated table of AI crawler user agents (as of November 14, 2026), a test matrix and a section on WordPress's built-in robots.txt. Every example.com URL is illustrative. Replace it with your own host and paths, review the complete existing file, and use the robots.txt generator as a drafting aid rather than a substitute for testing. We make PolyDraft; near the end we show the robots.txt on our own site, including what we'd change.
- Use robots.txt for cooperative crawler access rules.
- Use supported indexing directives for search exclusion, and authentication for private content.
- Decide per purpose: blocking a training crawler doesn't remove you from that operator's search results, while blocking a search crawler can.
- Do not assume a named crawler inherits the wildcard group's restrictions.
- User-triggered fetchers may not follow robots.txt; enforce hard blocks at the edge.
- Verify the delivered file and the actual target pages after deployment.
Choose the control that matches your goal
A robots.txt file is publicly readable. A rule asking a crawler not to fetch a path does not password-protect that path. For client documents or administrative data, require authentication and authorization at the application or server layer.
To keep a public HTML page out of Google results, use a supported noindex directive and allow Google to fetch the page so it can see that instruction. Google's robots meta documentation explains the distinction. Blocking the fetch can prevent the directive from being read; a disallowed URL can still be known through links.

AI access comes in three kinds, and each needs its own decision. Training crawlers or tokens collect content that may be used to train models. Search crawlers build the index an assistant cites from. User-triggered fetchers visit a page because someone asked the assistant about it, and some operators say robots.txt may not apply to them. Our guide to generative engine optimization explains why search, training and user-triggered access are separate decisions.
AI crawler user agents: a dated table
As of November 14, 2026, the main AI operators document separate robots.txt tokens for search, training and user-triggered fetches, and blocking one doesn't block the others. The table lists each token, what its operator says it does, whether the operator says it follows robots.txt, and what a Disallow changes. Where an operator's page says nothing, the cell says "not documented". Names are case-insensitive for Google, but copy them exactly from each operator's page.
| Token | Operator | Purpose | Follows robots.txt? (per operator) | What blocking it affects | Official doc |
|---|---|---|---|---|---|
OAI-SearchBot | OpenAI | Search: surfaces sites in ChatGPT's search features | Yes: OpenAI uses this robots.txt tag for search opt-outs, and changes take about 24 hours to apply | The site isn't shown in ChatGPT search answers, though it can still appear as a navigational link | OpenAI crawlers (read) |
GPTBot | OpenAI | Training: crawls content that may be used to train OpenAI's foundation models | Yes: a Disallow is OpenAI's documented training opt-out | Signals that your content shouldn't be used for training; doesn't affect ChatGPT search | OpenAI crawlers (read) |
ChatGPT-User | OpenAI | User-triggered: visits a page when a ChatGPT or custom GPT user asks | "Robots.txt rules may not apply" | Not used for automatic crawling or to decide search inclusion; a hard block needs an edge rule | OpenAI crawlers (read) |
OAI-AdsBot | OpenAI | Ad checks: visits only landing pages submitted as ChatGPT ads | Not documented | Not documented; OpenAI says its data isn't used for training | OpenAI crawlers (read) |
ClaudeBot | Anthropic | Training: collects web content for Anthropic's models | Yes: Anthropic says its bots honor robots.txt, including Crawl-delay | Signals that future materials should be excluded from training datasets | Anthropic crawlers (read) |
Claude-SearchBot | Anthropic | Search: analyzes content to improve Claude's search responses | Yes (same statement) | May reduce your visibility and accuracy in Claude's search results | Anthropic crawlers (read) |
Claude-User | Anthropic | User-triggered: fetches pages when a Claude user asks | Yes (same statement) | Controls whether user-initiated requests can reach your pages; may reduce visibility in user-directed search | Anthropic crawlers (read) |
PerplexityBot | Perplexity | Search: surfaces and links sites in Perplexity results; not used to train foundation models | Yes: Perplexity says it follows robots.txt, and changes can take up to 24 hours to apply | PerplexityBot stops crawling your pages for Perplexity's search results | Perplexity crawlers (read) |
Perplexity-User | Perplexity | User-triggered: fetches a page in real time for a user's question | "Generally ignores robots.txt rules" | A robots.txt rule may not stop it; use an edge rule | Perplexity crawlers (read) |
Googlebot | Search: Google Search and all its features, Discover, Images and News | Yes: Google's common crawlers "always obey robots.txt rules when crawling automatically" | Pages can't be crawled for Google Search, which also feeds AI Overviews and AI Mode | Google common crawlers (read) | |
Google-Extended | Robots token, not a crawler: Gemini training and grounding | No user agent of its own; Google's existing crawlers read it as a control | Opts out of training future Gemini models and of grounding in Gemini Apps and Vertex AI; no effect on Search inclusion or ranking | Google common crawlers (read) | |
Applebot | Apple | Search: Siri, Spotlight and other Apple search features | Yes: Apple says it follows all robots.txt rules, and follows Googlebot's rules if Applebot isn't named | Your content may not appear in Apple's search results | About Applebot (read) |
Applebot-Extended | Apple | Robots token for training Apple's generative foundation models | A robots.txt control read alongside Applebot's rules | Opts out of training; pages can still appear in Apple's search results. It doesn't opt you out of Apple's AI-generated answers (see below) | About Applebot (read) |
CCBot | Common Crawl | Dataset: builds Common Crawl's free, public web archive | Yes, including Crawl-delay | Stops future crawls for the archive; removal of already published crawls is not documented | Common Crawl FAQ (excerpt) |
bingbot | Microsoft | Search: Bing's index; Microsoft has said Microsoft 365 Copilot Chat is grounded in Bing search | Not verified on November 14 (Bing's page couldn't be opened) | Pages can't be crawled for Bing; effects on Copilot not documented on a page we could read | Bingbot (not read) |
__usage_ai-training | Shared (IETF draft) | Training preference across providers | Support varies: Google says early 2027; Anthropic recognizes a signal from the same draft since October 30, 2026 (its page doesn't name the token) | Expresses a preference against AI training to the providers that support it (see below) | Google announcement (read) |
"Read" means we opened the operator's page on November 14, 2026, or, for Perplexity and Apple, on November 20, 2026. We reread the OpenAI, Anthropic and Google pages on November 20 and found no change to their rows. "Excerpt" means the page couldn't be opened from our tools, and the row matches the operator's own wording in search excerpts and in our earlier readings. Anthropic's page also names a Claude-Agent identifier for Anthropic-hosted browsers, which it doesn't list as a crawler.
Does blocking Bingbot affect ChatGPT? OpenAI's crawler page, read on November 14, 2026, doesn't mention Bing; it tells sites to allow OAI-SearchBot to appear in ChatGPT search. OpenAI's help center has said ChatGPT search uses third-party search providers, and that Bing is the only one for Enterprise and Edu workspaces (we last read that page on October 12, 2026). So allow Bingbot for Bing and Copilot, allow OAI-SearchBot for ChatGPT search, and don't assume one replaces the other. Our guide to how ChatGPT finds and cites sources covers the rest.


Newer user-triggered agents, and how to spot each bot in your logs
As of November 20, 2026, Google's list of user-triggered fetchers includes two entries the table above doesn't cover:
- Google-Agent. Google says it "is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request." It uses the IP ranges in
user-triggered-agents.json. Google adds that it is experimenting with the Web Bot Auth protocol for these requests, using thehttps://agent.bot.googidentity. - Google-GeminiNotebook. The Gemini Notebook fetcher requests URLs that Gemini Notebook users add as sources. Google lists
Google-NotebookLMas its former agent, supported until August 2026, so a rule or log filter that names only the old string no longer matches.
Google says user-triggered fetchers "generally ignore robots.txt rules", so treat both like ChatGPT-User and Perplexity-User: decide whether you want them at all, and enforce a hard block at the edge.
To confirm which bot actually visited, match the user-agent string and the source IP. These are the strings and IP lists each operator published on November 20, 2026. OpenAI and Perplexity say the version numbers may change, so match the name, not the version.
| Bot | Look for in the User-Agent | Published IP list |
|---|---|---|
| OAI-SearchBot | OAI-SearchBot/1.4; +https://openai.com/searchbot | openai.com/searchbot.json |
| GPTBot | GPTBot/1.4; +https://openai.com/gptbot | openai.com/gptbot.json |
| ChatGPT-User | ChatGPT-User/1.0; +https://openai.com/bot | openai.com/chatgpt-user.json |
| PerplexityBot | PerplexityBot/1.0; +https://perplexity.ai/perplexitybot | perplexity.ai/perplexitybot.json |
| Perplexity-User | Perplexity-User/1.0; +https://perplexity.ai/perplexity-user | perplexity.ai/perplexity-user.json |
| Applebot | Applebot/, with reverse DNS in applebot.apple.com | Applebot IP CIDRs (JSON file linked from Apple's page) |
| Google-Agent | compatible; Google-Agent | user-triggered-agents.json (linked from Google's page) |
Two details from OpenAI's page help when you read logs. When OAI-SearchBot or GPTBot fetches robots.txt itself, OpenAI says it may add a robots.txt marker to the user-agent string, so you can tell those requests apart even if your logs don't record paths. And if a site allows both bots, OpenAI says it "may use the results from just one crawl for both use cases."
Apple's page adds a control that robots.txt doesn't cover. Applebot data may also be used as context when Apple's AI models generate answers, such as broad world knowledge answers in Siri and Search. Apple says to opt out of those answers with the nosnippet robots meta tag on the content concerned. Disallowing Applebot-Extended only covers training. Apple also says pages marked isAccessibleForFree: false in structured data can still appear in search results but won't be used as that context, and that Applebot doesn't follow Crawl-delay.
The shared AI-training token: what's settled and what isn't
On October 30, 2026, Google said it will support __usage_ai-training, a robots.txt token from an IETF individual Internet-Draft, and Anthropic began recognizing a no-AI-training signal based on the same draft. The token isn't a standard yet, and it uses ordinary robots.txt syntax: you write a User-agent: __usage_ai-training group with Disallow or Allow rules. As of November 14, 2026:
- Anthropic says it began recognizing a "no AI training" signal based on the IETF draft work on October 30, 2026, and is rolling it out across its services over time; its page describes the signal without naming the token. Rules that name
ClaudeBottake priority over the shared signal. - Google says in its crawler documentation that it intends to support the token in early 2027. It will treat a shared-token opt-out as a
Google-Extendeddisallow unless your file names Google-Extended separately. - OpenAI's crawler page doesn't mention the shared token; it still documents only
GPTBotfor training opt-outs. - Microsoft: we couldn't find a Microsoft statement on the token or a support date.
A User-agent: * group doesn't express this preference, and the token doesn't remove content already collected. Until every operator you care about supports it, keep your named training rules (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) and treat the shared group as an addition.
Four robots.txt examples for search and AI crawlers
Example A: allow public pages, avoid an internal search path
Assume your public site has articles and product pages, while /search/ contains internal search-result URLs you do not want cooperatively crawled. This file expresses that narrow preference. It does not remove already indexed URLs and it is not an access restriction.
User-agent: *
Disallow: /search/
Sitemap: https://example.com/sitemap.xml
Do not paste the path into a site that uses /search/ for useful editorial pages. First list the URLs it would match. Keep scripts, stylesheets and images needed to understand public pages accessible unless you have a specific reason to restrict them.
Example B: permit ChatGPT search crawling and decline GPTBot crawling
OpenAI documents OAI-SearchBot for search and GPTBot for content that may be used in model training. Those settings are independent. The following policy keeps an internal search-path restriction for the explicitly named search bot and declines GPTBot access. Consult the official crawler reference when updating names or network rules.
User-agent: *
Disallow: /search/
User-agent: OAI-SearchBot
Disallow: /search/
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
Allowing search access does not guarantee a citation. It also does not control every user-triggered fetch: OpenAI describes ChatGPT-User separately and notes that robots rules may not apply to those requests. Do not treat one crawler group as a universal policy for every AI use. OpenAI says ChatGPT search can take about 24 hours to reflect a robots.txt change.
Example C: keep Google search crawling separate from Google-Extended
Google-Extended is a product-control token, not a separate HTTP crawler user-agent. Google's crawler documentation distinguishes it from Google Search and says it does not affect inclusion or ranking there. It addresses specified Gemini training and grounding uses, so do not describe it as a training-only switch.
User-agent: *
Disallow: /search/
User-agent: Google-Extended
Disallow: /
Sitemap: https://example.com/sitemap.xml
Blocking Google-Extended doesn't keep your pages out of AI Overviews, which use Googlebot; our guide explains how AI Overviews choose sources.
Example D: allow AI search, decline AI training
This is the policy many publishers want: stay visible in AI search answers, and say no to model training. It names the search crawlers so their restrictions are explicit, puts the training tokens from OpenAI, Anthropic, Google and Apple in one group, and adds the shared training token as a forward-looking extra.
User-agent: *
Disallow: /search/
# AI search crawlers: keep the same restriction as everyone else
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /search/
# AI training: decline
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
# Shared AI-training preference (support dates differ)
User-agent: __usage_ai-training
Disallow: /
Sitemap: https://example.com/sitemap.xml
Several User-agent lines can share one group of rules; Google's robots.txt specification supports this, but test any other parser you rely on. The user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) fall under the * group here, and two of their operators say robots.txt may not bind them. Googlebot and Applebot also use the * group, so Google and Apple search are unaffected. CCBot isn't in the training group because Common Crawl's archive serves more than AI training; add a User-agent: CCBot line to that group if you want out of its future crawls too.
These are alternative starting policies. Do not stack them into production without resolving duplicate groups and retaining your current intentional restrictions. If several people manage the site, name one policy owner and keep a short explanation beside each non-obvious rule in your change record.
Understand group selection before adding exceptions
For Google, the most specific matching user-agent group is selected; a specific group does not simply inherit the wildcard group's rules. Within the applicable rules, path specificity matters, and an equally specific allow rule wins over a disallow rule. See Google's robots.txt interpretation; validate other crawlers against their own behavior rather than assuming universal parser identity.
This is why Examples B and D repeat Disallow: /search/ in the search-crawler groups. A file that adds a named group containing only Allow: / can unintentionally lose a restriction the author expected to inherit.
In one sample, ChatGPT suggested exactly that pattern. We ran the prompt "how do I block GPTBot" through ChatGPT via DataForSEO's ChatGPT scraper on November 14, 2026 (US English, logged out, one run). The answer cited only OpenAI's crawler page and proposed a User-agent: OAI-SearchBot group with just Allow: /. That is harmless on a site with no other rules. On a file like Example A, it would quietly let OAI-SearchBot into /search/. Answers vary by run, so treat this as one sample, not ChatGPT's fixed advice.

Be precise about path boundaries. A prefix such as /search is broader than /search/. Case, trailing slashes and query patterns can change which URLs match. Before adding a wildcard, write at least one URL that should match and one nearby URL that must not match.
Build a small test matrix
For Example B, use the following expectations as a starting fixture. “Allowed” means the robots policy permits a fetch; the server could still return an error, require a login or block the request through a firewall.
| URL path | Googlebot | OAI-SearchBot | GPTBot |
|---|---|---|---|
/ | Allow | Allow | Disallow |
/blog/guide | Allow | Allow | Disallow |
/search/results | Disallow | Disallow | Disallow |
/media/diagram.webp | Allow | Allow | Disallow |
For Example D, we ran the file through two open-source parsers (Protego 0.7.0 and Python 3.13's urllib.robotparser), querying each token name rather than a full user-agent string, on November 6, 2026. Both returned these results:
| URL path | Googlebot | OAI-SearchBot, Claude-SearchBot, PerplexityBot | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended | Claude-User (falls under *) |
|---|---|---|---|---|
/ | Allow | Allow | Disallow | Allow |
/blog/guide | Allow | Allow | Disallow | Allow |
/search/results | Disallow | Disallow | Disallow | Disallow |
/media/diagram.webp | Allow | Allow | Disallow | Allow |

Extend the fixture with your most important product URL, one language variant and any custom path affected by a new rule. Store the expected result before running a parser. Otherwise, it is too easy to approve whatever result the tool returns.
There are two different tests here: parsing the policy and making an HTTP request. A successful browser request does not prove a bot is permitted. A parser's “allow” result does not prove your CDN lets the bot through. Test both layers, and verify real crawler identity from vendor guidance before creating firewall exceptions. If your site runs on Cloudflare or Vercel, check what your platform's AI crawler data shows: it records which crawlers were allowed or blocked at the edge.
When robots.txt isn't enough
robots.txt is a request to cooperating crawlers, not a lock. User-triggered fetchers such as ChatGPT-User and Perplexity-User visit a page because someone asked, and their operators say robots.txt rules may not apply or are generally ignored. Anyone can also send a bot's user-agent string. To enforce a block:
- Decide whether you really need one. A hard block on a user-triggered fetcher also stops that assistant from reading your page when a customer asks about you.
- Verify identity first. OpenAI, Anthropic and Perplexity publish IP address lists for their crawlers (linked from the pages above); match the request's source IP, not just its user-agent.
- Enforce at the edge with a firewall or CDN rule, for example Cloudflare's AI Crawl Control, which enforces each block with a WAF custom rule, and log what it blocks.
- Keep robots.txt readable. Anthropic notes that blocking its IP addresses can stop it from reading your robots.txt, so an IP block alone may not work as a lasting opt-out.
Deploy and verify the correct file
Publish the plain UTF-8 file at /robots.txt on the exact host and protocol you intend to control. Google's creation guide covers file placement and validation options. A file on the main domain does not automatically configure every subdomain. For a new site, our launch-day SEO checklist tests for a staging Disallow: / left in production.
- Save the current file and record the reason for the change.
- Review the diff against the expected URL matrix. A read-only Skill that reviews robots.txt changes can do a first pass, but a person signs off.
- Fetch the live file in a private browser session and inspect its text, status and content type. To see a page's robots rules while you browse, our guide to SEO Chrome extensions lists one whose store listing says it reports on AI bots, with the site access it requests.
- Confirm it is not an HTML error page or cached older version.
- Run policy tests, then test the important public pages and assets. Our GEO audit checklist repeats this check across a whole site.
- Check the robots.txt report and relevant URL Inspection results in Search Console where you have access.
Once the file is live, the rest of the setup (sitemap submission, Bing Webmaster Tools and IndexNow) is in our post-launch SEO checklist.
Keep the previous version available for rollback. If a release unexpectedly blocks product pages, restore the last reviewed policy, verify the restored response, and investigate the error before retrying. Do not “fix” it by granting every automated request access to private application routes.
robots.txt on WordPress
WordPress serves a virtual robots.txt: there is no file on disk until you add one. When a request for /robots.txt reaches WordPress, its do_robots() function writes the file on the fly. Upload a real robots.txt to the site root and the web server sends that file instead. You can also change the virtual file with the robots_txt filter. As of November 14, 2026, a standard install with pretty permalinks and search engines allowed outputs this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xml
The admin paths come from your admin URL. The Sitemap line comes from WordPress's built-in sitemaps (since version 5.5) and only appears while the site is public. WordPress only adds its robots.txt rewrite when it is installed at the domain root. A plugin or your host may also change this output, so read the live file rather than trusting the defaults.
Three ways to edit it
| Method | How it works | Watch out for |
|---|---|---|
| Upload a physical file | Place robots.txt in the web root. WordPress's standard Apache rules only route requests for files that don't exist, so the server returns your file. | Filters and plugin settings that edit the virtual file stop having any effect. Copy the Sitemap line in yourself. |
The robots_txt filter | A few lines in a small plugin or a child theme append or replace the generated text. The filter receives the output and whether the site is public. | Only works while no physical file exists. Test the live URL after each deploy. |
| An SEO plugin's editor | Some SEO plugins add a robots.txt editor to the admin screens. | Follow that plugin's own documentation, and check whether it writes a physical file or filters the virtual one. |
For example, this filter adds the decline-training group from Example D to the virtual file, and only while the site is public:
add_filter( 'robots_txt', function ( $output, $public ) {
if ( $public ) {
$output .= "\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: Google-Extended\nUser-agent: Applebot-Extended\nDisallow: /\n";
}
return $output;
}, 10, 2 );
Then open https://your-site/robots.txt in a private window and run the test matrix above against the result.
The "Discourage search engines" setting
The checkbox under Settings → Reading, "Discourage search engines from indexing this site", no longer writes Disallow: / into robots.txt. Since WordPress 5.3 it adds a noindex robots meta tag to pages instead, and the virtual file drops its Sitemap line. WordPress's own help text warns that "not all search engines adhere to this directive." Treat it as a staging switch, not access control. It also tells AI crawlers nothing about training.
"Indexed, though blocked by robots.txt" on WordPress
Search Console shows this warning when Google indexed a URL it wasn't allowed to crawl, usually because other pages link to it. Google's help says robots.txt "is not the correct mechanism" to keep a page out of results. To remove it, delete the robots.txt rule and serve noindex, for example through your SEO plugin's per-page setting. Some blocks are intentional, such as internal search pages. For those, Google notes that fixing every blocked URL may not make sense.
What our own robots.txt says
As of November 14, 2026, polydraft.ai/robots.txt allows every crawler and names seven AI tokens, all allowed: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended. That includes training tokens, so our site currently allows AI training as well as AI search.

Measured against the table above, the file has two flaws. The comment # AI Search Engine Crawlers sits above GPTBot, ClaudeBot and Google-Extended, which are training tokens, not search crawlers. And Claude-User, Perplexity-User, Applebot-Extended and CCBot aren't named; they fall under the * group, which allows them. Neither changes what crawlers may fetch, but a mislabelled file makes the next policy change harder to review.
We make PolyDraft, an AI website builder. Sites built with PolyDraft serve the same robots.txt as ours: a fixed allow-all preset that names the seven AI tokens above and links the sitemap. As of November 14, 2026, PolyDraft has no setting to switch that preset to a decline-training policy like Example D. Because the site runs on your own Cloudflare account, you can block individual AI crawlers in Cloudflare's AI Crawl Control; PolyDraft's own admin panel doesn't edit robots.txt. Cloudflare's managed robots.txt can also add Disallow rules for training crawlers to the file your site serves, so test the delivered file afterwards, as in the release checks above. PolyDraft is not WordPress, so the WordPress steps above don't apply to it.
Frequently asked questions
Does Disallow remove a page from Google?
No. It controls crawling by compliant clients. Search exclusion requires the appropriate indexing controls, and Google must be able to see them. Private material needs real access controls rather than a publicly visible list of paths in robots.txt.
Should I block every AI crawler?
That is a business policy choice. First distinguish search discovery, training preferences and user-requested access. Then decide per documented client and use. A blanket rule may conflict with your wish to appear in a search product; an allow rule still promises no placement.
What's the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot crawls for ChatGPT search results. ChatGPT-User fetches a page when a user asks ChatGPT about it, and OpenAI says robots.txt rules may not apply to it. Each setting is independent, so you can allow search and decline training.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended doesn't affect inclusion or ranking in Google Search, and AI Overviews and AI Mode use Googlebot. Blocking Googlebot would remove you from both Search and its AI features.
Can a generator guarantee the file is correct?
No. A generator can format rules, but it does not know every path on your site or your intended access policy. Review its output, test representative URLs and confirm the live response after deployment. Keep the configuration as small as the policy allows.
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's search crawler. It surfaces websites in ChatGPT's search features. If your robots.txt disallows it, OpenAI says your site won't be shown in ChatGPT search answers, though it can still appear as a navigational link. It is separate from GPTBot, which is for training, and changes take about 24 hours to apply.
What is CCBot, and does blocking it stop AI training?
CCBot is Common Crawl's crawler. It builds a free, public web archive that anyone can download, and Common Crawl says it obeys robots.txt, including Crawl-delay. Blocking it stops future crawls for that archive. It doesn't control copies other parties already hold, and it isn't a substitute for GPTBot, ClaudeBot or Google-Extended rules.
Where is the robots.txt file in WordPress?
By default there isn't one on disk. WordPress generates a virtual robots.txt at /robots.txt when no physical file exists. You can upload a real file to the site root, which replaces the virtual one, or change the virtual file with the robots_txt filter or an SEO plugin.
Turn the policy into a tested artifact
Draft your file with the PolyDraft robots.txt generator, check each token's purpose against the table above, then apply the checks in this guide. If the file is correct but assistants still cannot read a page, continue with our guide to checking website readability for AI assistants.
Continue with the next task
After testing your rules, check the HTML a permitted crawler receives.
If you also maintain an optional AI-facing page index, validate it separately from access rules. Read the llms.txt example and validation.
If your site runs on Wix or Webflow rather than WordPress, see which robots.txt settings each platform lets you change in our guides to Wix SEO and Webflow SEO.
Sources
- Google: robots meta documentation
- OpenAI: Overview of OpenAI crawlers
- Google: common crawlers (Googlebot, Google-Extended)
- Google: user-triggered fetchers (Google-Agent, Gemini Notebook)
- Google: robots.txt interpretation
- Google: creating a robots.txt file
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Apple: About Applebot
- IETF Internet-Draft: Machine-Readable Training Preference Signal
- Google Search Central Blog: Supporting a shared AI training control for websites
- Common Crawl: CCBot and FAQ
- Bing: Bingbot
- WordPress Developer Resources: do_robots()
- WordPress source: wp-includes/functions.php (do_robots)
- Google Search Console Help: Page indexing report
- Cloudflare: Manage AI crawlers and Managed robots.txt
Sources were checked on November 14, 2026. We read the OpenAI, Anthropic, Google and Cloudflare pages, Google's Search Console help and WordPress's source code directly that day. On November 20, 2026, we reread the OpenAI, Anthropic and Google crawler pages and read the Perplexity and Apple pages and Google's user-triggered fetchers page. The Common Crawl, Bing, WordPress Developer Resources and IETF pages couldn't be opened from our tools, so those rows rest on search excerpts of the official pages from November 14 and our earlier readings; the table marks which is which. Operators change these pages often, so recheck before you rely on a token.
Revision history: November 6, 2026: added AI crawler purposes, the shared AI-training token and an example that allows AI search but declines training. November 14, 2026: rechecked every AI crawler against its operator's docs, added CCBot and Bingbot to the table, added WordPress steps, and corrected who supports the shared AI-training token.
About this article: Prepared with AI assistance for PolyDraft, checked by an automated QA script and a separate AI fact-check, and approved by Vincent Chen. Worked examples and diagrams are illustrative, not customer results. The Example D parser results come from a run on November 6, 2026; the OpenAI GPTBot and robots.txt screenshots were captured from public pages that day, and the OAI-SearchBot screenshot on November 14, 2026. The ChatGPT answer was collected through DataForSEO's ChatGPT scraper on November 14, 2026, one run, logged out. The hero is an AI-generated editorial illustration. For corrections, contact PolyDraft with the page URL and supporting source.


