Product

Solutions

More

robots.txt Examples: Control Search and AI Crawler Access

Copyable robots.txt examples, a dated table of AI crawler user agents (GPTBot, OAI-SearchBot, ClaudeBot, CCBot), WordPress robots.txt steps and a test matrix.

robots.txt for AI crawlers: decide separately for search, training and user-requested visits, then test sample URLs and verify the deployed file.

Useful robots.txt examples begin with a policy decision: which automated clients should be allowed to fetch which public paths? For AI crawlers, decide separately for search, training and user-requested visits, because operators such as OpenAI, Anthropic, Perplexity, Google and Apple use different user agents for each. Write those choices explicitly, test representative URLs, and verify the file at the root of the correct host. Crawling, indexing and private access are separate controls.

Updated November 20, 2026: you now get Google's newer user-triggered agents (including the renamed Gemini Notebook fetcher), the user-agent strings and IP lists to check in your logs, and Apple's separate opt-out for AI answers in Siri and Search, all read from the operators' pages that day.

This guide gives website owners four adaptable configurations, a dated table of AI crawler user agents (as of November 14, 2026), a test matrix and a section on WordPress's built-in robots.txt. Every example.com URL is illustrative. Replace it with your own host and paths, review the complete existing file, and use the robots.txt generator as a drafting aid rather than a substitute for testing. We make PolyDraft; near the end we show the robots.txt on our own site, including what we'd change.

  • Use robots.txt for cooperative crawler access rules.
  • Use supported indexing directives for search exclusion, and authentication for private content.
  • Decide per purpose: blocking a training crawler doesn't remove you from that operator's search results, while blocking a search crawler can.
  • Do not assume a named crawler inherits the wildcard group's restrictions.
  • User-triggered fetchers may not follow robots.txt; enforce hard blocks at the edge.
  • Verify the delivered file and the actual target pages after deployment.

Choose the control that matches your goal

A robots.txt file is publicly readable. A rule asking a crawler not to fetch a path does not password-protect that path. For client documents or administrative data, require authentication and authorization at the application or server layer.

To keep a public HTML page out of Google results, use a supported noindex directive and allow Google to fetch the page so it can see that instruction. Google's robots meta documentation explains the distinction. Blocking the fetch can prevent the directive from being read; a disallowed URL can still be known through links.

A gate, a stamp and a locked box, each with its own pages: crawling uses robots.txt, indexing uses supported directives, and private content needs authentication.
A crawl restriction neither removes a URL from search by itself nor protects private data.

AI access comes in three kinds, and each needs its own decision. Training crawlers or tokens collect content that may be used to train models. Search crawlers build the index an assistant cites from. User-triggered fetchers visit a page because someone asked the assistant about it, and some operators say robots.txt may not apply to them. Our guide to generative engine optimization explains why search, training and user-triggered access are separate decisions.

AI crawler user agents: a dated table

As of November 14, 2026, the main AI operators document separate robots.txt tokens for search, training and user-triggered fetches, and blocking one doesn't block the others. The table lists each token, what its operator says it does, whether the operator says it follows robots.txt, and what a Disallow changes. Where an operator's page says nothing, the cell says "not documented". Names are case-insensitive for Google, but copy them exactly from each operator's page.

AI and search crawler tokens, as documented by each operator on November 14, 2026
TokenOperatorPurposeFollows robots.txt? (per operator)What blocking it affectsOfficial doc
OAI-SearchBotOpenAISearch: surfaces sites in ChatGPT's search featuresYes: OpenAI uses this robots.txt tag for search opt-outs, and changes take about 24 hours to applyThe site isn't shown in ChatGPT search answers, though it can still appear as a navigational linkOpenAI crawlers (read)
GPTBotOpenAITraining: crawls content that may be used to train OpenAI's foundation modelsYes: a Disallow is OpenAI's documented training opt-outSignals that your content shouldn't be used for training; doesn't affect ChatGPT searchOpenAI crawlers (read)
ChatGPT-UserOpenAIUser-triggered: visits a page when a ChatGPT or custom GPT user asks"Robots.txt rules may not apply"Not used for automatic crawling or to decide search inclusion; a hard block needs an edge ruleOpenAI crawlers (read)
OAI-AdsBotOpenAIAd checks: visits only landing pages submitted as ChatGPT adsNot documentedNot documented; OpenAI says its data isn't used for trainingOpenAI crawlers (read)
ClaudeBotAnthropicTraining: collects web content for Anthropic's modelsYes: Anthropic says its bots honor robots.txt, including Crawl-delaySignals that future materials should be excluded from training datasetsAnthropic crawlers (read)
Claude-SearchBotAnthropicSearch: analyzes content to improve Claude's search responsesYes (same statement)May reduce your visibility and accuracy in Claude's search resultsAnthropic crawlers (read)
Claude-UserAnthropicUser-triggered: fetches pages when a Claude user asksYes (same statement)Controls whether user-initiated requests can reach your pages; may reduce visibility in user-directed searchAnthropic crawlers (read)
PerplexityBotPerplexitySearch: surfaces and links sites in Perplexity results; not used to train foundation modelsYes: Perplexity says it follows robots.txt, and changes can take up to 24 hours to applyPerplexityBot stops crawling your pages for Perplexity's search resultsPerplexity crawlers (read)
Perplexity-UserPerplexityUser-triggered: fetches a page in real time for a user's question"Generally ignores robots.txt rules"A robots.txt rule may not stop it; use an edge rulePerplexity crawlers (read)
GooglebotGoogleSearch: Google Search and all its features, Discover, Images and NewsYes: Google's common crawlers "always obey robots.txt rules when crawling automatically"Pages can't be crawled for Google Search, which also feeds AI Overviews and AI ModeGoogle common crawlers (read)
Google-ExtendedGoogleRobots token, not a crawler: Gemini training and groundingNo user agent of its own; Google's existing crawlers read it as a controlOpts out of training future Gemini models and of grounding in Gemini Apps and Vertex AI; no effect on Search inclusion or rankingGoogle common crawlers (read)
ApplebotAppleSearch: Siri, Spotlight and other Apple search featuresYes: Apple says it follows all robots.txt rules, and follows Googlebot's rules if Applebot isn't namedYour content may not appear in Apple's search resultsAbout Applebot (read)
Applebot-ExtendedAppleRobots token for training Apple's generative foundation modelsA robots.txt control read alongside Applebot's rulesOpts out of training; pages can still appear in Apple's search results. It doesn't opt you out of Apple's AI-generated answers (see below)About Applebot (read)
CCBotCommon CrawlDataset: builds Common Crawl's free, public web archiveYes, including Crawl-delayStops future crawls for the archive; removal of already published crawls is not documentedCommon Crawl FAQ (excerpt)
bingbotMicrosoftSearch: Bing's index; Microsoft has said Microsoft 365 Copilot Chat is grounded in Bing searchNot verified on November 14 (Bing's page couldn't be opened)Pages can't be crawled for Bing; effects on Copilot not documented on a page we could readBingbot (not read)
__usage_ai-trainingShared (IETF draft)Training preference across providersSupport varies: Google says early 2027; Anthropic recognizes a signal from the same draft since October 30, 2026 (its page doesn't name the token)Expresses a preference against AI training to the providers that support it (see below)Google announcement (read)

"Read" means we opened the operator's page on November 14, 2026, or, for Perplexity and Apple, on November 20, 2026. We reread the OpenAI, Anthropic and Google pages on November 20 and found no change to their rows. "Excerpt" means the page couldn't be opened from our tools, and the row matches the operator's own wording in search excerpts and in our earlier readings. Anthropic's page also names a Claude-Agent identifier for Anthropic-hosted browsers, which it doesn't list as a crawler.

Does blocking Bingbot affect ChatGPT? OpenAI's crawler page, read on November 14, 2026, doesn't mention Bing; it tells sites to allow OAI-SearchBot to appear in ChatGPT search. OpenAI's help center has said ChatGPT search uses third-party search providers, and that Bing is the only one for Enterprise and Edu workspaces (we last read that page on October 12, 2026). So allow Bingbot for Bing and Copilot, allow OAI-SearchBot for ChatGPT search, and don't assume one replaces the other. Our guide to how ChatGPT finds and cites sources covers the rest.

OpenAI's crawler documentation on November 14, 2026: OAI-SearchBot is for search, sites opted out of it will not be shown in ChatGPT search answers though they can still appear as navigational links, plus its user-agent string and published IP list.
OpenAI's crawler overview, OAI-SearchBot row, captured November 14, 2026.
OpenAI's crawler table: GPTBot crawls content that may be used in training its generative AI foundation models, and disallowing it signals content shouldn't be used for training; ChatGPT-User visits pages for user actions, isn't used for automatic crawling, and robots.txt rules may not apply.
OpenAI's crawler overview, GPTBot and ChatGPT-User rows, captured November 6, 2026.

Newer user-triggered agents, and how to spot each bot in your logs

As of November 20, 2026, Google's list of user-triggered fetchers includes two entries the table above doesn't cover:

  • Google-Agent. Google says it "is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request." It uses the IP ranges in user-triggered-agents.json. Google adds that it is experimenting with the Web Bot Auth protocol for these requests, using the https://agent.bot.goog identity.
  • Google-GeminiNotebook. The Gemini Notebook fetcher requests URLs that Gemini Notebook users add as sources. Google lists Google-NotebookLM as its former agent, supported until August 2026, so a rule or log filter that names only the old string no longer matches.

Google says user-triggered fetchers "generally ignore robots.txt rules", so treat both like ChatGPT-User and Perplexity-User: decide whether you want them at all, and enforce a hard block at the edge.

To confirm which bot actually visited, match the user-agent string and the source IP. These are the strings and IP lists each operator published on November 20, 2026. OpenAI and Perplexity say the version numbers may change, so match the name, not the version.

User-agent strings and IP lists, as published by each operator on November 20, 2026
BotLook for in the User-AgentPublished IP list
OAI-SearchBotOAI-SearchBot/1.4; +https://openai.com/searchbotopenai.com/searchbot.json
GPTBotGPTBot/1.4; +https://openai.com/gptbotopenai.com/gptbot.json
ChatGPT-UserChatGPT-User/1.0; +https://openai.com/botopenai.com/chatgpt-user.json
PerplexityBotPerplexityBot/1.0; +https://perplexity.ai/perplexitybotperplexity.ai/perplexitybot.json
Perplexity-UserPerplexity-User/1.0; +https://perplexity.ai/perplexity-userperplexity.ai/perplexity-user.json
ApplebotApplebot/, with reverse DNS in applebot.apple.comApplebot IP CIDRs (JSON file linked from Apple's page)
Google-Agentcompatible; Google-Agentuser-triggered-agents.json (linked from Google's page)

Two details from OpenAI's page help when you read logs. When OAI-SearchBot or GPTBot fetches robots.txt itself, OpenAI says it may add a robots.txt marker to the user-agent string, so you can tell those requests apart even if your logs don't record paths. And if a site allows both bots, OpenAI says it "may use the results from just one crawl for both use cases."

Apple's page adds a control that robots.txt doesn't cover. Applebot data may also be used as context when Apple's AI models generate answers, such as broad world knowledge answers in Siri and Search. Apple says to opt out of those answers with the nosnippet robots meta tag on the content concerned. Disallowing Applebot-Extended only covers training. Apple also says pages marked isAccessibleForFree: false in structured data can still appear in search results but won't be used as that context, and that Applebot doesn't follow Crawl-delay.

The shared AI-training token: what's settled and what isn't

On October 30, 2026, Google said it will support __usage_ai-training, a robots.txt token from an IETF individual Internet-Draft, and Anthropic began recognizing a no-AI-training signal based on the same draft. The token isn't a standard yet, and it uses ordinary robots.txt syntax: you write a User-agent: __usage_ai-training group with Disallow or Allow rules. As of November 14, 2026:

  • Anthropic says it began recognizing a "no AI training" signal based on the IETF draft work on October 30, 2026, and is rolling it out across its services over time; its page describes the signal without naming the token. Rules that name ClaudeBot take priority over the shared signal.
  • Google says in its crawler documentation that it intends to support the token in early 2027. It will treat a shared-token opt-out as a Google-Extended disallow unless your file names Google-Extended separately.
  • OpenAI's crawler page doesn't mention the shared token; it still documents only GPTBot for training opt-outs.
  • Microsoft: we couldn't find a Microsoft statement on the token or a support date.

A User-agent: * group doesn't express this preference, and the token doesn't remove content already collected. Until every operator you care about supports it, keep your named training rules (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) and treat the shared group as an addition.

Four robots.txt examples for search and AI crawlers

Example A: allow public pages, avoid an internal search path

Assume your public site has articles and product pages, while /search/ contains internal search-result URLs you do not want cooperatively crawled. This file expresses that narrow preference. It does not remove already indexed URLs and it is not an access restriction.

User-agent: *
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

Do not paste the path into a site that uses /search/ for useful editorial pages. First list the URLs it would match. Keep scripts, stylesheets and images needed to understand public pages accessible unless you have a specific reason to restrict them.

Example B: permit ChatGPT search crawling and decline GPTBot crawling

OpenAI documents OAI-SearchBot for search and GPTBot for content that may be used in model training. Those settings are independent. The following policy keeps an internal search-path restriction for the explicitly named search bot and declines GPTBot access. Consult the official crawler reference when updating names or network rules.

User-agent: *
Disallow: /search/

User-agent: OAI-SearchBot
Disallow: /search/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Allowing search access does not guarantee a citation. It also does not control every user-triggered fetch: OpenAI describes ChatGPT-User separately and notes that robots rules may not apply to those requests. Do not treat one crawler group as a universal policy for every AI use. OpenAI says ChatGPT search can take about 24 hours to reflect a robots.txt change.

Example C: keep Google search crawling separate from Google-Extended

Google-Extended is a product-control token, not a separate HTTP crawler user-agent. Google's crawler documentation distinguishes it from Google Search and says it does not affect inclusion or ranking there. It addresses specified Gemini training and grounding uses, so do not describe it as a training-only switch.

User-agent: *
Disallow: /search/

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

Blocking Google-Extended doesn't keep your pages out of AI Overviews, which use Googlebot; our guide explains how AI Overviews choose sources.

Example D: allow AI search, decline AI training

This is the policy many publishers want: stay visible in AI search answers, and say no to model training. It names the search crawlers so their restrictions are explicit, puts the training tokens from OpenAI, Anthropic, Google and Apple in one group, and adds the shared training token as a forward-looking extra.

User-agent: *
Disallow: /search/

# AI search crawlers: keep the same restriction as everyone else
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /search/

# AI training: decline
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Shared AI-training preference (support dates differ)
User-agent: __usage_ai-training
Disallow: /

Sitemap: https://example.com/sitemap.xml

Several User-agent lines can share one group of rules; Google's robots.txt specification supports this, but test any other parser you rely on. The user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) fall under the * group here, and two of their operators say robots.txt may not bind them. Googlebot and Applebot also use the * group, so Google and Apple search are unaffected. CCBot isn't in the training group because Common Crawl's archive serves more than AI training; add a User-agent: CCBot line to that group if you want out of its future crawls too.

These are alternative starting policies. Do not stack them into production without resolving duplicate groups and retaining your current intentional restrictions. If several people manage the site, name one policy owner and keep a short explanation beside each non-obvious rule in your change record.

Understand group selection before adding exceptions

For Google, the most specific matching user-agent group is selected; a specific group does not simply inherit the wildcard group's rules. Within the applicable rules, path specificity matters, and an equally specific allow rule wins over a disallow rule. See Google's robots.txt interpretation; validate other crawlers against their own behavior rather than assuming universal parser identity.

This is why Examples B and D repeat Disallow: /search/ in the search-crawler groups. A file that adds a named group containing only Allow: / can unintentionally lose a restriction the author expected to inherit.

In one sample, ChatGPT suggested exactly that pattern. We ran the prompt "how do I block GPTBot" through ChatGPT via DataForSEO's ChatGPT scraper on November 14, 2026 (US English, logged out, one run). The answer cited only OpenAI's crawler page and proposed a User-agent: OAI-SearchBot group with just Allow: /. That is harmless on a site with no other rules. On a file like Example A, it would quietly let OAI-SearchBot into /search/. Answers vary by run, so treat this as one sample, not ChatGPT's fixed advice.

An envelope for the incoming crawler, a card file with one group card raised and a ruler for the path: identify the client, use its one matching group, then match the path.
Repeat intended restrictions in specific groups where needed; test the final policy.

Be precise about path boundaries. A prefix such as /search is broader than /search/. Case, trailing slashes and query patterns can change which URLs match. Before adding a wildcard, write at least one URL that should match and one nearby URL that must not match.

Build a small test matrix

For Example B, use the following expectations as a starting fixture. “Allowed” means the robots policy permits a fetch; the server could still return an error, require a login or block the request through a firewall.

URL pathGooglebotOAI-SearchBotGPTBot
/AllowAllowDisallow
/blog/guideAllowAllowDisallow
/search/resultsDisallowDisallowDisallow
/media/diagram.webpAllowAllowDisallow

For Example D, we ran the file through two open-source parsers (Protego 0.7.0 and Python 3.13's urllib.robotparser), querying each token name rather than a full user-agent string, on November 6, 2026. Both returned these results:

URL pathGooglebotOAI-SearchBot, Claude-SearchBot, PerplexityBotGPTBot, ClaudeBot, Google-Extended, Applebot-ExtendedClaude-User (falls under *)
/AllowAllowDisallowAllow
/blog/guideAllowAllowDisallowAllow
/search/resultsDisallowDisallowDisallowDisallow
/media/diagram.webpAllowAllowDisallowAllow
A clipboard of checked test cases beside a rack sorting sample URLs into allowed and disallowed: test the expected results before the policy goes live.
Expected robots-policy results do not prove that a firewall or application permits the request.

Extend the fixture with your most important product URL, one language variant and any custom path affected by a new rule. Store the expected result before running a parser. Otherwise, it is too easy to approve whatever result the tool returns.

There are two different tests here: parsing the policy and making an HTTP request. A successful browser request does not prove a bot is permitted. A parser's “allow” result does not prove your CDN lets the bot through. Test both layers, and verify real crawler identity from vendor guidance before creating firewall exceptions. If your site runs on Cloudflare or Vercel, check what your platform's AI crawler data shows: it records which crawlers were allowed or blocked at the edge.

When robots.txt isn't enough

robots.txt is a request to cooperating crawlers, not a lock. User-triggered fetchers such as ChatGPT-User and Perplexity-User visit a page because someone asked, and their operators say robots.txt rules may not apply or are generally ignored. Anyone can also send a bot's user-agent string. To enforce a block:

  1. Decide whether you really need one. A hard block on a user-triggered fetcher also stops that assistant from reading your page when a customer asks about you.
  2. Verify identity first. OpenAI, Anthropic and Perplexity publish IP address lists for their crawlers (linked from the pages above); match the request's source IP, not just its user-agent.
  3. Enforce at the edge with a firewall or CDN rule, for example Cloudflare's AI Crawl Control, which enforces each block with a WAF custom rule, and log what it blocks.
  4. Keep robots.txt readable. Anthropic notes that blocking its IP addresses can stop it from reading your robots.txt, so an IP block alone may not work as a lasting opt-out.

Deploy and verify the correct file

Publish the plain UTF-8 file at /robots.txt on the exact host and protocol you intend to control. Google's creation guide covers file placement and validation options. A file on the main domain does not automatically configure every subdomain. For a new site, our launch-day SEO checklist tests for a staging Disallow: / left in production.

  1. Save the current file and record the reason for the change.
  2. Review the diff against the expected URL matrix. A read-only Skill that reviews robots.txt changes can do a first pass, but a person signs off.
  3. Fetch the live file in a private browser session and inspect its text, status and content type. To see a page's robots rules while you browse, our guide to SEO Chrome extensions lists one whose store listing says it reports on AI bots, with the site access it requests.
  4. Confirm it is not an HTML error page or cached older version.
  5. Run policy tests, then test the important public pages and assets. Our GEO audit checklist repeats this check across a whole site.
  6. Check the robots.txt report and relevant URL Inspection results in Search Console where you have access.

Once the file is live, the rest of the setup (sitemap submission, Bing Webmaster Tools and IndexNow) is in our post-launch SEO checklist.

Keep the previous version available for rollback. If a release unexpectedly blocks product pages, restore the last reviewed policy, verify the restored response, and investigate the error before retrying. Do not “fix” it by granting every automated request access to private application routes.

robots.txt on WordPress

WordPress serves a virtual robots.txt: there is no file on disk until you add one. When a request for /robots.txt reaches WordPress, its do_robots() function writes the file on the fly. Upload a real robots.txt to the site root and the web server sends that file instead. You can also change the virtual file with the robots_txt filter. As of November 14, 2026, a standard install with pretty permalinks and search engines allowed outputs this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

The admin paths come from your admin URL. The Sitemap line comes from WordPress's built-in sitemaps (since version 5.5) and only appears while the site is public. WordPress only adds its robots.txt rewrite when it is installed at the domain root. A plugin or your host may also change this output, so read the live file rather than trusting the defaults.

Three ways to edit it

MethodHow it worksWatch out for
Upload a physical filePlace robots.txt in the web root. WordPress's standard Apache rules only route requests for files that don't exist, so the server returns your file.Filters and plugin settings that edit the virtual file stop having any effect. Copy the Sitemap line in yourself.
The robots_txt filterA few lines in a small plugin or a child theme append or replace the generated text. The filter receives the output and whether the site is public.Only works while no physical file exists. Test the live URL after each deploy.
An SEO plugin's editorSome SEO plugins add a robots.txt editor to the admin screens.Follow that plugin's own documentation, and check whether it writes a physical file or filters the virtual one.

For example, this filter adds the decline-training group from Example D to the virtual file, and only while the site is public:

add_filter( 'robots_txt', function ( $output, $public ) {
    if ( $public ) {
        $output .= "\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: Google-Extended\nUser-agent: Applebot-Extended\nDisallow: /\n";
    }
    return $output;
}, 10, 2 );

Then open https://your-site/robots.txt in a private window and run the test matrix above against the result.

The "Discourage search engines" setting

The checkbox under Settings → Reading, "Discourage search engines from indexing this site", no longer writes Disallow: / into robots.txt. Since WordPress 5.3 it adds a noindex robots meta tag to pages instead, and the virtual file drops its Sitemap line. WordPress's own help text warns that "not all search engines adhere to this directive." Treat it as a staging switch, not access control. It also tells AI crawlers nothing about training.

"Indexed, though blocked by robots.txt" on WordPress

Search Console shows this warning when Google indexed a URL it wasn't allowed to crawl, usually because other pages link to it. Google's help says robots.txt "is not the correct mechanism" to keep a page out of results. To remove it, delete the robots.txt rule and serve noindex, for example through your SEO plugin's per-page setting. Some blocks are intentional, such as internal search pages. For those, Google notes that fixing every blocked URL may not make sense.

What our own robots.txt says

As of November 14, 2026, polydraft.ai/robots.txt allows every crawler and names seven AI tokens, all allowed: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended. That includes training tokens, so our site currently allows AI training as well as AI search.

polydraft.ai robots.txt: User-agent * Allow /, then a comment reading AI Search Engine Crawlers above Allow / groups for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended, and a Sitemap line for https://polydraft.ai/sitemap.xml.
polydraft.ai/robots.txt, captured November 6, 2026 (text enlarged for legibility); the file was unchanged when we rechecked it on November 14.

Measured against the table above, the file has two flaws. The comment # AI Search Engine Crawlers sits above GPTBot, ClaudeBot and Google-Extended, which are training tokens, not search crawlers. And Claude-User, Perplexity-User, Applebot-Extended and CCBot aren't named; they fall under the * group, which allows them. Neither changes what crawlers may fetch, but a mislabelled file makes the next policy change harder to review.

We make PolyDraft, an AI website builder. Sites built with PolyDraft serve the same robots.txt as ours: a fixed allow-all preset that names the seven AI tokens above and links the sitemap. As of November 14, 2026, PolyDraft has no setting to switch that preset to a decline-training policy like Example D. Because the site runs on your own Cloudflare account, you can block individual AI crawlers in Cloudflare's AI Crawl Control; PolyDraft's own admin panel doesn't edit robots.txt. Cloudflare's managed robots.txt can also add Disallow rules for training crawlers to the file your site serves, so test the delivered file afterwards, as in the release checks above. PolyDraft is not WordPress, so the WordPress steps above don't apply to it.

Frequently asked questions

Does Disallow remove a page from Google?

No. It controls crawling by compliant clients. Search exclusion requires the appropriate indexing controls, and Google must be able to see them. Private material needs real access controls rather than a publicly visible list of paths in robots.txt.

Should I block every AI crawler?

That is a business policy choice. First distinguish search discovery, training preferences and user-requested access. Then decide per documented client and use. A blanket rule may conflict with your wish to appear in a search product; an allow rule still promises no placement.

What's the difference between GPTBot, OAI-SearchBot and ChatGPT-User?

GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot crawls for ChatGPT search results. ChatGPT-User fetches a page when a user asks ChatGPT about it, and OpenAI says robots.txt rules may not apply to it. Each setting is independent, so you can allow search and decline training.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended doesn't affect inclusion or ranking in Google Search, and AI Overviews and AI Mode use Googlebot. Blocking Googlebot would remove you from both Search and its AI features.

Can a generator guarantee the file is correct?

No. A generator can format rules, but it does not know every path on your site or your intended access policy. Review its output, test representative URLs and confirm the live response after deployment. Keep the configuration as small as the policy allows.

What is OAI-SearchBot?

OAI-SearchBot is OpenAI's search crawler. It surfaces websites in ChatGPT's search features. If your robots.txt disallows it, OpenAI says your site won't be shown in ChatGPT search answers, though it can still appear as a navigational link. It is separate from GPTBot, which is for training, and changes take about 24 hours to apply.

What is CCBot, and does blocking it stop AI training?

CCBot is Common Crawl's crawler. It builds a free, public web archive that anyone can download, and Common Crawl says it obeys robots.txt, including Crawl-delay. Blocking it stops future crawls for that archive. It doesn't control copies other parties already hold, and it isn't a substitute for GPTBot, ClaudeBot or Google-Extended rules.

Where is the robots.txt file in WordPress?

By default there isn't one on disk. WordPress generates a virtual robots.txt at /robots.txt when no physical file exists. You can upload a real file to the site root, which replaces the virtual one, or change the virtual file with the robots_txt filter or an SEO plugin.

Turn the policy into a tested artifact

Draft your file with the PolyDraft robots.txt generator, check each token's purpose against the table above, then apply the checks in this guide. If the file is correct but assistants still cannot read a page, continue with our guide to checking website readability for AI assistants.

After testing your rules, check the HTML a permitted crawler receives.

If you also maintain an optional AI-facing page index, validate it separately from access rules. Read the llms.txt example and validation.

If your site runs on Wix or Webflow rather than WordPress, see which robots.txt settings each platform lets you change in our guides to Wix SEO and Webflow SEO.

Sources

Sources were checked on November 14, 2026. We read the OpenAI, Anthropic, Google and Cloudflare pages, Google's Search Console help and WordPress's source code directly that day. On November 20, 2026, we reread the OpenAI, Anthropic and Google crawler pages and read the Perplexity and Apple pages and Google's user-triggered fetchers page. The Common Crawl, Bing, WordPress Developer Resources and IETF pages couldn't be opened from our tools, so those rows rest on search excerpts of the official pages from November 14 and our earlier readings; the table marks which is which. Operators change these pages often, so recheck before you rely on a token.

Revision history: November 6, 2026: added AI crawler purposes, the shared AI-training token and an example that allows AI search but declines training. November 14, 2026: rechecked every AI crawler against its operator's docs, added CCBot and Bingbot to the table, added WordPress steps, and corrected who supports the shared AI-training token.

About this article: Prepared with AI assistance for PolyDraft, checked by an automated QA script and a separate AI fact-check, and approved by Vincent Chen. Worked examples and diagrams are illustrative, not customer results. The Example D parser results come from a run on November 6, 2026; the OpenAI GPTBot and robots.txt screenshots were captured from public pages that day, and the OAI-SearchBot screenshot on November 14, 2026. The ChatGPT answer was collected through DataForSEO's ChatGPT scraper on November 14, 2026, one run, logged out. The hero is an AI-generated editorial illustration. For corrections, contact PolyDraft with the page URL and supporting source.

Founder of PolyDraft

Vincent Chen is the founder of PolyDraft and CEO of BlackMonolith, Inc. He has worked on technical SEO for multiple products and websites, and runs polydraft.ai on PolyDraft.

Put your site’s SEO and GEO on autopilot

PolyDraft builds your website, then finds the keywords you can win every week and writes the content.

Start your site

Start with a brief.
Publish when it’s right.

Build and preview your whole site for free. Every week after launch, PolyDraft finds the keywords you can win and writes the content.

Free to build and preview. Pay only to publish.