Free tool
Free robots.txt generator
Build a valid robots.txt in seconds, with one-click controls for AI crawlers like GPTBot, ClaudeBot and PerplexityBot alongside classic search engines. It runs entirely in your browser — no sign-up, no login, nothing sent to a server.
Search crawlers
AI crawlers
Tick a bot to welcome it explicitly; untick to block it with a Disallow rule.
robots.txt preview
User-agent: * Disallow:
Reminder: robots.txt controls crawling, not indexing or access. Never use it to hide sensitive URLs — the file itself is public.
The 2026 decision
Allow or block AI crawlers — but decide on purpose
Allowing AI crawlers makes your brand citable when buyers ask an assistant instead of searching. Blocking them protects proprietary content from being absorbed. Neither is universally right, but leaving the decision to a template you copied years ago is universally wrong.
A simple rule of thumb: if your business benefits from being found and quoted, allow them. If your content is the product — paid research, subscription journalism, proprietary datasets — block them.
| Crawler | What it does | Allow | Block |
|---|---|---|---|
| GPTBot | Crawls pages for OpenAI model training and retrieval | User-agent: GPTBot Allow: / | User-agent: GPTBot Disallow: / |
| OAI-SearchBot | Fetches pages for OpenAI search-style answers | User-agent: OAI-SearchBot Allow: / | User-agent: OAI-SearchBot Disallow: / |
| ClaudeBot | Crawls pages for Anthropic systems | User-agent: ClaudeBot Allow: / | User-agent: ClaudeBot Disallow: / |
| PerplexityBot | Indexes pages for Perplexity answers and citations | User-agent: PerplexityBot Allow: / | User-agent: PerplexityBot Disallow: / |
| Google-Extended | Controls Google AI use without affecting Search indexing | User-agent: Google-Extended Allow: / | User-agent: Google-Extended Disallow: / |
| CCBot | Common Crawl, a public dataset used by many AI systems | User-agent: CCBot Allow: / | User-agent: CCBot Disallow: / |
Every site PolyDraft builds welcomes GPTBot, ClaudeBot and PerplexityBot by default and publishes an llms.txt file — build one for any site with the free llms.txt generator.
Primer
What a robots.txt file actually does
robots.txt is a plain text file at your site root that tells crawlers which URLs they may request.
What it does not do matters just as much: it is not a security mechanism, because the file is public and compliance is voluntary. It does not reliably remove pages from search results either — a blocked URL can still be listed if other sites link to it, and a noindex directive placed inside robots.txt is not honored. For genuine removal, use a noindex meta tag on a crawlable page or proper access control.
The four building blocks
- User-agent
- Names the crawler the following rules apply to. * means every crawler.
- Disallow
- Blocks a path prefix. An empty Disallow value blocks nothing.
- Allow
- Carves an exception out of a broader Disallow rule.
- Sitemap
- Points crawlers at your XML sitemap. It is independent of any user-agent block.
How to use your robots.txt file
- 01
Configure your rules
Set the crawler policy, add any paths to block and enter your sitemap URL in the generator above.
- 02
Copy or download the file
The preview updates live. Copy it or download robots.txt directly to your machine.
- 03
Upload to your site root
The file must resolve at yourdomain.com/robots.txt — subdirectories are ignored by crawlers.
- 04
Verify and test
Load the URL in a browser to confirm it serves plain text, then check it in your search console’s robots.txt report.
PolyDraft generates this file — and keeps the sitemap reference correct — on every site it builds. See how it works.
Common rules, ready to copy
Allow everything
The correct default for most public sites.
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
Block a private folder
Keep admin or cart URLs out of crawl paths.
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/
Welcome AI crawlers explicitly
You want to be citable in AI answers.
User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: /
Block AI crawlers only
Your content is proprietary and should not train models.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: * Disallow:
Block a staging site entirely
A non-production site that must never be indexed.
User-agent: * Disallow: /
robots.txt questions
What is a robots.txt file?
robots.txt is a plain text file at your site root that tells crawlers which URLs they may request. It contains blocks of User-agent, Disallow and Allow directives, plus an optional Sitemap line pointing at your XML sitemap.
What should be in my robots.txt file?
For most public sites: a User-agent: * block that disallows nothing, an explicit decision about AI crawlers, disallow rules for private areas such as carts and internal tools, and a Sitemap line. Keep it short — every rule is a chance to block something by accident.
Do I need a robots.txt file at all?
A site works without one, but you should have it. Without a robots.txt file you have made no decision about crawler access, and you have nowhere to declare your sitemap. A permissive file is better than a missing one.
Why is robots.txt important for SEO?
It governs where crawl budget goes and keeps low-value URLs out of the crawl path. It is not a security or de-indexing tool: a blocked page can still appear in results if other sites link to it, and a noindex directive inside robots.txt is not honored by Google.
How do I allow or block AI crawlers like GPTBot and ClaudeBot?
Add a User-agent block naming each bot, then Allow: / to welcome it or Disallow: / to block it. Allowing them makes your content citable in AI answers; blocking them protects proprietary material. Make the choice deliberately — the default is whatever your host shipped.
Where do I put robots.txt and how do I test it?
It must sit at the root of the domain so it loads at yourdomain.com/robots.txt. Test it by opening that URL in a browser and by using the robots.txt report in Google Search Console, which shows how your rules are interpreted.
Built into every site PolyDraft ships
An AI-crawler-friendly robots.txt, an llms.txt file and answer-first content structure — generated automatically alongside the full technical SEO layer. This very site was built that way.