Robots.txt generator and tester for AI crawlers.

robots.txt is a plain text file at the root of a host, such as https://example.com/robots.txt, that tells crawlers which paths they may fetch. It was standardised as the Robots Exclusion Protocol in RFC 9309 in 2022, after nearly three decades as an informal convention. Each protocol, host and port needs its own file, so a subdomain does not inherit the main site's rules.

It controls crawling, not indexing. A disallowed URL can still appear in search results, without a description, if other pages link to it. To keep a page out of search, allow crawling and add a noindex robots meta tag or header, because a crawler has to fetch the page to see noindex. robots.txt is also not access control: the file is public and only well-behaved bots obey it, so protect private areas with authentication.

A file is a series of groups. Each group starts with one or more User-agent lines followed by Allow and Disallow rules. A crawler uses the group that names it most specifically and ignores the rest; only if no group names it does it use the * group. That catches many people out: if you add a group for Googlebot, Googlebot no longer reads the rules under *.

Within that group, the rule with the longest matching path wins, and if an Allow and a Disallow are equally long, Allow wins. Paths are case-sensitive and match from the start. * matches any sequence of characters and $ anchors the end, so Disallow: /*.pdf$ blocks every URL ending in .pdf. The tester applies exactly these rules and shows which line decided the result.

Google ignores Crawl-delay; Bing and some other crawlers honour it. Google reads at most 500 KiB of the file, and unrecognised lines are skipped, which the status bar flags so typos do not fail silently.

AI companies now run separate agents for separate purposes, and the difference matters. Training crawlers collect content that may be used to train models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, whose dataset is widely used for training), Meta-ExternalAgent, Amazonbot and Bytespider. Google-Extended and Applebot-Extended are not separate crawlers but tokens: Google's regular crawlers fetch the page, and the token decides whether the content may be used for Gemini training and grounding, or for Apple's foundation models.

Search crawlers index pages so an assistant can cite and link them: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude and PerplexityBot for Perplexity. Blocking these removes you from those answers. User-triggered fetchers, ChatGPT-User, Claude-User and Perplexity-User, load a page because a person asked about it. OpenAI and Perplexity say robots.txt may not apply to these; Anthropic says Claude-User honours it.

A common middle ground, and the default preset here, is to block training crawlers and allow search and user agents: your content is not used to train models, but you can still be cited and linked. Blocking Google-Extended does not affect Google Search rankings or crawling.

The most damaging mistake is shipping a staging file with Disallow: / to production, which asks every search engine to stop crawling the whole site. Others include blocking CSS and JavaScript files that Google needs to render pages, blocking a page you want deindexed (so Google never sees its noindex), relying on robots.txt to hide sensitive URLs (the file advertises them), and forgetting that rules are case-sensitive, so /Admin and /admin are different paths.

Crawlers cache robots.txt, typically for up to a day, so changes are not instant. If the file returns a 4xx error, crawlers treat the site as unrestricted; persistent server errors make Google pause crawling. Make sure the file returns 200 with a text/plain content type.

Sitemap lines can appear anywhere in the file and apply to all crawlers. List the absolute URL of every sitemap or sitemap index; it is the easiest way to help search engines discover new pages on a large site. They complement, not replace, submitting sitemaps in Google Search Console and Bing Webmaster Tools.

You may also see llms.txt, a proposed Markdown file that summarises a site for language models. It is a proposal rather than a standard, and the major AI crawlers do not document using it as a permission mechanism, so it does not replace robots.txt rules. See generative engine optimization for how sites are approaching AI search more broadly.

Paste your current live file into the text pane to test it as-is, or build a new one with the form. Then try the URLs that matter: the home page, a product or article, an admin path, a URL with tracking parameters, a PDF. The row of crawler chips shows the verdict for every bot at once, which makes it easy to spot a rule that blocks more than you intended.

After uploading, open yourdomain.com/robots.txt to confirm it is served, and check Google Search Console's robots.txt report, which shows the version Google fetched and any errors it found. For a full crawlability review of a site, our technical SEO and SEO services pages explain what else to check.

These three are often confused. robots.txt works at the level of URL paths and decides whether a crawler may fetch a page. The robots meta tag, placed in a page's head, and the X-Robots-Tag HTTP header, which also works for PDFs and images, tell search engines what to do with a page they have fetched: noindex keeps it out of results, nofollow asks them not to follow its links, and nosnippet or max-snippet limit how much of it appears in results.

Use robots.txt to keep crawlers out of areas that waste crawl capacity, such as internal search results, filtered listing URLs and cart pages. Use noindex for pages that must stay reachable but should not appear in search, such as thank-you pages or thin tag archives. Never combine a robots.txt block with noindex on the same URL, because the crawler cannot fetch the page to see the noindex.