robots.txt

Definitionrobots.txt is a text file at the root of your site that tells crawlers which paths they may or may not fetch, rule by rule for each user agent.

It controls crawling, not indexing: a blocked URL can still appear in Google if other pages link to it. To keep a page out of results, allow crawling and use noindex. robots.txt is also where you choose which AI crawlers to allow, for example blocking training crawlers like GPTBot while allowing search crawlers like OAI-SearchBot.

Well-behaved crawlers follow it, but some user-triggered fetchers (Perplexity-User, and in some cases ChatGPT-User) say robots.txt may not apply. Firewalls and CDNs can also block crawlers that robots.txt allows, so check both.

Learn it step by step

• AI crawlers vs AI-referred visitors (AEO and GEO fundamentals)
• Structured data and llms.txt: what they do and don't do (AEO and GEO fundamentals)

Related terms

• AI crawler: An AI crawler is a bot run by an AI company that fetches web pages, for model training, for an AI search index, or because a user asked the assistant to read a page.
• GPTBot: GPTBot is OpenAI's web crawler for training its generative AI foundation models. Blocking it in robots.txt signals that your content should not be used for training.
• XML sitemap: An XML sitemap is a file that lists the URLs on your site you want search engines to discover and crawl, optionally with the date each was last changed.
• llms.txt: llms.txt is a proposed Markdown file at the root of a website (/llms.txt) that gives AI tools a short summary of the site and a list of its key pages.