Create a robots.txt file in under a minute. Pick a template or add your own User-agent, Allow, Disallow and Crawl-delay rules, add your sitemap, then copy or download the file. Free, no account, and it works for search crawlers and AI bots like GPTBot and ClaudeBot.
配置你的 robots.txt 指令
常见使用场景的快速模板
你的 robots.txt 文件已准备就绪，可供下载
配置指令并生成以查看 robots.txt 输出内容
A robots.txt generator is a tool that writes the robots.txt file for you: the plain-text file at yourdomain.com/robots.txt that tells crawlers which paths they may fetch. In this free robots.txt generator you add User-agent, Allow and Disallow rows, an optional Crawl-delay and your sitemap URL, then copy or download the file. It runs in your browser, with no account.
Every rule is a row you can switch on or off, so you can try a stricter file without deleting anything.
Who the rules are for: * for every crawler, or a named bot such as Googlebot, Bingbot, GPTBot or ClaudeBot. Add as many as you need.
The paths crawlers may or may not fetch, such as /admin/ or /checkout/. Each enabled rule is written under every user-agent you list.
Seconds between requests, for crawlers that honour it. Off by default; Google ignores this directive.
The full URL of your XML sitemap, added at the end of the file so crawlers find every page.
Standard website, block all crawlers, and API-heavy site. Load one, then edit the rows.
Copy the generated text or download it as robots.txt, ready to upload to your site root.
Build the rules, generate the file, and upload it to the root of your domain.
Load Standard website, Block all crawlers or API-heavy site, or start from the default rows.
Use * for every crawler, or name bots such as Googlebot, GPTBot or ClaudeBot. Every enabled rule is written under each user-agent you list.
List the folders crawlers should skip, such as /admin/ or /checkout/, and any exceptions they may still fetch. Switch rows on or off to compare versions.
Enter the full sitemap URL, then click Generate robots.txt.
Copy the text or download robots.txt, upload it so it loads at https://yourdomain.com/robots.txt, then test the live file.
The generated file is a list of groups. Each group starts with one or more User-agent lines, followed by the rules for those crawlers.
The group applies to every crawler that has no group of its own. A crawler follows only the most specific group that names it.
Crawlers in this group should not fetch URLs that start with /admin/. An empty Disallow line means nothing is blocked.
An exception inside a blocked folder. Google and Bing apply the most specific (longest) matching rule.
Blocks the whole site for that group. Use it under a named bot, such as GPTBot, rather than under *.
Asks a crawler to wait 10 seconds between requests. Bing honours it; Google ignores it.
Points every crawler to your XML sitemap. It must be a full URL and can appear anywhere in the file.
Before a well-behaved crawler fetches a page, it reads /robots.txt at the root of the host. The file says which paths each crawler may request. It controls crawling, not indexing: a blocked URL can still appear in Google if other sites link to it. To keep a page out of search results, let it be crawled and add a noindex tag instead.
Each subdomain needs its own file. https://blog.example.com/robots.txt is separate from https://example.com/robots.txt, and a file in a subfolder is ignored.
Most sites need only a few lines: one group for all crawlers that blocks private or duplicate areas, and a Sitemap line. Add a named group only when one crawler needs different rules.
A common starting point for a blog or business site, including a custom robots.txt for Blogger or WordPress:
AI companies publish the user-agent names their crawlers use, so you can allow or block each one. GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity) and CCBot (Common Crawl) each read their own group, and Google-Extended controls whether Google may use your content for Gemini.
Blocking a crawler is a trade-off. It keeps your pages out of that company's crawls, but it can also keep them out of the answers its assistant builds from crawled pages. Decide per bot, and check later which bots actually visit.
我们的免费 robots.txt 生成工具可帮助你为网站创建正确的 robots.txt 文件。 Robots.txt 文件对于控制搜索引擎如何抓取你的网站至关重要，并且会 显著影响你的 SEO 表现。
要生成 robots.txt 文件，只需选择您希望允许或禁止的机器人。 我们的 robots.txt 生成器随后会为您创建代码，供您复制和粘贴。
随着 AI 代理的兴起，阻止 GPTBot、ClaudeBot 和 OpenAI-SearchBot 等 AI 抓取工具 比以往任何时候都更加重要，以保护您的知识产权。我们的工具可帮助您轻松 生成针对这些现代 AI 爬虫定制的指令。
Something else? Ask the team
If Googlebot is blocked from the whole site, it stops crawling, and pages can drop out of results or show without a description over time. Check every change before you upload it, and after you deploy, open the robots.txt report in Google Search Console to see the version Google fetched and any errors.
It depends on what you want. Blocking GPTBot, ClaudeBot, PerplexityBot or CCBot keeps those crawlers away from your pages, but it can also keep your content out of the answers their assistants give. Many sites allow the crawlers that power AI search and block only the ones they don't want. Decide per bot.
Yes. Add User-agent rows for the bots, such as GPTBot, ClaudeBot, PerplexityBot or CCBot, and a Disallow: / row, then generate. Every rule applies to every user-agent in that run, so generate the AI-bot group and your * group separately and combine them in one file. Reputable AI crawlers follow these rules; block bots that ignore robots.txt at your server or firewall.
A robots.txt file is a plain-text file at the root of your domain (yourdomain.com/robots.txt) that tells search engine and AI crawlers which pages they may and may not access. It is one of the first files crawlers request when they visit your site, and it plays a key role in both crawl budget management and AI bot access control.
No. robots.txt prevents crawling, not indexing. If other sites link to a blocked page, Google can still index it from those external links without crawling your page. To prevent indexing, use a noindex meta tag on the page itself — robots.txt and noindex solve different problems.
Crawl budget is the number of pages Googlebot will crawl on your site within a given timeframe. Blocking low-value pages in robots.txt — admin panels, duplicate content, faceted navigation URLs — redirects that budget toward your important content, helping them get crawled and indexed more frequently.
Never. Google needs to render your pages with all CSS and JavaScript assets to understand your content. Blocking stylesheets or scripts prevents Googlebot from seeing your pages as users see them, which can significantly hurt rankings by making pages appear sparse or broken to the crawler.
robots.txt must be in the root directory of your domain: https://yourdomain.com/robots.txt. It cannot live in a subdirectory. Each subdomain needs its own separate robots.txt — for example, blog.yourdomain.com/robots.txt is separate from yourdomain.com/robots.txt.
Pick one that lets you set rules per user-agent, add your sitemap and download a ready file without signing up. This robots.txt generator does all three and is free. Test the result with a robots.txt checker before you upload it.
Choose a template or add rows for User-agent, Allow and Disallow, add your sitemap URL, then click Generate. Copy the text or download robots.txt and upload it to the root of your domain, so it loads at https://yourdomain.com/robots.txt.
Yes. It needs no account and runs in your browser. Nothing you type is sent to our servers.
Yes. Build the rules here, then paste them into Blogger's custom robots.txt setting, or upload the file to your WordPress site root (or paste it into your SEO plugin's robots.txt editor).
Check the file after you upload it, and give AI engines a map of your best pages.
Test a live robots.txt for errors and blocked bots.
Create an llms.txt file for your site.
Validate an llms.txt file against the spec.
Crawl a site and export a sitemap.xml.
Check a sitemap for errors before you submit it.
Simulate how AI crawlers and agents read a page.
Score a page for search engines and AI answers.
A robots.txt file states the rules. AI Rank Lab shows which AI bots visit, which pages they read, and whether your pages are ready to be cited.
Add one tracking tag and see visits from AI crawlers and from people sent by AI assistants, page by page. Pair it with the free audit, which checks your robots.txt, sitemap and llms.txt along with the page itself.
AI Bots Tracker
Also records visits referred by AI assistants, per page.
Paste your site. The free audit scores it for SEO, answer engines and generative engines, with the fixes.
