/
Products
SideDock
Desktop Side Utility
Build a valid robots.txt file with
per-bot crawl rules, one-click AI crawler blocking, and sitemap links — then download it straight to your
site root.
Our free robots.txt generator builds a syntactically valid robots.txt file
in seconds — no need to memorize directive syntax or hand-edit plain text. Add rule groups for specific
crawlers, block sensitive paths, list your sitemap, and download a file ready to drop into your site's
root directory.
robots.txt is a plain-text file placed at the root of your domain
(https://example.com/robots.txt) that tells well-behaved web crawlers which parts of your
site they may or may not request. It uses a small set of directives — User-agent,
Disallow, Allow, and Sitemap — grouped into rule blocks that apply
to specific bots or to all crawlers via the * wildcard.
It's important to understand that robots.txt is an advisory, not a security control.
Compliant crawlers (Googlebot, Bingbot) respect it, but nothing stops a malicious scraper from ignoring
it entirely — never rely on robots.txt alone to protect private data.
Since the rise of large language models, a growing list of AI companies operate crawlers that scrape web
content for model training or live retrieval — OpenAI's GPTBot and
ChatGPT-User, Common Crawl's CCBot (a major source for open training datasets),
Google's Google-Extended (governs Gemini/AI training, separate from regular Search
indexing), Anthropic's ClaudeBot and anthropic-ai, and Perplexity's
PerplexityBot, among others. Use the "Block AI Crawlers" preset above to
instantly add a disallow-all rule group for the most common ones, or add/remove individual user-agents to
fit your policy.
* for
everyone) with its own Disallow/Allow paths./robots.txt.The Crawl-delay directive asks a bot to wait N seconds between requests. Bingbot and several
other crawlers honor it, but Google's crawlers ignore it entirely — Google recommends adjusting crawl
rate through Search Console instead. It's included here for completeness and for crawlers that do
respect it.
It must sit at the root of your domain, e.g.
https://example.com/robots.txt. A copy inside a subfolder like
/blog/robots.txt is ignored by crawlers.
Not reliably. Disallowing a URL stops Google from
crawling it, but if the URL is already linked elsewhere it can still appear in search
results without a description. To fully remove a page from indexing, use a
noindex meta tag or HTTP header instead — but note that Googlebot must be able to crawl
a page to see a noindex tag, so don't combine both on the same URL.
Yes, though most crawlers merge them. It's clearer to keep one rule group per user-agent with all its Disallow/Allow rules listed together, which is how this generator structures its output.
It's completely free, and the entire file is generated client-side in your browser — nothing you type is sent to or stored on any server.
Tell us what to improve or which tool to add next.