Build a valid robots.txt file with
per-bot crawl rules, one-click AI crawler blocking, and sitemap links — then download it straight to your
site root.
Open 24 built-in tools from your browser sidebar, use quick actions from the right-click menu, and reach every RizCraft tool without hunting through tabs.
Firefox and Edge versions are coming next. Free to install
Our free robots.txt generator builds a syntactically valid robots.txt file
in seconds — no need to memorize directive syntax or hand-edit plain text. Add rule groups for specific
crawlers, block sensitive paths, list your sitemap, and download a file ready to drop into your site's
root directory.
robots.txt is a plain-text file placed at the root of your domain
(https://example.com/robots.txt) that tells well-behaved web crawlers which parts of your
site they may or may not request. It uses a small set of directives — User-agent,
Disallow, Allow, and Sitemap — grouped into rule blocks that apply
to specific bots or to all crawlers via the * wildcard.
It's important to understand that robots.txt is an advisory, not a security control.
Compliant crawlers (Googlebot, Bingbot) respect it, but nothing stops a malicious scraper from ignoring
it entirely — never rely on robots.txt alone to protect private data.
Since the rise of large language models, a growing list of AI companies operate crawlers that scrape web
content for model training or live retrieval — OpenAI's GPTBot and
ChatGPT-User, Common Crawl's CCBot (a major source for open training datasets),
Google's Google-Extended (governs Gemini/AI training, separate from regular Search
indexing), Anthropic's ClaudeBot and anthropic-ai, and Perplexity's
PerplexityBot, among others. Use the "Block AI Crawlers" preset above to
instantly add a disallow-all rule group for the most common ones, or add/remove individual user-agents to
fit your policy.
* for
everyone) with its own Disallow/Allow paths./robots.txt.The Crawl-delay directive asks a bot to wait N seconds between requests. Bingbot and several
other crawlers honor it, but Google's crawlers ignore it entirely — Google recommends adjusting crawl
rate through Search Console instead. It's included here for completeness and for crawlers that do
respect it.
https://example.com/robots.txt. A copy inside a subfolder like
/blog/robots.txt is ignored by crawlers.
noindex meta tag or HTTP header instead — but note that Googlebot must be able to crawl
a page to see a noindex tag, so don't combine both on the same URL.
Tell us what to improve or which tool to add next.