Ctrl + K
Developer

Robots.txt Generator

Generate robots.txt files with custom rules for search engine crawlers, sitemaps, and crawl delays. Visual builder with live preview — no server needed.

Sitemaps
Preview
User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /

User-agent: Googlebot
Crawl-delay: 10
Allow: /
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

All generation runs entirely in your browser. No data is sent anywhere.

A robots.txt file is your website's first line of communication with search engine crawlers — telling them which pages to index, which to ignore, and how fast they should crawl. Getting it wrong can accidentally de-index your entire site from Google, while getting it right can dramatically improve crawl efficiency and server load. Our free Robots.txt Generator gives you a visual builder to create well-structured robots.txt files without memorizing the syntax. Configure rules for specific user agents (Googlebot, Bingbot, ChatGPT, and more), set per-agent crawl delays, define allow/disallow paths, and include your sitemap URLs — all with a live preview that updates as you type.

How to Use the Generator

  1. Select a user agent — Pick from preset crawler names like Googlebot, Bingbot, GPTBot, or use * for all crawlers. Different rules for different bots is fully supported.
  2. Add allow/disallow paths — Define which URL paths each crawler can access. Use Disallow: /admin/ to block sensitive areas, or Allow: / to permit everything by default.
  3. Set crawl delays — Optionally specify a crawl-delay in seconds per bot to limit server load from aggressive crawlers.
  4. Add sitemap URLs — Include your XML sitemap URLs so crawlers can discover all your important pages efficiently.
  5. Copy or download — The live preview shows the final robots.txt output. Copy to clipboard or download as a ready-to-deploy robots.txt file.

Key Features

  • 16 Preset User Agents

    Quick-select from Googlebot, Bingbot, GPTBot, DuckDuckBot, Baidu, Yandex, Applebot, and more.

  • Multi-Agent Rules

    Create separate rule groups for different crawlers — Googlebot gets one set of rules, Bingbot gets another.

  • Crawl Delay Control

    Set per-agent crawl delays to manage server load. Supported by Bing, Yahoo, and Yandex.

  • Sitemap Integration

    Add multiple sitemap URLs — the standard way to tell crawlers where your XML sitemaps live.

  • Live Preview

    See the exact robots.txt output update in real-time as you add, edit, or remove rules.

  • Copy & Download

    One-click copy to clipboard or download as a valid robots.txt file ready for your site's root directory.

Frequently Asked Questions

Do all crawlers respect robots.txt?
Reputable search engines (Google, Bing, Yahoo, Yandex, Baidu) fully respect robots.txt. However, malicious bots, scrapers, and some AI training crawlers may ignore it. Robots.txt is a voluntary standard — it works as a request, not an enforcement mechanism. For sensitive content, use proper authentication and authorization instead of relying on robots.txt to keep pages private.
Where do I put the robots.txt file?
Place robots.txt in the root directory of your website so it's accessible at https://yoursite.com/robots.txt. Crawlers look for it at this exact URL — subdirectory locations (like /blog/robots.txt) are not recognized. The file must also use the exact filename robots.txt (all lowercase, singular "robot").
Can I use robots.txt to hide pages from Google?
Yes — Disallow: /private/ prevents Google from crawling and indexing those pages. However, if the page is linked from other sites, Google may still show it in search results (without a description). To fully prevent indexing, use a noindex meta tag or X-Robots-Tag HTTP header in addition to robots.txt. Do not rely on robots.txt alone for privacy or security.

Related Tools

Sitemap Generator

Generate XML sitemaps to help search engines discover and index your website pages.
Meta Tag Generator

Generate SEO meta tags including title, description, Open Graph, and Twitter Cards.