SEO & Web

Robots.txt Generator

Create a robots.txt that tells search engines and AI crawlers what they may crawl — with per-bot allow and disallow rules, crawl-delay and your sitemap.

Crawler rules

Pick a crawler from the list (including AI crawlers like GPTBot and ClaudeBot) or type its name. A crawler follows only its own group if it has one — rules under * don't carry over.

Groups (2)

  • User-agent: *

    • Allow: /
    • Disallow: /admin
    • Disallow: /private

    Bing and Yandex respect this; Google ignores it.

  • User-agent: Bingbot

    • Allow: /
    • Disallow: /admin
    • Disallow: /private

    Bing and Yandex respect this; Google ignores it.

💡 robots.txt best practices

  • The file must live at the root: https://yourdomain.com/robots.txt.
  • robots.txt is public and only a request — never use it to hide private pages. Use logins or noindex.
  • Don't block CSS or JavaScript that your pages need to render.
  • Always list your sitemap, and test changes in Google Search Console's robots.txt report.

robots.txt

User agents
2
Rules
6
Sitemap
Yes
User-agent: *
Allow: /
Disallow: /admin
Disallow: /private

User-agent: Bingbot
Allow: /
Disallow: /admin
Disallow: /private
Crawl-delay: 1

Sitemap: https://example.com/sitemap.xml

Checks

  • Pass: 2 user-agent groups, each with at least one rule
  • Pass: Crawlers can reach everything you didn't block
  • Pass: Sitemap directive uses a full URL
  • Pass: File size 182 B (Google reads the first 500 KB)

  • Per-crawler rules

    Separate groups for Google, Bing, social preview bots and AI crawlers like GPTBot, ClaudeBot and Google-Extended.

  • Common paths built in

    Pick typical paths such as /admin, /wp-admin/, /cart or /*? from suggestions, or type your own.

  • Mistake checks

    Warns when you're about to block your whole site, a sitemap URL isn't absolute or a crawl-delay is invalid.

Free robots.txt generator

robots.txt is a small text file at the root of your website that tells crawlers which parts of the site they may visit. Search engines check it before crawling, and it's also where you point them to your XML sitemap. More and more site owners also use it to opt out of AI training crawlers.

How to use it

  1. Keep the User-agent: * group for all crawlers and add paths you don't want crawled, such as /admin.
  2. Add a group for any crawler that needs different rules — for example GPTBot with Disallow: / to opt out of OpenAI training.
  3. Add your sitemap URL on the “Sitemap & host” tab.
  4. Download robots.txt, upload it to your site root so it opens at https://yourdomain.com/robots.txt, and check it in Google Search Console.

Rules worth knowing

  • A crawler obeys only the most specific group that names it. If you add a Googlebot group, Google ignores everything under User-agent: * — repeat any rules it still needs.
  • When Allow and Disallow both match, Google follows the longer (more specific) path. * matches any characters and $ marks the end of a URL.
  • Blocking a page in robots.txt doesn't remove it from Google if other sites link to it — use a noindex tag for that, and never rely on robots.txt to hide private content.

Private by design. Everything runs in your browser — nothing you enter is uploaded or stored by AISeekho.

More free tools