Runs in your browser. Nothing you type is uploaded.

Your site

The file only works at the root of this host, so a subdomain needs its own.

Keep crawlers out of

The paths are our suggestions for the site type you picked. Check them against your own addresses.

Each name becomes a rule such as /*?*sort=, the pattern Google's own example uses for filter pages. Keep the plain category pages open.

For every crawler. Start a line with Allow: to keep one path open inside a blocked folder. Paths are case-sensitive.

AI crawlers

Googlebot and Bingbot are not in this list: they follow the rules for every crawler, so your site stays in Google and Bing. Each company documents its own crawlers.

AI training

Collect pages to train AI models. Blocking them does not block the AI search crawlers below.

  • Google-Extended

    Google-Extended

    Google: Use of your content to train and ground Gemini models (not a crawler; does not affect Search). Docs

  • GPTBot

    GPTBot

    OpenAI: Training OpenAI models. Docs

  • ClaudeBot

    ClaudeBot

    Anthropic: Training Anthropic models. Docs

  • Applebot-Extended

    Applebot-Extended

    Apple: Use of content for training Apple models (not a crawler). Docs

  • CCBot

    CCBot

    Common Crawl: Open web archive widely used to train AI models. Docs

  • meta-externalagent

    meta-externalagent

    Meta: Training Meta AI models. Docs

AI search and answers

Find pages to show and cite in AI answers. Blocking them can keep your pages out of those answers.

  • OAI-SearchBot

    OAI-SearchBot

    OpenAI: Showing your site in ChatGPT search answers. Docs

  • PerplexityBot

    PerplexityBot

    Perplexity: Showing your site in Perplexity answers. Docs

  • Claude-SearchBot

    Claude-SearchBot

    Anthropic: Claude web search results. Docs

Opened when a user asks

Fetch a page when a person asks the assistant to open it. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it.

  • ChatGPT-User

    ChatGPT-User

    OpenAI: Pages a ChatGPT user asks it to open (robots.txt may not apply). Docs

  • Perplexity-User

    Perplexity-User

    Perplexity: Pages a Perplexity user asks it to open. Docs

  • Claude-User

    Claude-User

    Anthropic: Pages a Claude user asks it to open. Docs

Sitemaps and pages to test

Full addresses, as your site or SEO plugin shows them.

Your main pages, or addresses from your sitemap. Each one is tested against the file for Googlebot.

Advanced (optional)

Your own groups, each starting with a User-agent line. They are explained and checked like the rest.

Free Robots.txt Generator. Runs in your browser, no sign-up. Open the full tool on DigitalSnot →