Skip to content

Free tools for SEO, AI search, Google Ads and local businesses. No sign-up. See all tools

DigitalSnot

Robots.txt Generator for WordPress, Shopify and other sites

Build a robots.txt for your site type, with AI crawler choices and a line-by-line explanation.

  • Free to use
  • Runs in your browser
  • No sign-up

How to use the Robots.txt Generator

Read what the guide says

This is the Robots.txt Generator.

Build a robots.txt for your site type, with AI crawler choices and a line-by-line explanation.

Pick your site type and enter your website's address. The usual paths for that type are filled in for you to check.

Or press Try an example to fill in sample details.

Tick the areas to keep crawlers out of, choose Allow or Block for each AI crawler,

and add your sitemap address and the pages that must stay open.

Download robots.txt,

upload it to the root of your site so it opens at yoursite.com/robots.txt,

then test a few addresses with the Robots.txt Tester.

It runs in your browser, so nothing you type is uploaded.

Try it free on this page.

Runs in your browser. Nothing you type is uploaded.

Your site

The file only works at the root of this host, so a subdomain needs its own.

Keep crawlers out of

The paths are our suggestions for the site type you picked. Check them against your own addresses.

Each name becomes a rule such as /*?*sort=, the pattern Google's own example uses for filter pages. Keep the plain category pages open.

For every crawler. Start a line with Allow: to keep one path open inside a blocked folder. Paths are case-sensitive.

AI crawlers

Googlebot and Bingbot are not in this list: they follow the rules for every crawler, so your site stays in Google and Bing. Each company documents its own crawlers.

AI training

Collect pages to train AI models. Blocking them does not block the AI search crawlers below.

  • Google-Extended

    Google-Extended

    Google: Use of your content to train and ground Gemini models (not a crawler; does not affect Search). Docs

  • GPTBot

    GPTBot

    OpenAI: Training OpenAI models. Docs

  • ClaudeBot

    ClaudeBot

    Anthropic: Training Anthropic models. Docs

  • Applebot-Extended

    Applebot-Extended

    Apple: Use of content for training Apple models (not a crawler). Docs

  • CCBot

    CCBot

    Common Crawl: Open web archive widely used to train AI models. Docs

  • meta-externalagent

    meta-externalagent

    Meta: Training Meta AI models. Docs

AI search and answers

Find pages to show and cite in AI answers. Blocking them can keep your pages out of those answers.

  • OAI-SearchBot

    OAI-SearchBot

    OpenAI: Showing your site in ChatGPT search answers. Docs

  • PerplexityBot

    PerplexityBot

    Perplexity: Showing your site in Perplexity answers. Docs

  • Claude-SearchBot

    Claude-SearchBot

    Anthropic: Claude web search results. Docs

Opened when a user asks

Fetch a page when a person asks the assistant to open it. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it.

  • ChatGPT-User

    ChatGPT-User

    OpenAI: Pages a ChatGPT user asks it to open (robots.txt may not apply). Docs

  • Perplexity-User

    Perplexity-User

    Perplexity: Pages a Perplexity user asks it to open. Docs

  • Claude-User

    Claude-User

    Anthropic: Pages a Claude user asks it to open. Docs

Sitemaps and pages to test

Full addresses, as your site or SEO plugin shows them.

Your main pages, or addresses from your sitemap. Each one is tested against the file for Googlebot.

Advanced (optional)

Your own groups, each starting with a User-agent line. They are explained and checked like the rest.

Share this free tool

Embed this tool on your websiteFree. Paste one code; it runs in your visitors' browsers.

Copy this code into a page of your site. The tool sets no cookies and sends nothing your visitors type to us. Change height="760" if it needs more room.

Preview the embed

How to embed a free tool on your website

Read what the video says

Want one of our free tools on your own website?

Here's how to embed it, in about a minute.

Open any of our browser tools, such as the GST Calculator.

Below the tool, find Embed this tool on your website, and open it.

Press Copy code.

The code is a frame with the tool in it, plus a short credit line that links to the tool's page.

In your website's editor, add an HTML block where you want the tool, and paste the code.

In WordPress, that's the Custom HTML block.

If the tool needs more room, change the height in the code.

Publish the page, and the tool works on your site.

Here, it's on a sample page for a made-up sweet shop.

Visitors use it right there. It sets no cookies, and sends nothing they type to us.

The credit line under it links to the tool on DigitalSnot.

You can embed as many of our free browser tools as you like.

About the Robots.txt Generator

Choose your site type (a small business site, WordPress, Shopify, an online shop or a blog) and tick what to keep crawlers out of: the admin area, cart and checkout, site search and filter pages. Decide crawler by crawler whether AI search, AI training and user-request bots may read the site, then add your sitemap and any rules of your own. You get the robots.txt to copy or download, an explanation of every line, and checks against Google's robots.txt rules and RFC 9309. It runs in your browser: nothing you type is uploaded.

How it works

  1. Step 1

    Pick your site type and enter your website's address. The usual paths for that type are filled in for you to check.

  2. Step 2

    Tick the areas to keep crawlers out of, choose Allow or Block for each AI crawler, and add your sitemap address and the pages that must stay open.

  3. Step 3

    Download robots.txt, upload it to the root of your site so it opens at yoursite.com/robots.txt, then test a few addresses with the Robots.txt Tester.

How robots.txt rules work

A crawler follows the one group that names it most specifically, or the * group if none does. Inside that group the rule with the longest path wins, and when rules conflict Google uses the less restrictive one. * stands for any characters, $ marks the end of the address, and paths are case-sensitive. Google reads the first 500 KiB of the file and supports only user-agent, allow, disallow and sitemap lines.

Blocking crawling is not hiding a page

robots.txt asks crawlers not to fetch pages; it does not keep them out of Google. A blocked address can still appear in results, without a description, if other pages link to it, and Google never sees a noindex tag on a page it may not crawl. To keep a page out of results, let it be crawled and give it noindex. And do not block the CSS and JavaScript files Google needs to understand your pages.

What this tool cannot do

It writes the file from your choices and does not read your live robots.txt or sitemap, so paste the pages that must stay open and it tests them for you. It cannot stop crawlers that ignore robots.txt: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. On Shopify you cannot upload a file; you add lines to the robots.txt.liquid template, which Shopify calls an unsupported customisation.

Frequently asked questions

Where do I upload robots.txt?

At the root of your site, so it opens at https://yoursite.com/robots.txt; a file in a folder is not read. It covers only that protocol, host and port, so a subdomain such as shop.yoursite.com needs its own file. On most hosting you can upload it with cPanel's File Manager or FTP, into the folder that holds your home page.

Should I block AI crawlers?

It is your choice, and the companies keep search and training apart. OpenAI says GPTBot (training) and OAI-SearchBot (ChatGPT search) are independent, and that sites which block OAI-SearchBot are not shown in ChatGPT search answers. Google-Extended covers Gemini training and grounding and does not affect Google Search. Blocking only the training crawlers lets AI search crawlers keep reading your pages.

Does Google follow Crawl-delay?

No. Google supports only user-agent, allow, disallow and sitemap lines, and Crawl-delay is not part of RFC 9309, though some other search engines use it. This generator does not add it; if you need it for another search engine, add it in the extra lines under that crawler's own group.

Should I block wp-content or wp-includes on WordPress?

No. Those folders hold your theme's CSS and JavaScript and your images. Google says not to block files it needs to understand a page, and it does not render JavaScript from blocked files. For WordPress this generator blocks /wp-admin/ and keeps admin-ajax.php open, which is our suggestion.

How long before crawlers see my new robots.txt?

RFC 9309 says crawlers should not use a cached copy for more than 24 hours, and OpenAI says changes take about 24 hours to reach ChatGPT search. Make sure the file opens at /robots.txt with a normal page status: Google treats a server error on robots.txt as a reason to stop crawling the site for a while.

Sources and checks

Rules last checked

The Robots.txt Generator follows these official pages:

Each AI crawler in the tool links to its own company's page; the paths suggested for each site type are our suggestions.

Scores, limits and suggestions marked "our checklist", "our measure" or "our suggestion" are ours, not Google's or the platform's. How our tools work

Related reading

Rather have us do it for you?

Tell us about your business and what you want to fix. We reply within one business day.

Talk to us Our service: Technical SEO services

More free tools

All tools

Want us to fix what the tool found?

Tell us about your business and what you want to grow. We reply within one business day with next steps, not a sales script.

  • A reply within one business day
  • A quick look at your site and search visibility before we talk
  • Your details are only used to answer your enquiry

Prefer email or a call?

[email protected]

+91 70079 75411

Popular