Robots.txt for SEO: what to block, what to allow and how to test it

Robots.txt for SEO is mostly about restraint. The file tells search engine crawlers which URLs on your site they may fetch, and a good one is short: it keeps crawlers out of admin screens, carts and endless filter pages, and leaves everything else open. A bad one can hide your whole site from Google, or leave pages you wanted gone sitting in results with no description.

This guide is for small-business owners, marketers and developers who look after a site's robots.txt. You will learn what the file can and cannot do, the syntax Google follows, what to block and never block, how Google reacts when the file fails, and how to test a change. Ready-to-copy files for a clinic and a small online shop are near the end. Every rule was checked against Google's documentation and RFC 9309 in October 2026.

What robots.txt does, and what it does not do

Robots.txt is a plain text file at the root of a host that asks crawlers not to fetch certain URLs. It controls crawling, not indexing. Google says the file is used mainly to avoid overloading your site with requests, and that it is not a way to keep a web page out of Google.

Google's introduction to robots.txt is direct about the limits. If a web page is blocked with robots.txt, its URL can still appear in search results, but the result will not have a description. Google can find the URL through links from other pages and index it without ever visiting it. To keep a page out of Google, the same page says to use noindex or to password-protect the page.

Robots.txt is not security either. The standard itself, RFC 9309, says the protocol is not a substitute for proper content security, and that listing paths in robots.txt makes them public and discoverable. Anyone can open yoursite.com/robots.txt. It is also a request, not a lock, and Google's robots.txt specification notes that Google does not apply it to fetches that users control, such as feed subscriptions, or to crawlers used for user safety, such as malware analysis.

What the file can and cannot do

Keep that table in mind for the rest of this guide. Many robots.txt problems start with asking the file to do a job from one of the "No" rows.

Where robots.txt must live

Put one file named robots.txt, in lower case, at the top level of each host you want to control. Its rules apply only to the protocol, host and port the file is served from, so every subdomain, and every protocol or port that serves content, is treated separately.

Google's guide to writing and submitting a robots.txt file says a site can have only one robots.txt file, that it must sit at the root of the site host it applies to, and that crawlers do not look for it in subdirectories. The file must be UTF-8 encoded plain text; Google may ignore characters outside the UTF-8 range, which can make rules invalid.

Size has a hard edge: Google reads only the first 500 KiB of the file and ignores anything after that. A small site needs a tiny fraction of it; a file near the limit usually lists single URLs that should be one pattern.

On a hosted platform you may not be able to edit the file at all; Google's introduction notes that platforms such as Wix or Blogger may offer a search settings page instead.

Robots.txt syntax, line by line

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines naming crawlers, followed by Disallow and Allow rules for those crawlers. Google supports four fields, user-agent, allow, disallow and sitemap, and ignores the rest.

# Sample file for a made-up clinic site
User-agent: Googlebot
Disallow: /drafts/

User-agent: *
Disallow: /admin/
Allow: /admin/help/

Sitemap: https://www.smilestudio.example/sitemap.xml

User-agent: which crawler a group is for

The user-agent value is a crawler's token, such as Googlebot or Bingbot. RFC 9309 says crawlers match it without regard to case, so googlebot and Googlebot are the same. A crawler follows only the group that matches it, ignoring every other group, and falls back to the User-agent: * group only when no group names it. If several groups name the same crawler, RFC 9309 says their rules are combined into one.

This catches people out. In the sample above, Googlebot has its own group, so it ignores the * group and may crawl /admin/. Repeat any rule you want a named crawler to follow inside its own group.

One Google crawler works the other way round. Google's list of special-case crawlers says AdsBot, which checks pages used in Google Ads, ignores the global * group. Only a group that names AdsBot-Google restricts it, and advertisers will usually want to leave it alone.

Disallow and Allow

Disallow names a path crawlers should not fetch; Allow names a path they may fetch, usually inside a disallowed folder. Each path starts with a slash and matches from the start of the URL's path, so a rule is a prefix. Paths are case-sensitive, so Disallow: /Admin/ does not block /admin/. A Disallow line with no path blocks nothing.

The first row is the trap. A prefix rule meant for one folder also blocks every URL that starts with the same letters, so add the trailing slash when you mean a folder.

Wildcards: * and $

Google supports two special characters in paths, as RFC 9309 does. An asterisk (*) matches any run of characters, including none, and a dollar sign ($) marks the end of the URL. Rules already match as prefixes, so a * at the end of a path adds nothing: Google treats /blog* the same as /blog.

A parameter can come first (after ?) or later (after &), so blocking it everywhere needs both forms, as the shop example later shows.

Which rule wins when rules conflict

Google picks a rule by length, not by its place in the file. Within a group, it uses the most specific rule, meaning the one with the longest matching path. When an Allow and a Disallow match equally, Google uses the least restrictive rule, which is the Allow. RFC 9309 says the same: the match with the most characters wins, and an Allow beats an equivalent Disallow.

Comments, and lines Google ignores

Anything after a # is a comment and is ignored, so use comments to record who changed the file, when and why. Google also skips lines it cannot parse and uses the rest; Search Console's robots.txt report lists the skipped lines.

Several rules you will find in old files do nothing for Google. In July 2019 Google announced it would retire all code that handles unsupported and unpublished rules, such as noindex, nofollow and crawl-delay, on 1 September 2019. It said those rules had never been documented by Google, and that their use was contradicted by other rules in all but 0.001% of robots.txt files on the internet.

Crawl-delay is not part of RFC 9309, and Google ignores it. Bing said in 2012 that Bingbot honours Crawl-delay, so the line is not wasted in a Bingbot group, but it will never slow Googlebot. To slow Googlebot in an emergency, Google's guide to reducing the crawl rate suggests returning 500, 503 or 429 status codes to crawl requests for a short time, and warns against keeping that up for long.

The Sitemap line

A Sitemap line tells crawlers where your XML sitemap or sitemap index lives. Use the full URL, including https://. Google's guide to building and submitting a sitemap says the line can go anywhere in robots.txt. Submitting the sitemap in Search Console's Sitemaps report as well shows you when Google read it and any errors it found.

The free XML Sitemap Generator turns a list of URLs or a crawl export into a valid sitemap and gives you the Sitemap line to paste. To check an existing one, the Sitemap Validator finds your sitemap from robots.txt or the usual places and flags listed pages that redirect, fail, carry noindex or point their canonical elsewhere.

What to block on a small site

Block URLs that waste crawling and that nobody should land on from a search: admin and login screens, cart and checkout steps, internal search results, and sort or filter parameters. On a small site, that is usually all you need.

Google's guide to optimising your crawl budget is written for large sites, but its advice works in miniature: consolidate duplicate content first, and block unimportant pages that cannot be consolidated, such as differently sorted versions of the same page, with robots.txt. Google says this significantly decreases the chance that those URLs are processed by other Google systems.

Filters are the classic case for online shops. Google's guide to managing faceted navigation says filter URLs, often built with parameters, can lead to overcrawling and slower discovery of new content, and recommends robots.txt rules that disallow them when you do not need them in search. It also notes that filters held in # fragments have no effect on crawling at all.

Thank-you pages, thin tag pages and old offers are missing on purpose: you want them out of results, not out of crawling, so they need noindex.

What never to block

Never block the files Google needs to render your pages, pages you want removed from results, pages listed in your sitemap, or the whole live site.

CSS, JavaScript and image files your pages need

Google needs your CSS and JavaScript to see what visitors see. Google's JavaScript SEO basics say Google Search will not render JavaScript from blocked files or on blocked pages, and its robots.txt introduction says to block resource files only when pages load without them and are not significantly affected. If a blocked script builds your menu, prices or reviews, Google may never see them. On WordPress this usually means leaving /wp-content/ and /wp-includes/ alone, because they hold theme files, scripts and uploaded images.

Pages you want removed from Google

This mistake is easy to make and slow to undo. If you block a page that is already indexed, Google can no longer crawl it, so it never sees the noindex tag you added, and the URL can stay in results as "Indexed, though blocked by robots.txt". Google's noindex documentation says that for noindex to work, the page must not be blocked by robots.txt. Its robots meta tag and X-Robots-Tag specification says the same of both: the rules can only be read and followed if crawlers are allowed to access the page.

The fix is the order of steps. Remove the Disallow rule, add noindex, and leave the page crawlable. Blocking it again later can let the bare URL come back if other sites link to it.

Pages in your sitemap, and the whole live site

A sitemap lists pages you want crawled and indexed, so blocking them sends Google opposite signals. A lone Disallow: / under User-agent: * blocks every compliant crawler from the whole host; it belongs only on a staging copy, and even there a password is the better lock.

Images and videos you want found

Google's introduction says robots.txt can keep image, video and audio files out of Google's search results. That is useful when you mean it, and harmful when you do not: a broad rule such as Disallow: /uploads/ or Disallow: /*.jpg$ can quietly remove your product photos from Google Images.

Robots.txt, noindex, X-Robots-Tag or a password: which to use

Use robots.txt to control crawling, noindex to control indexing, the X-Robots-Tag header for noindex on files that are not HTML, and a password for anything private. Google's page on how to control what you share with Google draws the same lines.

Google's noindex documentation says the meta tag and the response header have the same effect. The 404 and 410 row comes from Google's 2019 note on unsupported rules, and the Removals tool help says a removal lasts about six months and does not stop crawling.

If you use robots.txt and noindex on the same URL, robots.txt wins in the worst way. Google never fetches the page, so it never reads the noindex.

Worked example: a thank-you page in the results

Example: Smile Studio Dental, a made-up clinic in Kothrud, Pune, finds its /thank-you/ page showing in Google. The developer adds Disallow: /thank-you/ to robots.txt, and weeks later the page is still listed, now with no description. That is Google's normal handling of a blocked URL: its help page on missing page information in search results says this happens when a rule, such as one in robots.txt, stops Googlebot reading the page.

The fix takes three steps. Remove the Disallow line, add a noindex robots meta tag to the thank-you page, then run a live test in URL Inspection and confirm that it reports indexing as not allowed because of the noindex. Google's noindex documentation says the page is dropped from results when Googlebot next crawls it and sees the tag.

How Google fetches, caches and handles robots.txt errors

Google generally caches your robots.txt for up to 24 hours, so a change can take a day to apply. The status code your server returns for /robots.txt matters as much as the rules: a 404 means Google may crawl everything, while a server error can stop it crawling the whole site.

Caching

Google's specification says it generally caches robots.txt for up to 24 hours, and may keep a copy longer when it cannot refresh it, for example because of timeouts or 5xx errors. Google's page on updating your robots.txt file says its crawlers pick up changes when they refresh the cached copy, and that you can ask for a faster refresh with the recrawl option in Search Console's robots.txt report. RFC 9309 asks all crawlers not to use a cached copy for more than 24 hours unless the file is unreachable.

What each status code does

So a missing robots.txt is harmless and a broken one is not. Search Console's robots.txt report says having no robots.txt file is fine and means Google can crawl all your URLs. A file that times out or returns 503 can pause crawling across the whole site.

Treat robots.txt like code: change it on purpose and test it before upload. Your server logs show when Googlebot fetched the file and what it crawled afterwards; our server log analysis guide explains how to read them.

Common robots.txt mistakes and how to fix them

Most robots.txt damage comes from a short list of repeated mistakes. Check your file against each one.

1. The staging file goes live. A test site carries Disallow: / to keep crawlers out, the file is copied with the code at launch, and the new site cannot be crawled. Protect staging with a password instead, and open /robots.txt on the live site after every release.
2. Blocking pages you want removed, or the files pages need. Google cannot see a noindex on a page it may not crawl, or render a page whose scripts are blocked.
3. A named group that drops your * rules. A Googlebot or Bingbot group means that crawler stops reading the * group, so repeat the shared rules.
4. Case and prefix surprises. Disallow: /Search does not block /search, and Disallow: /cart also blocks /cartoon-mugs.
5. Rules Google ignores. noindex, nofollow, crawl-delay and host lines do nothing for Google.
6. The wrong place or host. A file in a folder is never read, and a file on www does not cover the bare domain, a subdomain or http.
7. A robots.txt that errors. Make sure the file loads quickly with a 200 status, and that firewalls and CDN rules let crawlers fetch it.
8. Listing private paths. Disallow: /client-reports/ tells everyone where your client reports live. Put private areas behind a login instead.

If you are moving or relaunching a site, our website migration checklist covers the staging-to-live step, where a leftover Disallow: / can slip through with everything else.

WordPress and Shopify: what is different

WordPress

When there is no robots.txt file in the site's root folder, WordPress generates one. Its do_robots function outputs a User-agent: * group with Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php. Since WordPress 5.5 the generated file also points to the built-in sitemap index at /wp-sitemap.xml, as the WordPress 5.5 sitemaps announcement explains.

The "Discourage search engines from indexing this site" box under Settings, Reading no longer writes Disallow: / into robots.txt. The do_robots changelog says that since WordPress 5.3 it uses a robots meta tag instead. Either way, untick it at launch and check the live page source for a noindex tag.

Shopify

Shopify generates a default robots.txt that, in its own words, works for most stores. Rather than uploading a file, you change it by adding a robots.txt.liquid template to your theme. Shopify's robots.txt documentation strongly recommends using its Liquid objects to add or remove rules rather than replacing the file with plain text, because Shopify updates the default rules regularly. Save a copy of your live /robots.txt before you start, and test the result like any other change.

Ready-to-copy robots.txt examples

Each example is a complete starting file for a made-up business. Replace the paths and sitemap address with your own, delete any line you do not need, and test before you upload. Remember that each one covers only the host it is uploaded to.

A small business site: Smile Studio Dental, Kothrud, Pune

A clinic site with service pages, a blog, a booking form and an admin area. It needs very little.

# robots.txt for www.smilestudio.example (sample file)
# Changed 23 Oct 2026 by the web team: added /search

User-agent: *
# Admin and login screens
Disallow: /admin/
Disallow: /login
# Internal site search results
Disallow: /search

Sitemap: https://www.smilestudio.example/sitemap.xml

Notice what is not there. No rules for CSS, JavaScript or images, because Google needs them. No /thank-you/ line, because that page gets noindex instead. No Crawl-delay or noindex lines, because Google ignores them. If the same clinic ran WordPress, the file would look like this:

# robots.txt for the WordPress version of the same site (sample file)
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# WordPress search results
Disallow: /?s=
Disallow: /search/

Sitemap: https://www.smilestudio.example/wp-sitemap.xml

A small online shop: Kesar Crafts, Jaipur

A made-up shop selling blue pottery and block-printed textiles, with category pages that can be sorted and filtered by colour and price.

# robots.txt for www.kesarcrafts.example (sample file)

User-agent: *
# Cart, checkout and customer accounts
Disallow: /cart
Disallow: /checkout
Disallow: /account/
# Site search results
Disallow: /search
# Sort and filter versions of category pages
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?colour=
Disallow: /*&colour=
Disallow: /*?price=
Disallow: /*&price=

Sitemap: https://www.kesarcrafts.example/sitemap.xml

Check three things before you copy it. First, prefixes: Disallow: /cart also blocks a product at /cartoon-mugs, so if a URL starts with one of these words, use Disallow: /cart$ for the exact page. Second, filters people search for: if customers type "blue pottery mugs" into Google, give that filter a proper category page with a clean URL. Third, your platform: Shopify and WooCommerce use their own paths, so start from what yours already serves.

A staging or test site

# STAGING ONLY. Never copy this file to the live site.
User-agent: *
Disallow: /

Pair it with a password. Robots.txt only asks, and Google lists password protection as the way to keep private content out of Search.

To build a file like these from tick boxes, the free Robots.txt Generator starts from your site type (a small business site, WordPress, Shopify, an online shop or a blog), lets you choose what to keep crawlers out of, and explains every line it writes. It runs in your browser, and nothing you type is uploaded.

How to test robots.txt before and after you publish

Test the draft before you upload it, then check what Google fetched and inspect a few key URLs. In Search Console, the robots.txt report and URL Inspection cover Google's side.

Before you upload: test the draft

1. Paste the draft into the free Robots.txt Tester, choose a crawler such as Googlebot or Bingbot, and test real paths: your home page, a service or product page, a CSS file and a filtered URL. It shows whether each path is allowed and which line decides, and warns if the whole site, CSS or JavaScript is blocked.
2. If you built the file in the Robots.txt Generator, read its checks, which flag lines Google ignores and capital letters in paths.
3. Save a dated copy of the current live file, so you can roll back in minutes.

After you upload: check what Google sees

1. Open https://yoursite/robots.txt in a browser. It should load as plain text, not a login page, an error page or a challenge screen.
2. Open Search Console's robots.txt report under Settings. It shows the robots.txt files Google found for the top 20 hosts on your site, when each was last crawled, and any warnings or errors. For an urgent fix, such as removing an accidental Disallow: /, request a recrawl there. The report is available only for domain-level properties.
3. Inspect two or three key URLs with the URL Inspection tool and run a live test. Crawl allowed should say Yes; if it says No, a robots.txt rule is blocking Google.
4. For a quick view of one page, the free Indexability Checker checks the status code, redirects, robots.txt for Googlebot, noindex in the robots meta tag and X-Robots-Tag header, and the canonical. It cannot see Google's index; only URL Inspection shows that.
5. Watch the Page indexing report over the next few weeks for a rise in pages reported as blocked by robots.txt, or as "Indexed, though blocked by robots.txt". Our guide to page indexing issues explains both.

Other search engines and site-wide checks

Bing Webmaster Tools has its own robots.txt tester. Bing's announcement of the tool says it shows which statement blocks a URL and for which user agent, and lets you tell Bing the file has changed. For a site-wide view, the free Website SEO Audit crawls up to 50 pages the way a search engine does, reading your sitemap and robots.txt, and lists issues with the pages they affect. A section missing from the crawl is worth checking against your rules.

AI crawlers and robots.txt

AI crawlers read the same file, and the real decision is which kind of AI bot to block. OpenAI, for example, documents GPTBot and OAI-SearchBot as separate crawlers, one for training and one for ChatGPT search, with independent robots.txt settings, and says changes take about 24 hours to register. Google's common crawlers page describes Google-Extended as a robots.txt token for Gemini training and grounding, and says it does not affect inclusion in Google Search and is not a ranking signal.

Our guide on how to block AI crawlers without disappearing from AI search covers every major bot, three ready-made policies and the CDN settings that can block Googlebot by accident. To see what your current file allows, the free AI Crawler Checker reads your live robots.txt and shows, crawler by crawler, whether ChatGPT search, Perplexity, Claude, Google, Bing and training crawlers may fetch a page and which rule decides.

The group rules apply here too: a group that names an AI bot does not inherit your * rules, and does not change what Googlebot may crawl.

A robots.txt checklist

• One file per host, at /robots.txt, in UTF-8 plain text, loading with a 200 status.
• Only user-agent, allow, disallow and sitemap lines, plus dated comments.
• No Disallow: / under User-agent: * on the live site.
• CSS, JavaScript and images that pages load are left open.
• No page you want removed is blocked; it carries noindex instead.
• Named groups repeat any * rules they need.
• Paths checked for case and prefix surprises against real URLs.
• The full sitemap URL is listed.
• Tested in a tester, then in the robots.txt report and URL Inspection.

Robots.txt sits at the start of every crawl, so it belongs in any technical SEO audit. If you would like someone to review your robots.txt alongside crawling, indexing and site structure, that is part of our technical SEO service, and you can get in touch when you are ready.