Robots.txt Tester: Check Whether a URL Is Blocked

This is the Robots.txt Tester.

It shows whether a crawler may fetch a page on your site, and which line decides.

Use it when you edit your robots.txt, to test a path before the change goes live.

It also warns if the whole site, CSS or JavaScript is blocked.

Fetch a site's robots.txt, or paste the file you're editing.

Here, it's a sample site for a made-up dental clinic.

Choose a crawler, then type the path to test.

Then press Test.

In a few seconds, you see that Googlebot may not fetch this page, and the rule that decides.

Disallow stops crawling, not indexing. To keep a page out of search, use noindex.

This box shows what to fix first: here, a blocked folder of CSS and JavaScript.

The checks that passed: the whole site isn't blocked, and the sitemap loads.

Open How to fix to see why it matters, the steps and where to edit the file on your platform.

Here, remove the rule that blocks /wp-includes/, which holds CSS and JavaScript.

Below, the file shows the deciding line, and lists any lines Google ignores.

Below that is a corrected file. Review it, then replace yours.

Edit your file, then test the path again.

To start a new file, build one below from simple options.

You can also email yourself a PDF of the result.