Tripline

Blocked by robots.txt

Google says a page is blocked by robots.txt.

A rule in your robots.txt tells Googlebot not to read the page. Sometimes that's on purpose; often it's a staging rule or a pattern that matches more than it should. See which rule is doing it.

Free, with no sign-up. Takes a few seconds.

What it means

robots.txt controls crawling, not indexing. A blocked page can't be read, so Google can't see its content, its title or any noindex on it. It may still list the bare address if other sites link to it, which is what Indexed, though blocked by robots.txt means.

Google picks one group of rules: the one naming Googlebot most specifically, or User-agent: * if none does. Rules in other groups are ignored. Inside the group, the longest matching rule wins, and Allow wins a tie.

Why it happens

robots.txtUser-agent: *
Disallow: /

The staging rule that went live

Staging sites block everything, and the file gets copied to production with the rest of the site.

PatternDisallow: /p

A rule wider than it looks

Rules match from the start of the path, so /p also blocks /pricing and /products.

Two groupsUser-agent: Googlebot
Disallow: /search

A group just for Googlebot

Once a group names Googlebot, Google ignores everything under User-agent: *, including the Allow rules you rely on.

WildcardsDisallow: /*?

A wildcard catching real pages

* matches anything and $ anchors the end. A rule meant for tracking parameters can block every page with a query string.

How to fix it

  1. Find the rule.

    Run the check above: it applies Google's own rule order to your home page and a sample from your sitemap and says which pages are blocked.

  2. Check which group applies.

    If any group names Googlebot, that is the only group Google reads. Make sure the rule you meant to change is in it.

  3. Remove or narrow the rule.

    Delete the Disallow, or make it specific, for example Disallow: /search/ instead of Disallow: /s. An empty Disallow: allows everything.

  4. Don't use robots.txt to hide pages.

    To keep a page out of search results, let Google crawl it and add noindex. Blocking it stops Google seeing the noindex at all.

  5. Let Google pick up the change.

    Google caches robots.txt for up to a day. Search Console's robots.txt report can request a fresh read, and URL Inspection can then request indexing.

Common questions

What's the difference between blocked and noindex?

Blocked means Google may not read the page. Noindex means it may read it but must not show it in results. To remove a page from search, use noindex and leave it crawlable.

Does Allow beat Disallow?

Only when it is at least as long. The longest matching rule wins, and Allow wins when two rules are the same length.

My robots.txt returns an error. Is that the same thing?

Worse. If robots.txt returns a server error, Google may stop crawling the whole site until it can read the file. A missing robots.txt (404) is fine: everything is allowed.

Can I see what Google sees?

Search Console's robots.txt report shows the version Google last fetched. This check fetches it fresh and applies the rules to real pages from your sitemap.

Other fixes

SPF PermError: too many DNS lookupsYour SPF record needs more than 10 DNS lookups. Excluded by ‘noindex’ tagGoogle says your page is excluded by a noindex tag. DMARC policy: p=noneYour DMARC policy is p=none, so fake mail still gets through.