X-Robots-Tag

Definition

The X-Robots-Tag is a header a web server includes in its HTTP response for a URL. Google's specification states that any rule usable in a robots meta tag can also be given as an X-Robots-Tag. The header and its values are case-insensitive.

HTTP/1.1 200 OK
X-Robots-Tag: noindex

Several headers can be sent in one response, or several rules can be listed with commas. A user-agent token can precede the rules, so X-Robots-Tag: googlebot: nofollow applies only to Google while X-Robots-Tag: otherbot: noindex, nofollow targets another crawler. Rules without a user agent apply to all crawlers.

In practice

The header exists mainly for files that cannot hold a <meta> tag. Google recommends it for keeping PDFs, video files and images out of results. Because it is set in server configuration, one rule can cover a whole file type: an Apache or NGINX directive can add noindex, nofollow to every .pdf on a site.

Commonly used values:

As with the meta tag, the header is only seen if the URL is crawlable. A URL disallowed in robots.txt never returns its headers to Googlebot.

Relation to backlink indexing

The header is invisible in a browser's view-source, which is why it causes confusing indexing failures. A guest post can look clean in its HTML yet be served with X-Robots-Tag: noindex by a CDN rule, a security plugin or a staging configuration that leaked into production. Every crawl request a link indexer triggers will then reinforce the exclusion.

Checking response headers (for example with curl -I) belongs in any backlink audit before resubmission. The IndexChex backlink monitor reads both meta robots and X-Robots-Tag on each source page, so a header-level noindex is reported rather than mistaken for a slow index.

Related terms

• noindex, the rule most often carried by the header
• robots.txt, a crawl control at the site level
• Googlebot, the crawler that reads the header
• Canonical URL, which also has an HTTP header form

Browse the complete term list.