Indexing Glossary: crawling, indexing and backlink indexing terms A-Z

About this glossary

Search engine optimization has its own vocabulary, and the part that deals with getting pages into Google's index is especially easy to misread. Phrases such as "crawled" and "indexed" are often used as if they meant the same thing, and Search Console status labels like Crawled - currently not indexed are frequently quoted without their official meaning. This dictionary collects those terms in one place and defines each one in neutral language.

Google describes Search as working in three stages: crawling, indexing and serving results, and notes that not every page makes it through each stage. Most entries below sit somewhere along that path. A second group covers the tools and tactics built around it, in particular the backlink indexer, a category of service that tries to get links on third-party sites crawled so they can count.

Definitions of Google behavior are checked against Google Search Central and Search Console Help pages, which are cited on each entry. The glossary is published by IndexChex, a backlink indexing service, and says so openly; see about this site for the editorial policy.

How to use an entry

Each term page opens with a one-paragraph definition designed to be quoted on its own. Below it, "In practice" explains where you meet the term, and "Relation to backlink indexing" explains why it matters when links sit on pages you do not control. Related terms are cross-linked so you can follow a concept from discovery to verification.

Terms A-Z

B

• Backlink indexer: a service that submits URLs, usually pages carrying backlinks, so that search engine crawlers fetch them.
• Backlink indexing: the process of getting the page that hosts a backlink crawled and included in a search index.

C

• Canonical URL: the URL Google selects as the representative version among a group of duplicate pages.
• Crawl budget: the set of URLs on a host that Google can and wants to crawl.
• Crawled - currently not indexed: a Search Console reason meaning Google fetched the page but did not add it to the index.
• Crawling: the stage where automated programs download pages they have discovered.

D

• Discovered - currently not indexed: a Search Console reason meaning Google knows the URL but has not crawled it yet.
• Drip feed: spreading a batch of URL submissions over days instead of sending them at once.

G

• Googlebot: the generic name for Google Search's web crawlers, in smartphone and desktop variants.

I

• Index checker: a tool that reports whether given URLs are present in a search engine's index.
• IndexChex: a backlink indexer and bulk Google index checker sharing one credit balance.
• Indexing: the stage where Google analyzes a crawled page and may store it in its index.
• Indexing API: a Google API for notifying crawls of job-posting and livestream pages.
• IndexNow: an open protocol for telling participating search engines that a URL has changed.
• inurl: operator: a search operator that restricts results to URLs containing a given string.

L

• Link rot: the gradual loss of links as pages are removed, moved or edited.

N

• noindex: a rule, set in a meta tag or HTTP header, asking search engines not to index a page.

R

• robots.txt: a file at the root of a host that tells crawlers which paths they may fetch.
• rel=nofollow: a link attribute hinting that the linking site does not endorse the target.
• rel=sponsored and rel=ugc: link attributes that label paid links and user-generated links.

S

• site: operator: a search operator that limits results to a domain, URL or URL prefix.

T

• Tiered links: a structure where secondary links point at pages that hold primary backlinks.

U

• URL indexer: a general term for any tool that pushes URLs toward crawling, regardless of link context.

X

• X-Robots-Tag: an HTTP response header that carries indexing rules such as noindex.
• XML sitemap: a file listing URLs a site owner wants search engines to discover.

Where to start

Readers new to the topic can follow three entries in order: crawling, indexing and Crawled - currently not indexed. Together they explain why a page can be visited by Googlebot and still not appear in search. Readers evaluating link services should start with backlink indexing and index checkers.