Crawling

Definition

In search, crawling means fetching a web page so its content can be processed. Google lists it as the first of three stages, followed by indexing and serving results, and notes that not every page makes it through each stage. The program that does the fetching for Google Search is Googlebot.

Crawling is preceded by URL discovery. Google has no central registry of web pages, so it learns about URLs from pages it already knows, from links on newly crawled pages, and from submitted sitemaps.

In practice

Google uses an algorithmic process to decide which sites to crawl, how often, and how many pages to fetch from each. Crawlers slow down if a site responds with errors such as HTTP 500, so a struggling server receives fewer visits. The amount a site can receive is summarized as its crawl budget.

During the crawl Google renders the page and executes JavaScript in a recent version of Chrome, so content that only appears after scripts run can still be seen.

Things that stop a crawl include:

• a robots.txt rule disallowing the path;
• a login requirement;
• server and network problems.

Crawling is not indexing

The most common misunderstanding about the term is treating a crawl as proof of indexing. They are different events. A page can be crawled and rejected, which Search Console reports as Crawled - currently not indexed. A page can also be known but not yet crawled, reported as Discovered - currently not indexed. And because robots.txt blocks crawling rather than indexing, a blocked URL can occasionally appear in results without its content having been fetched.

Relation to backlink indexing

For a link on a third-party page, crawling is the step the link builder can actually influence. A backlink indexer creates discovery signals so Googlebot fetches the host page sooner than it would otherwise. Providers that are precise about this, including IndexChex, guarantee the crawl rather than the index entry, because what follows the crawl is Google's decision.

Related terms

• Indexing
• Googlebot
• Crawl budget

Browse the A-Z index of terms.