Search engine index

The term index has many meanings. Usually it refers to a directory that makes information findable in an ordered way – like the index of a book. On the internet, it mainly refers to the data store of search engines. Without it, web search would not be possible.

The best-known index is Google's. To build it, crawlers such as Googlebot search the web: they follow links from page to page and from domain to domain, call up URLs and read their content. This process is called crawling. Google then analyses the content, assigns it to topics, recognises duplicates and decides whether and in what form a page is added to the index – indexing. Important: not every crawled page is indexed. Google only includes pages that are accessible, not excluded and useful enough from Google's point of view. Only indexed pages can appear in the search results.

How do search engines work with the index?

Without an index, a search engine would have no data from which to deliver results. When a user enters a search query, the search engine doesn't search the web but its index, and sorts the matching pages according to numerous ranking factors. Which factors these are is partly known; their exact weighting in the algorithm remains secret. A major challenge is the meaning of search queries: many terms are ambiguous. Google therefore works with entities and context to understand what is meant. For website owners, this means: content should cover its topic clearly and comprehensively so that Google can classify it unambiguously.

To keep the index up to date, crawlers revisit pages again and again and update the stored version. Important or frequently changed pages are crawled more often than rarely updated ones. And because Google regularly adjusts its evaluation criteria with core updates, one-off optimisation isn't enough: pages that no longer meet new standards can lose visibility or even drop out of the index.

Which pages are indexed – and which aren't?

Whether a page is added to the index depends on technical and content factors. Technically, it must be accessible (status code 200), must not be blocked via robots.txt or marked “noindex” and should be the canonical version. In terms of content, Google checks whether the page offers value of its own or is just a duplicate or a very thin page. Good internal linking, an XML sitemap and backlinks that draw Google's attention to important pages all help. The “Pages” report in Google Search Console shows which pages are indexed and why others aren't – with reasons such as “Crawled – currently not indexed”, “Duplicate” or “Excluded by noindex tag”. Whether a robots.txt blocks important areas can be checked with the robots.txt Checker.

Google also removes content from the index for legal reasons, for example following copyright complaints or removal requests. Such pages remain accessible but can no longer be found via search.

In addition, a website can be removed from Google's index for serious violations of the spam policies – such as cloaking, massive link spam or other black hat methods. Website owners who want to deliberately remove a single page from the index use the “noindex” meta tag or, temporarily, the removals tool in Search Console.