Googlebot

Every page of your store has a first visitor, long before any customer arrives: Googlebot. It doesn’t decide rankings, but it decides whether a page gets a chance to rank at all. For online stores with thousands of products, variants and filters, this is no formality. This is where it becomes clear whether Google spends its time on your most important pages or on endless parameter URLs.

What exactly Googlebot is

“Googlebot” is the umbrella term for the crawlers of Google Search. Since the switch to mobile-first indexing, Google crawls mainly with the smartphone crawler. So the mobile version of your store is what counts for ranking, not the desktop version. There are also specialised crawlers:

• Googlebot Image and Googlebot Video: for images and videos in image and video search.

• Storebot-Google: fetches product pages for Google Shopping and Merchant Center, for example to check prices and availability.

• Google-InspectionTool: becomes active when you inspect a URL in Search Console.

• Google-Extended: is not a crawler of its own but a control token in the robots.txt. It lets you decide whether your content may be used to train Gemini. It has no influence on Google Search or AI Overviews.

How Googlebot works

• Discovering: Googlebot finds new URLs via links, internal and external, and via the XML sitemap. A sitemap contains at most 50,000 URLs; large stores split it across several files.

• Fetching: Googlebot loads the page via HTTP/1.1 or HTTP/2. It processes the first 15 MB of each HTML file and fetches images, CSS and JavaScript separately. It adapts its speed to the server: if the store responds slowly or with errors, it comes less often.

• Rendering: Google executes JavaScript and sees the page much like a browser. But rendering can happen with a delay. Content that is created only by JavaScript is therefore captured later or incompletely.

• Indexing: Only then does Google decide whether the page is added to the index. Crawled does not mean indexed.

Crawl budget: why it matters for stores

Google devotes only limited resources to each website, the so-called crawl budget. For a small company website, this hardly matters; for a store, it very much does. An example: 5,000 products in 100 categories, each category with filters for colour, size, brand and price as well as several sort orders. This quickly creates hundreds of thousands of URLs such as /trail-shoes-women?colour=red&size=8&sort=price. If Googlebot crawls these variants, it has no time left for new products and changed prices.

The most important levers:

• Block unimportant areas: Cart, customer account, internal search results and filter combinations without their own demand should be blocked via robots.txt or not linked at all.

• A clean sitemap: It contains only pages that should be indexed, no redirects, no error pages, no variants.

• Clear signals for duplicates: The canonical tag shows Google which of several similar pages is the authoritative one. More on this under duplicate content.

• Clean status codes: Permanently removed products return 404 or 410 or redirect to a suitable successor. Endless redirect chains cost crawl time.

• A fast server: The faster the store responds, the more pages Googlebot gets through. Google PageSpeed Insights shows where to start.

robots.txt, noindex, canonical, nofollow: what each is for

These four tools are often confused, and that is exactly where the most expensive mistakes come from:

• robots.txt: prohibits crawling. It is not access protection and does not reliably prevent indexing. A blocked URL can end up in the index if other pages link to it.

• noindex: allows crawling but prohibits inclusion in the index. This is the right tool to keep a page out of the search results.

• canonical: tells Google which of several similar pages is meant. Google treats this as a strong hint, not as a command.

• nofollow, sponsored, ugc: mark links that are not an editorial recommendation. Since 2019, Google has treated them as hints. They are no use for keeping pages out of the index. More on link attributes under link building.

A classic mistake: a page is set to noindex and at the same time blocked via robots.txt. Then Googlebot is not allowed to fetch the page and never sees the noindex. The page can still stay in the index.

What Googlebot often doesn’t see

• Content that only loads after a click, such as product descriptions in tabs that JavaScript creates only on demand.

• Infinite scrolling without real page links: Without links to page 2, 3 and beyond, Googlebot doesn’t find the further products.

• Content behind a login or country selector: Whatever requires input stays closed to the bot.

• Blocked resources: If CSS or JavaScript is blocked via robots.txt, Google sees a broken page.

• Overzealous bot protection: Firewalls and bot defences at hosting providers or CDNs sometimes block the real Googlebot too. This is often only noticed when pages drop out of the index.

Observing Googlebot

• Crawl stats in Search Console: how often Googlebot comes, how fast the server responds and which errors occur.

• Page indexing report: Two messages are particularly revealing. “Discovered – currently not indexed” means Google knows the URL but hasn’t fetched it yet, often a sign of tight crawl budget. “Crawled – currently not indexed” means Google has seen the page and found it not important enough, usually a quality or duplication issue.

• URL inspection: shows what Google sees for a single page and lets you request indexing. How this works is described in the article requesting Google indexing.

• Log file analysis: The server logs show, unfiltered, which URLs Googlebot really fetches. For large stores, this is the most honest view of crawl budget.

• Verifying authenticity: Many bots pretend to be Googlebot. A reverse DNS lookup shows whether a request really comes from Google.

Googlebot and AI crawlers

Besides Googlebot, other crawlers visit your store today, such as GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic or PerplexityBot. They follow their own rules and can be allowed or blocked individually in the robots.txt. If you want to be named in AI answers, you should allow the crawlers used for search. More on this under Generative Engine Optimization. Which of these crawlers your robots.txt allows today is shown by the free robots.txt Checker.

How Googlebot is controlled in Shopware and Shopify is shown on Shopware SEO and Shopify SEO. The bigger picture is described on SEO for online stores.

Googlebot is the first visitor to every page and the only one that sees all of them. Clear the way to the important pages and keep it away from the unimportant ones, and you create the foundation on which every other SEO measure builds.