Robots.txt

robots.txt in the root directory of a website uses user agent, Allow and Disallow to specify which areas crawlers may access. It controls crawling, not indexing, and doesn't protect against access; noindex is responsible for exclusion from the index.

Reviewed by Zoran Artmagic · November 8, 2026

The robots.txt is a simple text file with which a website operator tells search engines and other automated programs which areas of the website they may access and which they may not. It is the first thing a reputable crawler such as Googlebot requests on a domain before visiting pages. For online stores, it is one of the most important tools for directing crawling to the pages that should be found and keeping it away from filter combinations, the cart or customer accounts.

Where the robots.txt is located

The file is always called exactly robots.txt, in lower case, and sits in the top-level directory of a host:

https://www.your-store.com/robots.txt

A robots.txt only applies to exactly this host, this protocol and this port. A subdomain such as shop.your-store.com or a country domain needs its own file. If the robots.txt is placed in a subdirectory, it is ignored.

How the robots.txt is structured

The file consists of groups. Each group begins with one or more User-agent lines that specify which crawler the group applies to. The rules follow below:

User-agent: the name of the crawler, such as Googlebot, or an asterisk for all crawlers.
Disallow: a path that should not be accessed.
Allow: an exception within a blocked area.
Sitemap: the full address of the XML sitemap. This line doesn’t belong to any group and may appear anywhere in the file.

A typical example for an online store:

User-agent: *
Disallow: /checkout/
Disallow: /account/
Disallow: /*?order=
Disallow: /search

Sitemap: https://www.your-store.com/sitemap.xml

Since 2022, the format has been laid down as an internet standard (Robots Exclusion Protocol, RFC 9309). The basic idea dates back to 1994.

The most important rules

Wildcards: the asterisk stands for any characters, the dollar sign for the end of a URL. Disallow: /*.pdf$ blocks all URLs ending in .pdf.
Upper and lower case: paths are compared exactly as written. /Checkout/ and /checkout/ are two different paths.
The most specific rule wins: if Allow and Disallow match the same URL, the rule with the longer path applies. If both are equally long, Allow wins at Google.
One group per crawler: a crawler only follows the group that matches its name most precisely. If there is a dedicated group for Googlebot, Googlebot ignores the rules under User-agent: * completely. This is one of the most common mistakes of all.
Empty Disallow: Disallow: without a path allows everything.
File size: Google only reads the first 500 KiB of a robots.txt.

What the robots.txt can’t do

The most important distinction: the robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other pages link to it. Google then shows it without a description, and Search Console reports “Indexed, though blocked by robots.txt”. If a page is to disappear from the index, it needs a noindex in its meta tags, and Google must be allowed to crawl it to see this noindex.

In addition:

No protection: the robots.txt is publicly readable and a request, not a lock. Reputable crawlers respect it, others don’t. Listing paths such as /internal/ reveals them to anyone who opens the file. Confidential areas belong behind a password.
No noindex: Google has not supported the noindex directive in robots.txt since 2019.
No crawl-delay at Google: Google ignores this line. Bing respects it.

Tool	Affects	Typical use in a store
robots.txt	crawling	filters, sorting, cart, customer account
noindex	indexing	thank-you pages, internal search results
canonical tag	choice of the main version	variants, parameters, duplicate paths
password protection	access	test systems, internal areas

More on choosing the main version is in the entry on the canonical tag.

What happens if the robots.txt is missing or unreachable

File missing (404): Google assumes everything may be crawled. That isn’t an error, but it is a missed opportunity.
Server error (5xx): Google temporarily stops crawling the website because it doesn’t know what is allowed. If the error persists, Google falls back on the last known version.
Caching: Google usually caches the robots.txt for up to 24 hours. Changes therefore don’t take effect immediately.

Typical mistakes in online stores

Disallow: / carried over from the test system: at launch, the staging environment’s robots.txt is copied along, and the whole store is blocked for search engines.
CSS and JavaScript blocked: Google can then no longer render the pages properly and rates them lower.
Blocking pages that should leave the index: the block prevents Google from seeing the noindex. The pages stay in the index.
Forgetting the dedicated group: anyone who creates a group for a single crawler must repeat there all the rules that should also apply to it.
Important filter pages blocked: not every filter page is superfluous. Combinations with real demand can be valuable landing pages.

robots.txt in Shopware and Shopify

In Shopware 6, depending on the version and hosting, the robots.txt is placed as a file in the public directory or maintained via the configuration. Typical candidates for Disallow are checkout, customer account, sorting and filter parameters and the internal search. With several sales channels on their own domains, each domain needs the right file.

Shopify generates the robots.txt automatically. It already blocks cart, checkout, customer account, admin area and many filter and sorting URLs and contains the reference to the sitemap. It can be customised via the robots.txt.liquid template in the theme.

Ready-made templates for both systems and how to control AI crawlers such as GPTBot, ClaudeBot or Google-Extended are shown in the article robots.txt for Shopware and Shopify in the AI era. More on the peculiarities of the systems under Shopware SEO and Shopify SEO.

How to check your robots.txt

In the browser: open your-store.com/robots.txt and read it. If it says Disallow: / under User-agent: *, the store is blocked.
In Google Search Console: the robots.txt report shows which version Google last fetched and whether there were errors.
With the [robots.txt Checker](rc): it checks your store’s file in seconds, including typical Shopware and Shopify mistakes and the rules for AI crawlers.

Whether blocked pages are still in the index is shown in the article on Google indexing.

Frequently asked questions

Do I need a robots.txt at all?

Not necessarily. Without a file, everything may be crawled. For online stores it is still useful, because otherwise filters, sorting and customer areas tie up an unnecessary amount of crawling.

Does a Disallow rule remove a page from Google?

No. Disallow only prevents crawling. Removing a page from the index requires noindex, and the page must remain crawlable for that.

How quickly does a change take effect?

Google re-fetches the robots.txt regularly and usually caches it for up to 24 hours. A faster fetch can be triggered in Search Console.

Can I block AI crawlers with the robots.txt?

Yes, the major providers respect the robots.txt. What matters is the distinction between crawlers for training AI models and crawlers for real-time search. Blocking the latter makes you disappear from these assistants’ answers. More on this in the entry on Generative Engine Optimization.

Is a robots.txt a security risk?

Only if it reveals confidential paths. Anything that needs protection belongs behind a password, not in the robots.txt.