Every filter a visitor clicks on a Shopify collection page can create a new URL.
Google treats each one as a separate page.
How bad does it get? Google's own crawling documentation lists faceted navigation as one of the most common causes of overcrawling on the web. In one Botify crawl, a store with fewer than 200,000 products exposed more than 500 million crawlable URLs.
Shopify handles some of this out of the box. But not all of it.
In this guide, I'll show you exactly what the defaults cover, where they waste crawl budget, and how to decide which filter combinations deserve real collection pages.
Here's what you'll learn:
- How filtered collection URLs multiply (and what that costs in crawl budget)
- What Shopify's canonical and robots.txt defaults already cover, and the three gaps they leave
- How to find crawl waste in Search Console without log file access
- Which crawl control method fits each URL type, with the exact robots.txt.liquid edits to apply
- A four-check test for promoting a filter combination to a real collection page
Key takeaways:
- Five ordinary filter groups can turn one collection into 720 crawlable URL variants. 100 collections become 72,000.
- Shopify canonicalizes filtered URLs to the base collection by default. The bigger cost on Shopify is crawl waste.
- Google's updated guidance recommends robots.txt for facet URLs you never want indexed, and Shopify's robots.txt.liquid makes that a 20-minute edit
- Filter combinations with verified search demand should become collection pages. The rest should stay canonicalized filters that crawlers skip.
- AI crawlers fetch far fewer pages than Googlebot and fall into the same facet traps, so crawl waste now costs you AI visibility too
Let's dive in.
How One Collection Turns Into 720 URLs?
A collection with five filter groups can generate hundreds of crawlable variants of itself. Without anyone touching the theme.
Here's how it works:
Storefront filtering appends a query parameter for every active filter value. A shopper narrowing shirts to blue, large, and in stock ends up on /collections/shirts?filter.v.option.color=Blue&filter.v.option.size=L&filter.v.availability=1.
This is what that looks like on a real store from one of our client stores with three filters applied, every one of them sitting in the URL.

Three filters applied on a live Shopify collection page. The URL in the red box carries all three as parameters, and every different combination is a different crawlable URL.
Each added filter multiplies the combinations.
Twelve colors, six sizes, five price ranges, and two availability states turn one collection into 720 URL variants, before anyone sorts or paginates.

The multiplication behind facet bloat: five ordinary filter groups turn one collection URL into 720 crawlable variants. Across 100 collections, that is 72,000.
Why does this matter?
Crawlers cannot tell which of those URLs are useful without fetching them first. Google's documentation is blunt about the cost: crawling faceted URLs eats large amounts of computing resources, and the crawler spends less time on the pages you actually want discovered.
The practical damage shows up as slow indexing.
New products and fresh collection pages wait in the queue while Googlebot keeps fetching filter permutations nobody searches for.
What Shopify Handles For You (and the 3 Gaps It Leaves)
First, the good news:
Shopify's defaults cover the duplicate content half of the problem. Filtered URLs carry a canonical tag pointing at the clean collection URL. The default robots.txt already blocks sort parameters and tag combination URLs.
That's more protection than most platforms give you out of the box. It's also why generic faceted navigation advice written for Magento or custom builds mostly misses the point on Shopify.
But here's the thing....
The gaps are specific URL types. I keep finding the same three in every store audit: filter parameters, single tag URLs, and /collections/all.
| URL type | Example | Default handling | Risk |
|---|---|---|---|
| Filter parameters | ?filter.v.option.color=Blue | Canonical to base collection, still crawlable | Crawl waste |
| Sort parameters | ?sort_by=price-ascending | Blocked in default robots.txt | Low |
| Single tag URLs | /collections/shirts/blue | Crawlable, indexable in most themes | Thin duplicates plus waste |
| Tag combinations | /collections/shirts/blue+large | Blocked in default robots.txt | Low |
| /collections/all | Every product under a second path | Crawlable, indexable | Duplicate product paths |
| Vendor and type URLs | /collections/vendors?q=BrandName | Crawlable | Crawl waste |
| Pagination | ?page=7 | Crawlable, self canonical | Normal, leave it alone |
Shopify's default handling by URL type. The canonical tag stops duplicate indexing on filter parameters, but nothing in the default setup stops the crawling itself.
The canonical tag is the detail most store owners misread.
It consolidates indexing signals. But it does nothing to stop crawling, because Googlebot has to fetch the page before it can read the tag.
Don't take my word for it. You can check this on any store in ten seconds: view source on a filtered URL and look for the canonical line.

The filtered URL carries three parameters, and line 63 canonicalizes it back to the clean collection URL. This is Shopify's default and it is the reason duplicate content is rarely the real problem.
Gap #2: single tag URLs. These deserve special attention because they look legitimate.
A URL like /collections/shirts/blue renders a real page on a crawlable path. In most themes it's indexable. Stores end up with dozens of accidental near-duplicates of each collection that nobody wrote copy for.
Gap #3 is /collections/all.
Shopify generates it whether or not your navigation links to it. Every product becomes reachable under a second path, a full duplicate route through the catalog.

Filter parameters sit in the canonical circle but outside the blocking circle, so they are consolidated but still crawled. The three URL types in red have no default protection at all.
How to Find Crawl Waste in 10 Minutes (No Log Files Needed)
Search Console shows you facet bloat for free.
Many Shopify store operators never look.
The Page indexing report is the fastest read. The excluded URLs sitting in "Alternate page with proper canonical tag" and "Crawled, currently not indexed" are largely your filter permutations.
Here's the exact check you can run on any store, step by step:
- Open Page indexing in Search Console and compare the excluded count against the indexed count. In client audits, it is common to find the excluded bucket several times larger than the indexed one.
- Open the "Alternate page with proper canonical tag" bucket and scan the sample URLs. If they are full of filter parameters, that is your crawl budget being spent.
- Open Settings, then Crawl stats, and scan the sample URLs there for the same patterns. This report shows what Googlebot actually fetched in the last 90 days.
- Crawl the store with Screaming Frog or Ahrefs with query parameters enabled, and compare the URL count against your sitemap. The gap is your filter URL count.
Here's step one on a real store: a Shero client on Shopify with a large catalog.
70,700 URLs excluded against 14,300 indexed. Five excluded URLs for every one that can rank.

The Page indexing report on a client store, client name blurred. The gray card in the red box is the excluded count.
Scroll down in that same report, and GSC tells you why those pages are excluded.
On this store, the top reason says it all

The same store's top exclusion reasons. The alternate canonical bucket alone, 25,189 URLs and largely filter permutations, is bigger than the store's entire indexed page count.

What facet bloat looks like at scale in Botify's published crawl data: 19 of 20 million crawlable pages on one site were facet URLs.
Botify's example at the beginning of this post is a bigger site. But it's the same pattern as the client report above.
However, store size does not exempt you. The multiplication works the same way for 200 products as it does for 200,000, and smaller stores often have less crawl budget to waste.
Bottom line?
Are you wondering why new products take weeks to index? The answer sits in this report. The crawl budget those product pages needed was spent on filter permutations, and nobody connects the two.
Crawl Control on Shopify, Ranked by What It Actually Stops
For facet URLs you never want indexed, robots.txt disallow is the control Google's own documentation now recommends.
Why?
It's the only common method that stops the crawling itself. Everything else either consolidates signals after the crawl or hides links without removing the URLs.
| Method | Stops crawling | Stops indexing | On Shopify |
|---|---|---|---|
| Canonical tag | No | Consolidates, mostly | Already the default on filter parameters |
| robots.txt disallow | Yes | Not by itself | Edit robots.txt.liquid |
| Noindex meta tag | No | Yes | Conditional in theme.liquid, works for tag URLs |
| Link hygiene in the theme | Reduces discovery | No | Worth doing on facet-heavy sidebars |
The four controls compared. Only robots.txt prevents the fetch itself, which is why it is the first move on high-volume filter parameters.
One warning before you start:
There's an ordering mistake to avoid when facet URLs are already indexed.

The removal sequence. Blocking a URL in robots.txt while it sits in the index freezes it there, because Googlebot can no longer fetch the page to see the noindex tag.
The robots.txt.liquid Edits to Apply
Shopify lets you append rules through a robots.txt.liquid template. Two rules cover most stores:
{%- if group.user_agent.value == '*' -%}
Disallow: /*?*filter.v.availability=
Disallow: /*?*filter.v.price*
Disallow: */collections/*filter*&*filter*
{%- endif -%}
The third rule is the one our team deploys most often. It blocks any URL that matches two or more combined filters, which is where most of the extra URLs come from, while single-filter URLs remain crawlable.
Which filters are safe to block?
Availability and price are the safest individual filters. Almost no search query maps to them.
Color, size, and material deserve more care, since those are the combinations you might promote to real pages later.
Never block with a blanket Disallow: /*?* rule. It takes out pagination and legitimate parameters with it.
| Safe to block | Block with care | Never block |
|---|---|---|
| Combined filter URLs (2 or more parameters) | Single color, size, or material filters | ?page= pagination |
| Availability and price filters | Filters that map to real queries | A blanket /*?* wildcard |
The blocking decision at a glance. When in doubt, block the combinations and leave the singles alone.
Here's the same approach live on a client store:

A live client robots.txt. The rule in the red box blocks any URL carrying two or more filter parameters, while single filter URLs stay crawlable in case they earn promotion to real pages later.
This is a 20-minute edit that belongs on every technical SEO checklist for stores with more than a few hundred SKUs.
Two cautions before you touch the template:
First, keep a copy of the default rules. Shopify's stock robots.txt already does useful work you do not want to undo.
Second, run the Search Console robots.txt tester again after every edit. One misplaced wildcard can block collections you rely on.
Your sitemap needs the same discipline.
Shopify's generated sitemap only includes canonical URLs, which is correct. Resist any app that injects filtered URLs into it: a sitemap full of parameter URLs invites exactly the crawling you just blocked.
When a Filter Deserves Its Own Collection Page
Here's where it gets interesting:
Some filter combinations should not be blocked at all. They're pages shoppers actually search for. A collection page built for a real query can rank in a way a parameter URL never will. Case in point:

The top organic result for a granular product query is a Shopify collection URL, /collections/chocolate-covered. A collection page built for this exact query, ranking above every recipe and marketplace page.
This is the strategy we advise on most catalogs past a few hundred SKUs: identify the filter combinations with verified demand and promote them to dedicated collections.
But not every combination qualifies. Run each one through the four checks below before you build anything. Spinning up a collection for every permutation, down to size level, produces the thin, cannibalizing pages the rest of this guide exists to prevent.

The four-check test to apply before promoting any filter combination to a collection page. Demand comes first, so the catalog and the SERP both support the page before it exists.
When a combination passes, build it properly. That means a smart collection with its own title, unique copy, and internal links from the parent collection.
A URL swap with a heading change fails the distinct intent check. It will split impressions with the parent instead of adding any.
The Case for Indexing Everything
Enterprise retailers do run granular indexable facets profitably. So the opposite approach is worth a look.
On catalogs with tens of thousands of products, nearly every combination clears the demand and depth checks by sheer scale. The long tail adds up to real revenue.
The failure mode? Copying that playbook onto a 300-SKU store.
The same architecture that adds traffic at 50,000 products adds thin pages and crawl waste at 300. That's why we run the decision per combination.
One more thing to watch when you build these pages: facet order.
A color-then-size URL and a size-then-color URL showing the same products are two more duplicates for the crawler. Pick one canonical facet order and generate pages only in that order.
The AI Crawler Wrinkle
Facet URLs also waste AI crawler visits.
Those crawlers fetch a fraction of what Googlebot does.
GPTBot, ClaudeBot, and PerplexityBot read raw HTML, respect robots.txt, and do not execute JavaScript. When 95% of the URLs a crawler can reach are filter permutations, those visits go to filter URLs instead of products.
The good news? The robots.txt.liquid rules above apply to those crawlers automatically. That makes facet control one of the cheaper AI SEO wins available.
The same cleanup also feeds the product discovery layer behind agentic commerce.
| Crawler | Executes JavaScript | Respects robots.txt |
|---|---|---|
| Googlebot | Yes, on a second wave | Yes |
| GPTBot | No | Yes |
| ClaudeBot | No | Yes |
| PerplexityBot | No | Stated, with documented lapses |
How the major crawlers handle JavaScript and robots.txt. AI crawlers see only raw HTML, so a fetch spent on a facet URL is a product page they never read.
Now It's Your Turn
Faceted navigation on Shopify is not a duplicate content problem. The canonical defaults already handle that half.
It's a crawl allocation problem.
Fixing it gets real pages discovered faster, by Google and by AI crawlers.
So here's what to do: run the Search Console check this week. If the excluded bucket is several times your indexed count, you know where the crawl budget went.