Index Bloat: Why Your Shop Has 140,000 Pages

A shop with 4,000 products reports 140,000 indexed URLs. This is the single most common serious technical problem in eCommerce SEO, and almost every catalogue over a certain size has some version of it.

Where the URLs come from

Faceted navigation. Every filter combination generates a URL. Six filters across forty categories produces hundreds of thousands of permutations, and each is a crawlable, near-duplicate page.

Add internal search result pages, sort-order parameters, session identifiers, pagination and tag archives, and the count climbs further.

What it costs

Crawl budget goes to the wrong places. Google allocates finite crawl attention. Spend it on filter permutations and your genuine product pages get crawled less often — so price changes, stock changes and new products take longer to register.

Quality signals get averaged down. Site-wide quality assessment does not ignore the thin pages just because you did not mean to publish them.

Duplicate clusters split relevance. Three near-identical URLs competing for the same query means none of them accumulates the signals it needs.

Diagnosing it

• Search Console → Pages. Compare indexed count against your actual page count.
• Look at "Crawled — currently not indexed". A large number here is Google telling you it found the pages and decided they were not worth storing.
• Run site:yourdomain.co.uk inurl:? to see how many parameterised URLs are indexed.
• Crawl the site with a tool that follows filter links. The URL count usually surprises people.

Classify every facet

The fix is not to block everything. Some filter combinations have genuine search demand and deserve to rank. Classify each into one of three buckets:

Index. Combinations with real search volume — "womens waterproof walking boots" is a query people type. Give these a proper landing page with unique content, a self-referencing canonical, and internal links.

Crawl but do not index. Combinations with some value but no independent demand. Let Google crawl them so it can follow links to products, but canonical them to the parent category.

Block. The long tail — multi-filter combinations, sort orders, session parameters. Block in robots.txt and do not link to them internally.

Implement consistently

This is where most attempts fail. Four mechanisms must agree:

• robots.txt — blocks crawling of the long tail.
• Canonical tags — consolidate the crawlable middle tier.
• Meta robots — for anything that must be crawled but not indexed.
• Internal linking — do not link to URLs you do not want crawled.

The classic error is blocking a URL in robots.txt and putting a noindex tag on it. Google cannot crawl the page, so it never sees the noindex, so the URL can persist in the index indefinitely. Pick one mechanism per URL pattern.

Platform notes

Shopify gives limited robots.txt control and forces /collections/ and /products/ structure, with the same product reachable through multiple collection paths. Canonical handling is the main lever.

WooCommerce generates attribute and tag archives by default; most stores should disable them.

Magento layered navigation is the worst offender by volume and needs explicit configuration.

How long to see results

Deindexing takes time — Google must recrawl each URL to notice the change. Expect three to six months for a large catalogue to fully resolve. The crawl budget benefit arrives sooner, usually within weeks, because Google stops spending attention on blocked patterns almost immediately.