We Crawled 7,000 Sites: Most Flagged Problems Repeat Across a Layout

We crawled 7,000 websites, 1,000 on each of 7 platforms, and counted where the problems our crawler flags sit. On the median site, 75.3% of all flagged page problems are on 90% or more of the pages of one layout. A first wave of 2,800 sites gave 75.4%. The same 2,800 sites crawled again gave 75.3%, and 4,200 sites we had never crawled gave 75.3%.

See where your own funnel leaks

Funnels, recordings and heatmaps for your own pages, counted from every visit.

Start free

Check your site free first: an SEO audit of up to 25 pages, no account.

So a site’s problem list is mostly a short list repeated across many pages. Below are the figures by platform, what each platform’s sites have in common, what we found in sitemaps, and what the study cannot tell you.

What is left out of every figure below. 353 of the 7,000 crawls (5%) were stopped by our time limit before they finished, and only complete crawls are counted. The share is not the same on every platform: WooCommerce 10.5%, WordPress 9.2%, custom 5.6%, Shopify 5.5%, Webflow 1.9%, Wix 1.7%, Squarespace 0.9%. The one table that includes them says so.

What did we crawl?

  • 7,000 sites, 1,000 per platform: Shopify, WooCommerce, WordPress (without WooCommerce), Webflow, Wix, Squarespace, and “custom”, meaning sites that show none of those six platforms’ marks on their home page.
  • Where the sites come from: the Tranco list of popular domains.[1] These are live sites with traffic. They are not a random sample of each platform’s sites.
  • When: 12 to 18 November 2026. 2,800 of the 7,000 are the sites of our first wave, which ended on 9 November. We crawled them again to see whether the figures hold.
  • How much of each site: up to 1,000 pages, within a time limit of 90 minutes per site.
  • How: with ConvertlyftBot,[2] Convertlyft’s own crawler. It says who it is, obeys robots.txt and slows down when a site asks it to. It read the HTML each server sent. It did not run a browser.

5,860 of the 7,000 sites are in the figures (83.7%). The rest are left out: 342 were stopped by the time limit, 172 blocked or refused the crawler or could not be read, 10 gave no report because the crawler ran out of memory, and 616 finished with fewer than 10 pages. The note above counts 353 stopped crawls because 11 of the 172 also ran out of time.

We do not publish the list of sites, and no site is named here.

What does “layout-wide” mean?

The crawler groups a site’s pages into layouts by URL pattern and page structure: product pages, blog posts, category pages and so on. We call a problem layout-wide when it is on 90% or more of the pages of one layout that has 5 or more pages.

That is what we measured. We did not measure where a problem comes from. When a problem sits on 90%+ of one layout’s pages, a shared template is the first place we would look, but that is our reading, not something we measured.

How much of a site’s problem list is layout-wide?

Take every (page, problem) pair the crawler flagged on a site, and ask what share of them belongs to a layout-wide problem. The table gives the median site and the middle half of sites for each platform.

PlatformSites countedShare of flagged page problems that are layout-wide, medianMiddle half of sites
WooCommerce83378.3%69.9% to 83.9%
Shopify91178.1%72.7% to 83.0%
Custom68477.2%66.9% to 84.3%
WordPress82276.6%68.3% to 82.6%
Squarespace87675.4%69.3% to 80.9%
Webflow92471.9%64.1% to 77.9%
Wix81067.6%57.0% to 75.6%
All seven, counted equally5,86075.3%67.0% to 81.7%

Crawls stopped by the time limit are left out: see the note at the top.

Three things to know when reading it:

  • Nearly every site has repeating layouts. 98.7% of the sites have at least one layout of 5 or more pages. The median site has 6 of them, and they hold 94.1% of its audited pages.
  • The median site has 23 layout-wide problem types. The middle half of sites has between 18 and 29.
  • Bigger sites show a higher share. The median is 62.6% on sites of 10 to 99 audited pages, 72.9% on sites of 100 to 499, and 78.8% on sites of 500 or more. 44.4% of the sites hit our 1,000-page cap. For those, every figure describes the first 1,000 pages. Their median is 79.8%, against 70.9% for sites we read in full.

Which layout-wide problems does each platform have?

Some problems are layout-wide on almost every site of one platform and rare on another. The table gives the share of sites where the problem is on 90%+ of the pages of at least one layout.

ProblemCustomShopifySquarespaceWebflowWixWooCommerceWordPress
Resource links that start with // instead of https://26.1%97.1%95.3%40.3%7.9%45.5%37.5%
Links to other sites that open a new tab without rel="noopener"58.1%61.5%90.1%90.9%23.9%64.5%64.5%
No HSTS header on https pages51.8%1.3%4.4%15.3%3.9%71.4%66.7%
No <main> landmark for screen readers54.5%5.4%10.7%77.9%11.9%51.6%46.1%
Links to the site’s own URLs that its robots.txt blocks25.2%59.6%70.3%8.2%1.5%20.6%9.6%
Sites counted (those with a layout of 5+ pages)664911872910788829811

Crawls stopped by the time limit are left out: see the note at the top.

Read across a row and the platforms split into groups. Missing HSTS is layout-wide on about 7 in 10 WooCommerce and WordPress sites and on about 1 in 100 Shopify sites. Resource links that start with // are layout-wide on more than 95% of Shopify and Squarespace sites and on 7.9% of Wix sites.

These five are examples, picked because the platforms differ widely on them. They are not the five most common problems.

What do sitemaps say about dates?

A sitemap can give each URL a last-modified date. Search engines can use it to decide what to crawl again, so it only helps if it is true.[3]

  • Shopify: the dates are stamped at once. We could judge 794 Shopify sitemaps. In 729 of them (91.8%), 90% or more of the dates fall inside one hour. On WooCommerce it is 13.9% (84 of 604), on WordPress 13.3% (77 of 578), on Webflow 4.0% (26 of 651), on custom sites 27.2% (46 of 169).
  • Squarespace and Wix: a date without a time. Most entries carry only a day on 96.6% of Squarespace sites and 97.5% of Wix sites that give dates. That is valid. It also means we could judge only 23 and 16 of their sitemaps for one-hour stamping, so we give no figure for them.
  • Custom sites: often no date. 20.9% of the custom sites whose sitemap entries we could read give no dates at all (100 of 479). And we found a sitemap on only 71.5% of custom sites (489 of 684). On the six platforms we found one on 90.7% to 99.5%.
  • Webflow: the sitemap is behind the page. On 52.1% of Webflow sites, the crawler found sitemap dates a week or more older than the date the server itself sends for the page. On every other platform it is under 10%. This is one of two figures that moved by more than 5 points when we crawled the same sites again: it was 59.1% in the first wave and 52.3% on those same sites in this crawl.

Crawls stopped by the time limit are left out: see the note at the top.

What do sitemaps list that should not be there?

A sitemap is the list of URLs a site asks search engines to index. We checked what each listed URL does when you ask for it. The table gives the share of sites whose sitemap lists at least one URL of each kind.

The sitemap lists at least one...AllHighest platformLowest platform
URL that redirects27.1%WooCommerce 53.4%Shopify 3.2%
URL whose canonical tag points to another URL26.2%Squarespace 90.2%Shopify 3.5%
URL marked noindex16.0%WooCommerce 31.8%Squarespace 4.0%
Broken URL (answers that only refuse our crawler are not counted)13.1%Custom 21.7%Wix 5.5%
URL that the site’s robots.txt blocks4.9%Webflow 10.1%Wix 0.1%

5,386 sites counted: those in the figures whose sitemap we read. Crawls stopped by the time limit are left out: see the note at the top.

One more result from the same family. We asked each site for an address that cannot exist: a random name in a folder we had just crawled. A site should answer 404. On 33.7% of Wix sites and 24.1% of custom sites it answered 200 with a page. On Squarespace it is 1.5%, and on the other four platforms between 4.1% and 6.6%.

Which platform is hardest to crawl?

Shopify, by a wide margin. This table is about the crawls and not their pages, so it includes the crawls stopped by the time limit: 6,929 crawls that left a full report, 344 of the 353 stopped ones among them.

ShopifyThe other six platforms
Crawls that got a 429 or 503 answer at least once89.5% (875 of 978)3.3% to 21.8%
Crawls shown a bot-protection page at least once83.9% (821 of 978)0.3% to 8.4%
Median crawl time2,055 s144 s to 951 s

A polite crawler slows down when it gets those answers, so a Shopify crawl takes longer. It also bends one figure: among the complete Shopify crawls, “slow response” is flagged on 88.8% of the 830 sites that answered 429 or 503, and on 63.0% of the 81 that did not. Part of that slowness is the store holding the crawler back, not the store being slow for visitors.

Do the figures hold when you crawl again?

This is why we crawled the same sites again. The tables behind this article hold 582 percentage figures that have 50 or more sites behind them.

  • Same 2,800 sites, crawled again the week after: 2 of the 582 figures moved by more than 5 points. None moved by more than 10. The two are the Webflow sitemap figure above and one-hour date stamping on custom sites (29.2% in the first wave, 23.8% in this crawl).
  • 4,200 new sites against the first wave: 40 of the 582 moved by more than 5 points. Two samples of these sizes can differ by about 7 points by chance alone when a share is near 50% (our calculation, 95% range).

See where your own funnel leaks

Funnels, recordings and heatmaps for your own pages, counted from every visit.

Start free

Check your site free first: an SEO audit of up to 25 pages, no account.

Questions

What can this study not tell you?

  • It is not the whole web. The sites come from a list of popular domains. “All seven” counts each platform equally. It is not an average of the web.
  • No browser. Every check reads the HTML the server sent. A site that builds its pages with scripts can look worse than it is on content checks.
  • First 1,000 pages. 44.4% of the sites are bigger than what we read.
  • Stopped crawls are left out. Most of the 353 were stopped because the crawler was being polite: it was backing off after 429 or 503 answers, or held by its own limits such as a robots.txt crawl delay. So the figures leave out more of the sites that slow crawlers down.
  • A problem is what our crawler flags. Another tool draws the lines elsewhere. The crawler’s list covers indexing, sitemaps, links, headings and meta tags, accessibility, security and speed.
  • Where a problem comes from is not measured. “Layout-wide” says where a problem sits.

How did we check the figures?

  • A second program. Every figure in the tables was computed twice, by two programs written separately (one in JavaScript, one in Python), from the same per-site data: 1,054 figures, 0 differences.
  • The raw data. On 140 sites, 20 per platform, we counted pages per problem from the raw crawl tables instead of the crawler’s report: 7,924 of 7,924 counts equal. Layouts matched on 140 of 140 sites. Every one of the 16,542 layout-wide (layout, problem) pairs in the reports was confirmed; the raw count found 35 more (0.2%).
  • Not checked against raw data: the sitemap figures, the made-up-address figures and the crawl figures for Shopify. They are the reports’ own numbers, added up twice, but not recomputed from the raw tables.

What should a site owner do with this?

Sort your problem list by layout before you sort it by page. If the same problem is on 90%+ of one layout’s pages, look first at what those pages share. On the median site in this study, that covers three quarters of the list.

Check your site free first: an SEO audit of up to 25 pages, no account.

Sources

  1. Tranco, a ranking of popular domains (its title: A research-oriented top sites ranking hardened against manipulation). Page checked 19 November 2026. ↩︎
  2. Convertlyft, ConvertlyftBot: it identifies itself as ConvertlyftBot/1.0, reads robots.txt first and follows it, honours Crawl-delay, and slows down when a server answers 429 or 503. Page checked 19 November 2026. ↩︎
  3. Google, Search Central documentation on sitemaps (its title: Build and submit a sitemap): Google uses the <lastmod> value if it is consistently and verifiably accurate. Page checked 19 November 2026. ↩︎

Start on Pro, free for 14 days

Your new account gets 10,000 visits a month, no card needed. After 14 days it moves to Free, up to 5,000 visits a month with every feature, unless you choose Pro.