The distinction that matters​

An audit produces a list. A programme produces a system that stops the list from growing.

The test of whether you have the second thing is simple and uncomfortable: ask who is responsible for the URL space. If the answer is a person in a marketing team with no ability to change a routing rule, you have an audit programme and not a management one. If the answer is a named engineering team with the URL surface in their remit, you have a programme.

Ownership is where it actually fails​

On a large site the URL space is written by people who are not thinking about indexation: the merchandising team adds a colour filter because customers asked for it, the campaign tool appends a tracking parameter, the international team launches a new locale, the app team ships a client-side route. Every one of these is individually reasonable. Collectively they produced the 900,000.

Organisations attempt to fix this with policy. Policy fails because the people breaking the surface do not read it and are not measured by it. It works when ownership is explicit and when the teams that generate URLs have indexation in their objectives.

That last column is the change. Not "tell the teams about SEO" but "give each team one URL-space decision and make them accountable for it". Two decisions per team. It is achievable, and once each team owns decisions, the audit conversation becomes a review rather than a confrontation.

Change control: the deployment gate​

Large sites generate URLs through deployment, which means the intervention point is the release process.

The pattern that works is unglamorous. A short, automated check runs against any change that touches templates, routing or URL generation:

Nobody needs to write this by hand as a process document. It can be a set of assertions in CI against a crawled baseline. The value is that it moves the conversation from "we found this after the audit" to "the release was blocked and someone made a decision".

At enterprise scale, the cheapest SEO win is not a fix. It is a check that stops the problem recurring, added once.

Multi-brand and multi-domain​

Groups operating several domains face a specific set of problems that single-site advice does not cover.

The pattern to avoid is the one that produces the most waste: a microsite per campaign, each independently built, each with its own duplicate brand pages, each left alive after the campaign ends. A campaign registry is a dull artefact and it removes a surprisingly large share of a group's indexation problem. It needs three fields — owner, launch date, expiry date — and a report of expired campaigns still live.

International complexity​

Hreflang at enterprise scale is less about the syntax and more about the operational process, because a misconfigured implementation at this scale is worse than none.

The recurring failures are specific and checkable: return tags missing, self-references absent, hreflang pointing to redirected URLs, and pages in the index for locales that no longer exist. Each is a pattern-level check rather than a URL-level one, which is fortunate, because on an enterprise estate the URL count makes URL-level checking impossible.

Also worth stating: hreflang is a hint about equivalence and language, not a directive. It does not make a page indexable, and it cannot rescue a page that is blocked or noindexed.

Migrations as the highest-risk project​

Most enterprise SEO damage is caused not by neglect but by a migration. The reason is structural: a migration is the one moment when a large number of simultaneous URL changes are made with a partial view of the consequences.

The one mitigation that reliably reduces migration risk is a pre-cutover crawl of staging, compared against a crawl of production. Pattern-level differences surface in minutes, and they catch the majority of these failures before launch rather than after.

Redirect mapping is the other discipline worth insisting on. A mapping table with machine-checkable coverage — every old URL either maps to exactly one new URL or is explicitly retired — is worth more than any amount of post-launch analysis, because post-launch analysis happens after the equity is already gone.

Measuring the programme​

Programme-level measurement should be boring, and boring is the point. Quarterly, pattern-level, with sample sizes printed:

Throughput and regression rate are the two that distinguish a programme from a project. A project completes its list. A programme sustains a low regression rate, which is only measurable if you re-measure on a schedule and compare against a stored baseline.

The metric that gets left out is the one finance cares about most, and there is a reason: an indexation programme has a long and uncertain payoff that is very hard to attribute. Be honest about that rather than manufacturing attribution, and report the mechanism — URLs removed, patterns fixed, coverage moved — which is directly measurable and defensible. The sampling confidence and extrapolation page covers how to report those with stated intervals, which is what keeps the programme's credibility intact over years rather than quarters.

Treat the surface as debt​

The most useful mental model for an enterprise is that every generated URL is a liability with an ongoing carrying cost. It must be crawled, rendered, stored, migrated, analysed and — one day — removed.

That framing changes which arguments win in a planning meeting. "Add a colour filter" stops being a small feature request and becomes "commit the organisation to carrying roughly 4,000 additional URLs, each of which must be crawled, analysed and migrated at every future re-platform". For a filter that three percent of visitors use, that calculation usually ends the conversation, which is the correct outcome and one that is almost impossible to make with an audit report alone.

Where to start​

Pick the two teams that generate the most URLs and give each one named ownership of one URL-surface decision. Add the release checks for robots.txt, sitemap and canonical changes. Then establish a quarterly pattern-level measurement with a stored baseline.

Everything else — tooling, automation, vendor selection — is secondary to those three things and will not compensate for their absence.

PruneSEO supports the measurement half of the programme across multiple properties. Agency workspaces and permissions covers running it across brands and clients without cross-contamination, health and reporting covers the stored baseline and the regression comparison, and remediation campaigns is how a pattern-level finding becomes a tracked change rather than a line in a report. For multi-site structures, plans and upgrading covers how properties are counted on a plan.

The five checks before anything else​

Almost every investigation starts by skipping these and looking at Search Console coverage, which is usually where the symptom is least visible.

1. Did the site deploy? Check the release timeline. On large sites a deploy is the single most common cause of a sudden change, and it is invisible in the data you are looking at.
2. Did robots.txt change? Diff it against the previous version. Almost nothing else produces a step change in crawl traffic.
3. Did response times change? p95 for Googlebot, not average for humans. A CDN rule or a slow query added in a release will slow crawls without slowing users.
4. Did the sitemaps change? A sitemap index that lost a child file is a large, silent removal.
5. Is the WAF still letting Googlebot through? Check the logs for 403s by user agent. This is checked last because it is least likely, and it produces the most confusing symptoms.

Run those five before opening a single coverage report. Each takes minutes and together they eliminate the majority of incidents.

The symptom index​

Symptom: crawl traffic collapses​

A step change of more than half in Googlebot fetches is almost never a ranking decision. It is a mechanical failure and it is diagnosable from logs in a few minutes.

# Fetch volume and status mix, day by day, by user agent
awk '{print $4, $0}' access.log \
  | grep -E 'Googlebot|bingbot' \
  | sed -E 's#^([0-9/]+).*"([A-Z]+) ([^ ]+).* ([0-9]{3}) .*#\1 \2 \4#' \
  | sort | awk '{key=$1" "$3; c[key]++} END {for (k in c) print k, c[k]}' \
  | sort

Read the status column. If the fetches are still arriving and returning 403, your web application firewall is the cause — check whether a bot-protection rule went live with the release, and whether the rule tests for a user agent header that your proxy strips. A WAF that inspects the wrong end of a proxied request is a classic.

If fetches have dropped to nothing and the remaining ones return 200, check robots.txt and check origin availability. A Disallow: / deployed by mistake takes a site off the crawl map within a day and looks identical from the outside.

If fetches are arriving and returning 429 or 503, the origin is the cause and the correct fix is to reduce demand, not to raise the limit.

Symptom: one template is entirely unindexed​

A whole template failing is far more common than a page failing, and it almost always has one of four causes.

Sample at least 20 URLs from the pattern before concluding. A template of forty thousand URLs where you checked three will produce wrong answers at least twice a month, because a single misconfigured page in the sample looks identical to a template-wide fault.

The client-rendering case is the one that fools people most often. curl the URL, find the meta tags and the canonical, see them present, conclude the template is fine — when what the crawler received contained no article body at all. Check the response for actual text volume, not for tag presence.

Symptom: the wrong URL is indexed​

Canonical tags are routinely ignored for reasons that have nothing to do with whether they are correctly written.

The third of those is the most commonly missed and the easiest to check. If your canonical target 301s anywhere, every page pointing at it is being asked to canonicalise to a non-canonical URL, and the request is discarded.

And the last one links back to the earlier rule: a URL you want in the index must be crawlable. Canonicalising to a robots-blocked URL does nothing at all.

Symptom: sitemap errors​

Sitemap warnings look alarming and are usually cheap.

The two rows worth internalising are the second and the fourth. A URL that is blocked in robots.txt and listed in a sitemap is sending two contradictory instructions, and the general behaviour is to distrust both. The same applies to a noindex page listed in a sitemap. Both situations are usually invisible on small sites and represent tens of thousands of URLs on large ones.

Symptom: organic decline after a release​

If the decline began on a deployment date, treat it as a regression until proven otherwise. The check order:

1. Redirects. Confirm that every retired URL still resolves. A dropped 301 becomes a 404 and a chain of 301s adds hops. Both are common migration regressions.
2. Canonicals. Check a sample across templates for a template-level tag that now points somewhere unintended.
3. Status codes. Any template now returning 302 where it returned 200, or a soft 404 introduced by a new layout.
4. Rendered output. A new client-side shell that drops content from the initial response.
5. Sitemap submission. A release that regenerated sitemaps with a different URL format invalidates the submitted set.

The useful discipline is to diff the crawl, not to reason about it. Crawl the site before and after, compare by pattern, and the regressions are immediately visible as status or canonical changes on specific patterns. Diffing by pattern rather than by URL is what makes it tractable on a large site — the differences cluster into perhaps a dozen families.

Symptom: one section declines, the rest hold​

A localised drop is a strong diagnostic signal because the rest of the site is acting as a control group.

Run the same question against the same cluster in Search Console for a wider date range. If the decline predates the current release by months, this is not an incident; it is a slow structural problem and it needs a pattern analysis rather than a rollback.

A reproducible triage sequence​

Whatever the symptom, the same four-step discipline resolves most cases:

1. Establish a baseline. The state before the change, from logs and Search Console, not from memory.
2. Group by pattern. Every finding belongs to a URL family, not to an individual URL.
3. Sample and state the sample size. Every verdict carries the number of URLs inspected.
4. Change one thing, then wait long enough. Indexing changes take weeks. Changing five things means five weeks with no information about any of them.

That last point is where most troubleshooting actually fails. Teams make a bundle of changes in a maintenance window, see no improvement in two weeks, conclude none of it worked, and revert everything including the parts that were working.

Where to start​

Run the five checks at the top before you investigate anything else. Then, if the problem is not an incident but a structural one, go and get a pattern-level view of the site rather than continuing to work symptom by symptom.

PruneSEO is built for the second half of that sentence. Troubleshooting covers the in-product version of the checks above, running and monitoring crawls covers crawl execution and status breakdowns, and triage and verdict classification is where a sampled inspection result becomes a defensible verdict with a stated confidence. When you find the pattern, remediation campaigns is how the change gets tracked to completion rather than forgotten in a ticket.

The shared surface, honestly​

Read that last five rows before the first nine. A surprising amount of published "SEO and accessibility" material is entirely about those five, and a surprising amount of the actual engineering value is in the first four.

The parse is the shared artefact​

Both disciplines need to answer the same question: what is this page made of, and how is it arranged? A screen reader needs it to be answerable by an assistive technology reading linearly. A crawler needs it to be answerable by extracting entities and relationships. When a page answers it with headings and elements, both get what they need from the same markup.

When a page answers it with styling and JavaScript, both get nothing.

That is the whole argument, and it is why "make the HTML semantic" is simultaneously the correct accessibility advice and the correct SEO advice. It is not a compromise between the two. It is the single highest-leverage change available to most pages, and it is free.

Links are where the overlap pays​

Link text is the clearest case. Two pages with identical anchor text everywhere have an accessibility defect and an SEO defect, and they are the same defect.

<!-- Both audiences lose. No context out of context. -->
<p>Read <a href="/specs/zinc-plating">more</a> for details.</p>
<p>See <a href="/specs/zinc-plating">our guide</a> and
   <a href="/specs/anodising">here</a>.</p>

<!-- Descriptive, unique, and usable without surrounding context. -->
<p>Read the <a href="/specs/zinc-plating">zinc plating specification</a>
   for thickness tolerances.</p>

On a large site this is a mechanical fix that applies across tens of thousands of pages at once. Crawl the site, group anchor text values by target pattern, and the ones appearing on many internal links are the defects. It is the same crawl you run for indexation work, which is the practical reason to do both together rather than commissioning two separate audits.

There is a second link-related rule that people get wrong in both directions. A link whose text is identical across a cluster is a problem; a link whose text is keyword-stuffed to compensate is a worse one, and it is a pattern you can detect in the same crawl. If the anchor text for every product link is "buy PRODUCT online cheap", you have traded an accessibility problem for a spam problem.

Alt text, filenames and structured data: three different jobs​

Image optimisation gets conflated constantly, so it is worth separating three mechanisms that are frequently presented as alternatives when they are complementary.

A useful discipline: the filename names the thing, the alt text describes what it conveys, and the structured data declares the relationship. A product page with industrial-pipe-fittings.jpg as the filename, "Four-way galvanised pipe fittings arranged on a workshop bench" as the alt text and a Product block with an image property is doing all three jobs correctly.

Where accessibility and search genuinely diverge is decorative imagery. A divider or a tracking pixel needs empty alt="", which is the correct accessibility markup, and several teams "fix" this by removing the attribute entirely. That produces an image with a synthesised, meaningless description read aloud to a screen-reader user, and it is one of the most common accessibility regressions on commercial sites. For search it makes almost no difference. Leave it empty.

Where the two disciplines collide​

The genuine conflict is client-side rendering, and it is worth being direct about it.

A single-page app is often the accessible solution — focus management, routing announcements, dynamic content updates announced politely to assistive technology — and it is also, in its default configuration, the worst possible architecture for a crawler that does not execute JavaScript reliably or promptly.

The resolution exists and is well established: server-render or pre-render the initial view, hydrate the rest. Both audiences get what they need. The teams that do not do this typically argue that accessibility and SEO are in conflict, which is almost never true. It is usually that one of the two requirements has not been specified.

The second collision is more mundane and more common on large sites: accessibility remediation adds pages. A separate accessible version of a PDF. An accessible text alternative for a complex chart. A dedicated help page explaining keyboard navigation. Each is a defensible accessibility artefact, each is a new URL, and together they can add a few thousand pages of near-duplicate content to a site that already has an indexation problem.

The right answer is to govern them the way you govern any other URL family: decide the canonical version, noindex the alternates, keep them out of the sitemap, and check them in the same pattern analysis as everything else. Adding them to the accessibility backlog and adding them to the indexation inventory is the same hour of work, done once.

What has no search effect at all​

Worth stating without hedging, because these get bundled into SEO strategy decks:

• Colour contrast ratios. Meet 4.5:1. It has no measurable ranking effect.
• ARIA roles, states and live regions. These describe behaviour to assistive technology. The visual DOM they annotate carries the search signal.
• Focus order and visible focus indicators. Purely interaction.
• Skip links. Useful for keyboard users, invisible to crawlers beyond being a link.
• Touch target sizes. A mobile usability concern.
• Reduced-motion preferences. Relevant to rendering stability, which touches Core Web Vitals indirectly and marginally.

There is no ranking benefit for being WCAG compliant, and there is no accessibility benefit for being well indexed. Treating them as one budget produces a report that talks to neither audience.

The correct framing is that they share a root cause and mostly separate from there. Semantic, correctly structured HTML serves both. Everything downstream of that is largely a single-discipline concern, and conflating them makes it harder to get either one right because the success criteria become mushy.

Auditing both from one crawl​

There is real efficiency in doing this together, and the method is straightforward. A single crawl that captures response status, canonical, heading outline, alt-text presence and coverage, anchor text distribution and link destination quality can produce both an accessibility findings report and an indexation report from the same data.

The last row is the one teams find most valuable. Template-level defects are worth fixing in the template, once, and they tend to surface in both reports simultaneously because both are symptoms of a page that was assembled without reference to its structure.

Where to start​

Pick a high-traffic template rather than a page. Audit its heading outline, its link text and its alt coverage once, fix it in the component, and you have improved the accessibility of every URL using that template and the structure of every URL using it too. That is the argument for treating the overlap as real rather than rhetorical.

PruneSEO's crawl captures the structural signals that both reports need. Triage and pattern analysis groups findings by pattern so a template defect becomes one finding rather than forty thousand, site configuration covers which signals you choose to capture per run, and indexation waste detection is where the accessibility artefacts you have added get counted against the URLs you actually want indexed. For how the crawl is scoped before it starts, see site configuration.

There are two queues, not one​

The most useful structural fact is that crawling and rendering are separate processes with separate backlogs.

A URL enters the crawl queue when it is discovered through links, sitemaps or prior knowledge. It is fetched. The response is parsed, links are extracted, and new URLs enter the crawl queue. If the HTML does not contain the page content, the URL is added to the render queue, where a separate service executes JavaScript and produces a rendered version.

This explains a result that otherwise looks absurd: a page with perfect server logs, a healthy 200 response, correct content in the DOM after hydration, and no presence in the index. The crawl succeeded. The render queue is backed up — commonly on large JavaScript-heavy sites where the render queue is far deeper than the crawl queue — and the URL sits there.

It also explains why fixing render failures and fixing crawl waste are different projects with different owners. Crawl waste is a templates and robots problem. Render waste is a rendering architecture problem.

Crawl budget is a tolerance, not an allocation​

The phrase "crawl budget" implies a fixed number of fetches per day, granted and revocable. That is a convenient mental model and it is not what is actually happening. What exists is a per-site tolerance that Google adjusts continuously based on how your site responds to being crawled and how much the pages it fetches appear to be worth.

Practical consequences:

• There is no setting and no report showing your number.
• Tolerance is mostly a function of fetch cost, not page count. A site where the crawler fetches eight hundred thousand URLs a month and a site fetching eight million can sit at similar tolerances if the first one is fast and clean and the second is slow and dirty.
• The fastest way to lose tolerance is to make the crawler fetch things that return nothing. A 404 from a crawler is a wasted fetch, and the accounting is unforgiving.
• Adding pages does not lower tolerance by itself. Adding bad pages lowers it, because the crawler learns the site is not worth much attention.

Crawl budget is not the number of URLs you have. It is the rate at which Googlebot is willing to look, and you change that rate by changing what it finds when it looks.

Estimating your own​

Since the number is not published, you estimate it from logs. The method:

# Total daily Googlebot fetches over a fortnight
grep -c 'Googlebot' access.log.1 access.log

# Status breakdown — wasted fetches are the signal
grep 'Googlebot' access.log \
  | awk '{print $9}' \
  | sort | uniq -c | sort -rn

# Which patterns absorb the fetches?
grep 'Googlebot' access.log \
  | grep -oE '"GET /[^ ?"]+' \
  | sed -E 's#"GET (/[a-z0-9_-]+).*#\1#' \
  | sort | uniq -c | sort -rn | head -20

The status breakdown is the most important output and it is the one most teams never look at. A healthy large site might see 95% of Googlebot fetches returning 200. A site with a URL-space problem frequently sees 15 to 30% returning 404 or 410 from a facet that generates combinations which do not exist — and every one of those is a fetch that produced nothing.

Compare two weekly totals and you have your crawl rate. Compare that against the size of your intended indexable surface and you have the ratio worth putting in front of a stakeholder:

Site B fetches more than twice as much and gets less than half the coverage. It is not being throttled. It is spending its tolerance on 29% of requests that return nothing.

Crawl traps, ranked by cost​

Date archives are the one that surprises people. A URL pattern containing /archive/2026/12/day-31/ does not have a bound, because you will publish that URL again. With one archive link per page you have an unbounded discovery chain that a crawler will happily walk, indefinitely, producing mostly-empty pages.

The precise semantics of robots.txt​

robots.txt is more often misapplied than any other file on a large site. The rules are exact and the precedence order matters.

User-agent: *
Disallow: /*?
Disallow: /search/
Disallow: /cart/
Disallow: /account/
Disallow: /*?sort=
Disallow: /*?page=

# Refine: allow a parameterised family you DO want crawled.
# Longest-match-wins, not first-match-wins.
Allow: /*?collection=
Allow: /products/industrial/pipes/

Sitemap: https://example.com/sitemap-index.xml

Two things about that file that are usually misunderstood. First, the more specific rule wins, not the first one matched — Allow: /products/industrial/pipes/ will be crawled even though a broad Disallow: /*? would otherwise block it. Second, $ anchors to the end of the URL including the query, so Disallow: /*?$ behaves very differently from Disallow: /*?. Almost every over-blocking incident traces back to a missing anchor or an unanchored *.

What robots.txt cannot do​

This is the single most consequential misconception in the file, and it has its own category of folklore around it. Three things:

1. It cannot remove anything from the index. A blocked URL that is already indexed stays indexed. It can simply be dropped at crawl time, and it may reappear served from the index without the crawler ever seeing it again — which looks like the rule "not working" and is not.
2. It cannot be used for de-indexing. That is noindex, which the crawler must fetch in order to see. The combination is contradictory: a robots.txt block prevents the fetch that would deliver the noindex.
3. Googlebot does not support Crawl-delay. It is ignored. Any number in a production robots.txt is documentation, not control.

If you take one sentence away, take this one:

A URL you want out of the index must be crawlable. A URL you want out of your crawl budget must be blocked. Those are different requests and they need different files.

Which is why the correct sequence for retiring a large pattern is: block in robots.txt to stop further fetches, mark the template noindex so already-crawled pages can be seen and dropped, and remove it from the sitemap — all three, because any one alone leaves residue.

Mobile-first and the user-agent line​

The User-agent line is a targeting mechanism for rules, not a signal about rendering. There is no supported way to ask Googlebot to render the desktop version, and the mobile-first indexing model means the mobile version of your site is the one that gets indexed and ranked.

The practical consequences for a large site are mundane and important: if your mobile template drops the canonical tag, if it strips structured data, or if it renders content client-side only, then indexing is being judged on the version that is missing those things. Auditing the rendered mobile output is not optional.

Server behaviour the crawler cannot ignore​

The last row causes real incidents. Security teams regularly block anything on a bot-blocking list, and a misconfigured rule that returns 403 to Googlebot looks exactly like a server outage. If crawl traffic suddenly falls by 80% and nothing on the server changed, check the WAF before you check anything else.

Where to start​

Pull two weeks of logs and produce three numbers: total Googlebot fetches, the fetch-to-200 ratio, and the top five patterns by fetch volume. That is usually enough to identify the trap, and the trap is usually one of the eight in the table above.

PruneSEO measures the same things without the log spelunking. Running and monitoring crawls covers crawl execution and status breakdown, crawl budgets and allowances covers how fetch volume is distributed across patterns and what a removal is expected to release, and indexation waste detection is where waste per pattern is quantified against what the site intended to be crawled. If you want the practical consequences worked through in detail, triage and pattern analysis is the next step.

Three different delays wearing one name​

The word "delay" collapses at least three separate things, and confusing them produces most of the wrong advice.

On a large site with an indexation problem, the first one dominates and the other two are noise. A page published yesterday is not "not indexed yet" — it is most likely "not crawled yet", and no amount of index-related tuning will change that.

Crawl delay is the one that matters at scale​

A crawler has a finite number of fetches and a prioritisation function. For a small site, the function never binds: everything is fetched within a day or two. For a site with 900,000 addressable URLs, the function binds constantly, and your new page is competing against a queue that may already contain hundreds of thousands of stale entries.

The queue is populated by whatever the crawler has been told to keep looking for. Internal links, sitemaps, and previously seen URLs all feed it. So the delay on any individual new page is a function of how much noise the site presents:

The last row is the counter-intuitive one. A smaller sitemap gets crawled more thoroughly than a large one. Google treats sitemaps as a discovery hint and applies its own prioritisation on top, so an 8,000-URL sitemap of pages you actually want indexed is a stronger signal than a 400,000-URL sitemap of everything the CMS emits.

Measuring your own delay from logs​

You do not need to guess. Server logs will tell you the crawl delay precisely, per pattern.

# Sample access log format that makes this analysis possible
log_format seo_log '$remote_addr - $remote_user [$time_local] '
                   '"$request" $status $body_bytes_sent '
                   '"$http_referer" "$http_user_agent" '
                   '"$remote_addr" $request_time';

The two fields that matter for this analysis are the timestamp and the user agent. Split the log by crawler and look at what arrives:

# Daily Googlebot fetch volume, by status
grep 'Googlebot' access.log \
  | awk '{print $4, $9}' \
  | sort | uniq -c | sort -rn | head -40

# Fetch latency distribution — slow responses starve crawl
grep 'Googlebot' access.log \
  | awk -F'"' '{print $1}' \
  | awk '{print $NF}' \
  | sort -n \
  | awk '{a[NR]=$1} END {print "p50", a[int(NR*0.5)];
        print "p95", a[int(NR*0.95)];
        print "max", a[NR]}'

The three numbers that come out of that second command matter more than most sites realise. p95 response time above roughly 500ms for Googlebot degrades crawl rate measurably. On a large catalogue where uncached facet URLs hit the origin and render a full page, p95 is routinely above two seconds. The crawler does not complain, does not send you a message, and simply fetches less of your site per day.

Now add pattern breakdown, which is where the delay becomes actionable:

# Which URL patterns are actually being fetched?
grep 'Googlebot' access.log \
  | sed -E 's#.*"(GET|HEAD) ([^ ?"]+).*#\2#' \
  | sed -E 's#^(/[a-z0-9_-]+/[a-z0-9_-]+).*#\1#' \
  | sort | uniq -c | sort -rn | head -30

Run that on a large site's logs and the shape of the problem is usually unmistakable: the top three lines are search results, filters and pagination, and your new article is nowhere near the top. That is not an indexing bug. It is a priority problem, and priority problems are solved by removing demand, not by submitting more.

Why indexing delay itself happens​

Once a page has been fetched, a further delay applies before index inclusion. The common causes:

The render failure deserves emphasis because it is invisible in every tool you own. A page can return 200, contain its content only after JavaScript executes, and never enter the index. Search Console will report it generically. The server log will show it as a successful fetch, because from the server's point of view it was one.

The fixes that do nothing​

Worth stating plainly, because each is widely recommended:

• Submitting the URL to Search Console. Useful for a small number of URLs and for the diagnostic data it returns. It does not change crawl priority and does not shorten queueing.
• Adding the URL to a 400,000-URL sitemap. Adding one URL to a large sitemap changes essentially nothing. Removing 380,000 does.
• Ping services and "index now" third-party tools. At best neutral, and frequently a spam signal.
• Changing lastmod frequently. A lastmod that changes on every request trains the crawler to distrust it. Accurate lastmod is valuable; a constantly-firing one is worse than none.
• Internal links from every page. A sitewide navigation block that changes daily guarantees re-fetching of the whole navigation graph and nothing else.

The micro-sitemap, honestly assessed​

The micro-sitemap technique is the one intervention with a real mechanism, so it deserves an honest description rather than a dismissal or an endorsement.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/new-article/</loc>
    <lastmod>2026-12-08</lastmod>
  </url>
  <url>
    <loc>https://example.com/new-article/amp/</loc>
    <lastmod>2026-12-08</lastmod>
  </url>
</urlset>

What it does: gives the crawler a small, high-priority set of specific URLs, which is a different input from a large sitemap or a link on an unlinked page. Combined with the Sitemaps API, you can submit and then remove it, which prompts re-evaluation.

What it does not do: compel an index decision, guarantee anything, or scale. It is a two-URL tool. Applying it to a site with 120,000 URLs to index is a waste of the mechanism, and some teams end up submitting a 50,000-URL "micro"-sitemap, at which point it is just a sitemap with a different name.

The one thing worth saying firmly: every URL in one of these ought to be a URL you would be content to see indexed, because submitting it is a statement of intent rather than a neutral technical act.

The remedy that actually works​

The remedy is to reduce the queue. In order of impact on a large site:

1. Remove demand at the source. Block internal search results, facets you do not want indexed, and deep pagination in robots.txt. Demand that never forms a fetch never occupies queue space.
2. Shrink the sitemap to the surface you actually want indexed. This is the single highest-leverage sitemap action and it is counterintuitive.
3. Fix response times on uncached and parameterised URLs. p95 is the number to watch.
4. Consolidate duplicates so a cluster returns one indexable candidate rather than forty.
5. Only then use micro-sitemaps for genuinely new, high-value URLs.

Steps one and two are measured in crawl budget released. That is why crawl budgets and allowances is the first thing to look at when you are wondering why new pages are slow.

Where to start​

Run the log analysis above for a single week and produce one table: patterns fetched, share of total fetches, p50 and p95 response time. That table is the honest version of "why is my site slow to index", and it will usually identify the answer in the first ten lines.

PruneSEO does the pattern half of this for you. Running and monitoring crawls covers crawl execution and scheduling, setup and Search Console covers property connection and inspection quota, and triage and pattern analysis is where the fetched-versus-valuable comparison turns into a ranked list of patterns to fix. Start with the pattern list, because the queue is made of your own URLs.

Why organic is harder to attribute than paid​

Paid attribution is hard because of incrementality: you spend money and traffic moves, but how much would have happened anyway? Organic attribution is hard for the opposite reason — the input is not controlled, so there is no natural experiment to point at.

The structural problem is that organic revenue is a residual. It is what remains after direct, referral, paid, email and brand behaviour have taken their share of a total that moved for reasons unrelated to any of them. When total conversions drop, organic revenue "drops" proportionally even if organic sessions and conversion rate both held steady. That artefact alone accounts for a large share of the volatility in most organic revenue dashboards.

No single method is correct. What matters is that you pick one, state it, hold it steady, and stop comparing numbers that were produced by different methods.

The four problems that produce wrong numbers​

Bot and AI crawler sessions. Analytics tools filter bots with a user-agent blacklist that is several years out of date. Newly identified machine crawlers get logged as sessions, and on a site with a large crawlable surface that can be a meaningful share of recorded traffic. This inflates page views and deflates conversion rates across the whole dataset.

Sampling. At higher traffic volumes analytics switches to statistical sampling, typically by session. The error is small in aggregate and non-trivial for any specific landing page, which is exactly where SEO reports draw conclusions.

Consent mode and modelled data. Under consent regimes, analytics reconstructs some conversions. The reconstruction is good work and it is not observation. Segment performance figures by consent state before comparing segments against each other.

Last-modelled channels. Anything with "modelled" in the name carries a confidence band. Those bands belong in the report, next to the number, not in a footnote.

The seasonality trap​

The most expensive analytical mistake in SEO is the before-and-after comparison, because before-and-after is intuitive, easy to present and almost always wrong on a site with any seasonal component.

Consider an indexation remediation that lands in November and removes 180,000 URLs over six weeks. Two months later, organic sessions are up 12%.

Three explanations are consistent with that:

1. The remediation worked and lifted indexed coverage.
2. The site's seasonal peak began, as it does every year.
3. A competitor had a technical incident in the same period.

You cannot distinguish these from a single two-point comparison. This is the reason the field has an entire genre of chart that ends with a red line — and, to be fair, usually that chart is honest about the seasonality.

A defensible comparison method​

The fix is not more data. It is a method that admits uncertainty. Three approaches, in increasing order of effort and reliability.

Approach 1: Index-date alignment. Align the comparison period to the date of the change, not to a convenient reporting boundary. Compare the four weeks after the change against the eight weeks before, and against the same eight weeks in each of the previous two years. If the post-change period is up against both baselines, you have something. If it is up against one, you have a seasonal explanation.

Approach 2: Year-on-year with change-point marking. Plot two years of weekly data and mark every technical change. Trend lines through periods containing a migration, a redesign or a plugin change are not comparable to lines that do not. This is unglamorous and it is the single highest-value thing you can add to an organic report.

Approach 3: A matched holdout. If the remediation is reversible — a noindex deployment, for instance — hold a slice of the affected pattern out of the change for four to six weeks and compare the treated and untreated slices. This is the only approach that actually establishes incrementality, and it is why reversible remediations are worth preferring over irreversible ones even when the irreversible version is technically better.

week,segment,indexed_urls,organic_sessions,organic_conversions,organic_revenue,note
2026-04-06,all,131400,284100,1420,318400,baseline
2026-04-13,all,131900,290400,1455,326100,baseline
2026-04-20,all,130800,302900,1490,341800,pre-peak
2026-04-27,all,129100,318600,1610,372200,pre-peak
2026-05-04,all,117600,326400,1655,381900,noindex deployed 2026-05-02
2026-05-11,all,114200,331800,1690,389400,post-change
2026-05-18,all,113800,335200,1712,394100,post-change

Look at that extract carefully. Sessions rose 18% across the window. Sessions were already rising 12% in the fortnight before the change. The honest conclusion is that the remediation did not visibly damage anything and probably helped at the margin — not that it added 6% of growth. If someone presents the 18% as an effect of the change, they have made a causal claim the data does not support.

The genuinely attributable signal in that extract is the first column. Indexed URLs fell 13%, which is the remediation doing exactly what it was designed to do. That number is direct, measurable and immune to seasonality. Measuring the mechanism directly is usually more honest than inferring it from revenue.

The self-attribution filter​

One measurement is cheap and worth doing on every large site: how much organic traffic is actually branded.

-- Conceptual, not vendor-specific
SELECT
  landing_page_group,
  SUM(sessions)                                   AS sessions,
  SUM(sessions) FILTER (WHERE is_branded_query)    AS branded_sessions,
  SUM(sessions) FILTER (WHERE is_branded_query)
    / SUM(sessions)                                AS branded_share
FROM organic_sessions_daily
WHERE date >= current_date - INTERVAL '90 days'
GROUP BY 1
ORDER BY sessions DESC;

The output that changes decisions looks like this:

A 71% branded share on a landing page group means most of that traffic would have arrived anyway — someone who typed your brand name was not a discovery. Reporting it as SEO revenue overstates the channel badly. Conversely, the 6% branded share on help content is a genuinely earned result, and help content often has low commercial value per session precisely because it resolves a question rather than selling something.

If your branded share is above 50% across the site, your organic revenue figure is measuring brand demand rather than acquisition, and it will flatter every initiative equally.

Bot traffic from machine consumers​

If PruneSEO imports server logs, or if you analyse them yourself, expect to find machine crawler traffic that your analytics tool counted as human. A quick way to see the scale is to compare log lines to analytics sessions for a sample day:

# Conceptual sample, one day, grouped by user agent family
awk '{print $NF}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn \
  | head -20

The lines worth reading are the identified machine crawlers — Googlebot, GPTBot, ClaudeBot, PerplexityBot and their siblings. On a large catalogue site these can be a substantial percentage of total fetches. Two consequences:

• Analytics is overstating your page views, which deflates engagement metrics and makes content look worse than it is.
• Your capacity planning should include them. If four retrieval crawlers each pull a full crawl of a 900,000 URL site weekly, that is real origin load.

Running and monitoring crawls covers log import and how crawl frequency breaks down by pattern, which is where this shows up.

Where to start​

Three things, in order. First, split branded from non-branded organic and never report them merged. Second, put change points on your trend chart so nobody attributes a seasonal peak to an SEO intervention. Third, measure the mechanism directly — indexed URLs, coverage rate, crawl efficiency — because those numbers are attributable, and let revenue be the lagging confirmation rather than the headline.

PruneSEO's sampling confidence and extrapolation page is the one to read before you put any of these numbers in front of finance, because the difference between a defensible estimate and a misleading one is almost entirely about sample size. For how to keep the underlying findings consistent between runs, health and reporting covers scoring and history. And campaigns and artifacts is where a remediation gets its own baseline and its own change-point marker, which is what makes Approach 3 above possible next time.

Sorting the claim into three buckets​

Most published GEO advice is bucket two wearing a bucket three label. That is not a criticism of the practitioners — a lot of it is good advice delivered badly — but it is a problem for planning, because it makes a modest engineering task look like a strategy.

The genuinely new part: crawler policy​

For twenty years there was effectively one crawler that mattered and one robots.txt file that mattered. There are now several distinct machine consumers, and they are separated by purpose rather than by user agent string alone. The distinction that matters most for SEO is between crawlers that retrieve pages for search indexing and crawlers that collect content for training or grounding.

Google's own arrangement is the clearest illustration. Googlebot is the search index crawler and its behaviour is governed by Google-Extended, which is a separate control covering Gemini training and grounding. Blocking Google-Extended does not affect your rankings. Teams get this wrong constantly and occasionally block the wrong one.

A representative policy, with the reasoning attached:

# Robots.txt — machine consumers
# 1. Search indexing. Do not touch.
User-agent: Googlebot
Disallow: /search/
Disallow: /*?*sort=
Disallow: /*?*page=
Crawl-delay: 0

# 2. Google grounding and Gemini training. Optional, reversible.
User-agent: Google-Extended
Disallow: /

# 3. Other AI retrieval crawlers. Decide per crawler.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

# 4. Bulk training scrapers. Block as default.
User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /search/
Disallow: /cart/
Disallow: /account/
Allow: /

The important operational point: a crawler allowed in robots.txt still cannot fetch a URL that your server blocks, rate-limits or serves with a 429. Machine consumers amplify the failure modes you already have. If Googlebot is already struggling with a slow response time on facet URLs, adding four more respectful crawlers who each fetch a document set will not improve it.

Which is the GEO argument that actually lands on a large site, and it is not really about GEO at all.

The second new part: entity clarity​

Retrieval systems need to resolve what a page is about and what it is about in relation to other things. Schema.org structured data is the least ambiguous way to express that, and it is genuinely under-used on large sites.

The most valuable thing a large catalogue site can declare is the relationship between a product, its brand, its category and its availability:

{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Heavy Duty Galvanised Steel Shelf, 1200mm",
  "sku": "HDG-1200-GAL",
  "brand": { "@type": "Brand", "name": "Northgate Storage" },
  "category": "Industrial Shelving",
  "material": "Galvanised steel",
  "offers": {
    "@type": "Offer",
    "priceCurrency": "GBP",
    "price": "184.00",
    "availability": "https://schema.org/InStock",
    "itemCondition": "https://schema.org/NewCondition"
  },
  "additionalProperty": [
    { "@type": "PropertyValue", "name": "Load capacity", "value": "270 kg per shelf" },
    { "@type": "PropertyValue", "name": "Depth", "value": "400 mm" }
  ]
}

The additionalProperty block is doing the work that prose usually does badly. A specification is a table. A table extracts cleanly. A sentence explaining the load rating in forty words of marketing copy does not.

The same logic applies site-wide: an Organization block that names the legal entity and links it consistently to your brand, a BreadcrumbList that matches the visible breadcrumb, and — on a large site — a rule that every template emits the same shape. Structured data that varies by template is worse than none, because it produces contradictions a system has to resolve.

The third new part: extractable structure​

A retrieval system assembling an answer needs to lift a claim away from its context without losing the meaning. The design rules that follow are not exotic:

1. Answer first. The specification, price or definition in the first paragraph, not after the hero image and the marketing subhead.
2. One idea per heading. A heading that contains three questions extracts as three ambiguities.
3. Tables for structured facts, prose for everything else. Bullet lists for genuinely enumerable items, not for paragraphs split into fragments.
4. Explicit naming. "Schedule 40 PVC pipe" beats "our premium piping solution".
5. Stable, descriptive URLs. /products/schedule-40-pvc-pipe-50mm is a better extraction target than /p/88213.

None of these are new, which is the point. They are the same principles as good writing, good accessibility and good information architecture, arriving with a new justification.

llms.txt: what it is and what it is not​

llms.txt is a proposed convention: a plain-text markdown file at the site root intended as a curated entry point for language model retrieval. The proposal is sensible as documentation and weak as an optimisation.

What it is: a human- or curator-written summary of what the site contains, with links, that a system could read once and use as a map. Useful if you run a documentation site, a large catalogue or a knowledge base.

What it is not:

• It is not a ranking signal. No major engine has confirmed using it in retrieval.
• It is not enforced. There is no mechanism that makes a crawler prefer it.
• It is not a substitute for robots.txt, sitemaps or crawlable HTML.

If you add one, keep it current. A stale llms.txt is worse than none, because it is an authoritative-looking document that is wrong. PruneSEO's artifact generation and deployment is the natural home for a generated version of one, because the surface it describes is the same surface the crawl measures — but the judgement about what to include has to be yours.

The part that is unmeasurable, and how to handle it​

There is no reliable way to measure "visibility inside a language model". Every tool claiming to do so is inferring from indirect signals, and their confidence intervals are not published. The honest position is:

Server logs are the only honest measurement of machine consumption, and they are worth looking at because the volume can be surprising. A site with four identified AI crawlers hitting 200,000 URLs per month is not a future problem, it is a current capacity line item. PruneSEO's running and monitoring crawls guidance covers log import and pattern-level crawl statistics, which is where this shows up first.

Where to start​

Do the cheap new things and keep doing the old important things. Concretely: make the machine crawler decision explicit in robots.txt and write down the reasoning; validate structured data per template rather than per URL; put the answer at the top of the page; and then go back to the indexation problem you already have, which is almost certainly worth more.

The site configuration section covers crawl policy and how PruneSEO treats robots.txt and sitemaps during a crawl. For the structure-level question, vertical policy presets encodes what should be indexed by default for ecommerce, publishing and programmatic templates. And if you want the unglamorous version of this advice stated plainly, start with triage and pattern analysis, because the first thing any large site should check is how much of its machine-readable surface is worth reading in the first place.

What the click-through evidence says, and where it stops​

There is a substantial body of research on AI Overview click-through, and it needs reading carefully rather than quoting selectively. The consistent findings:

• Click-through declines on informational queries with a strong single-answer intent. A query like "how do I remove a background from a photo" no longer needs a click.
• Click-through is largely unchanged for navigational queries, where the user already knows the destination.
• Declines are smaller and noisier on commercial queries, though the direction is consistent.
• The effect is highly query-dependent. A median across all queries hides more than it reveals.

The failure mode in how this gets reported is treating the median as the finding. If your site sells industrial pipe fittings, the AI Overview change to "what is schedule 40 pipe" is noise and the change to "schedule 40 pipe fittings supplier near me" is not. Nobody's dataset can tell you which of these dominates your portfolio without your data.

Which raises the obvious question, and the honest answer is that most published analysis cannot tell you either.

Do not accept any AI Overview claim for your site that does not name the query set it was measured on. "Traffic dropped 30%" is not a finding; "traffic dropped 30% across our top 400 non-branded informational queries, measured year on year" is.

Three models of how citations work​

The mechanism matters because the correct strategy differs between them, and almost nobody is clear on which one they are operating in.

These are not mutually exclusive and the industry treats them as if they are, which is how you end up with contradictory advice. Retrieval is real: the systems that assemble an answer fetch pages, and the pages that can be quoted cleanly have a materially better chance of being selected. But extraction is an addition to ranking, not a replacement for it, and the pages that rank are still the pages that get fetched. Anything that damages ranking damages citation, and anything that damages fetching damages both.

The practical conclusion is unglamorous: the technical fundamentals that made a page rank — being reachable, being fast, being crawlable, having a coherent URL, having original content — are the same fundamentals that make it citable. There is no separate AI layer of optimisation.

Where this intersects with indexation​

This is the part most AI commentary skips entirely, and it is the part that matters most on a large site.

An AI system assembling an answer for a query does not typically read your whole site. It reads a small number of high-confidence sources. On a site with 900,000 addressable URLs and 120,000 indexed, the question of which 120,000 is not academic — it is the question of what is available to be retrieved from.

Three structural consequences follow:

1. Crawl waste becomes citation waste. If Googlebot spends its fetches on facet combinations and internal search results, the pages you actually wanted to have in the retrievable pool are being discovered slowly or not at all.
2. Signal dilution is real but overstated. Pages that duplicate one another dilute relevance judgements on a cluster of topics. On a large site this is a genuine problem; on a small site it is a non-event.
3. Extraction favours clean, self-contained sections. A page that opens with a definition, a specification or a price table is easier to quote than one that requires eight paragraphs of context.

The third point is genuinely useful and rarely stated correctly. It is not about writing for machines. It is about the well-established practice of putting the answer near the top, which is good practice for humans, for accessibility, and for extraction, for the same reason.

The three responses that do not work​

Most sites responded to AI Overviews in one of three ways. Two of them are mistakes.

FAQs on every page. Adding a schema-marked question-and-answer block to every product page does not make a page citable and does not make it rank. It produces duplicate, low-value sections, bloats the page, and in the cases where the FAQ is written by a machine it actively damages the page's quality assessment. Schema is a label on content that already exists, not a mechanism for creating it.

Rewriting everything into listicles. The instinct to match the Overview's format is understandable and wrong. The Overview's list structure is generated from sources; it is an output shape, not a quality signal. Rewriting a technical reference page as six bullet points produces a page that reads worse, quotes worse and ranks no better.

Publishing large volumes of AI-generated content. On a large site the failure mode is not that Google punishes machine text directly, because it does not reliably. The failure is combinatorial: the content is substitutable, so it competes with itself, and a large volume of near-identical pages across a cluster of topics is exactly the pattern that gets a cluster demoted as a whole.

The one response that works is unglamorous: fewer, better pages, each with something a summariser cannot invent. Original data, original photography, specific pricing, operational detail, first-hand experience, a document only you have. That was already the correct strategy before AI Overviews and it remains so.

What the traffic model looks like​

For a large site, the useful reframe is to sort queries by what an Overview does to them, then work out what proportion of your sessions live in each bucket.

The row that should worry a large commercial site is "commercial, non-branded". That is where a synthesised answer that says "the leading suppliers are X, Y and Z" removes the click entirely, because the answer is the shopping list. The defence is not to write better listings. It is to make sure the brand is one of the names that appears.

What to do about measurement​

The measurement question is genuinely awkward and worth handling honestly rather than pretending it is solved.

• Segment by query intent, not by traffic source. An AI Referral source appears in analytics; the queries it corresponds to are what matter.
• Watch indexed page count against indexed query count. If page count is rising and non-branded impressions are flat, you are adding pages that are not being selected.
• Do not chase AIO "rankings" as a discrete metric. Most teams cannot reliably identify them, and tools that claim to are extrapolating from third-party data.
• Watch the branded-query trend. Rising brand search with flat non-branded traffic is the signature of a site being mentioned rather than visited. It is not necessarily bad, but it is a different business and should be planned for.

And the metric that has not changed: the number of pages that are reachable, read and judged good. A site with 900,000 URLs and 12% coverage still has a structural problem, and AI Overviews do not fix it. They make it harder to ignore.

Where to start​

Separate the questions. Ask "which of my query categories have been affected" separately from "is my indexable surface efficient", and answer them with different data. Almost every badly-reasoned AI conversation collapses because those two questions got merged into one and answered with one chart.

PruneSEO handles the second question. The indexation waste detection page sets out how patterns are grouped and classified, and crawl budgets and allowances covers the part where waste directly consumes the fetches that retrieval systems need. For how to keep a report honest while the ground shifts under it, read health scoring and reporting alongside sampling confidence and extrapolation.

Why keyword gaps mislead​

A keyword gap report compares the queries you rank for against the queries a competitor ranks for. The output is a list of terms where they appear on page one and you appear on page thirty. Acting on it means writing content, and that is often the right advice and occasionally the wrong one.

The problem is that the gap report cannot tell you why they are on page one. There are at least five quite different reasons, and they call for five different responses:

Notice that four of those five have nothing to do with content volume. A team that reads a gap report as "we need 300 more articles" will spend a year producing pages on a site where 70% of its existing surface is waste, and the new pages will be discovered slowly, indexed slowly and cannibalised into the same handful of queries.

The structural comparison​

The comparison worth running takes four measurements per competitor. None require a crawl budget you do not have, because you only need to crawl enough to estimate surface, not to enumerate it precisely.

1. Addressable surface. How many URLs the site can generate. Estimated by sampling URL patterns rather than by exhaustive crawling.
2. Index coverage. How many of those URLs Google keeps. Available from the competitor's own public signals — sitemap sizes, index estimates, or your own Search Console data if you have worked with the property.
3. Crawl efficiency. Of what Google fetched, what share produced a live result. Roughly, indexed over fetched.
4. Routing decisions. The structural choices that explain the gap: URL length, parameter handling, pagination style, facet exposure, redirect chains, canonical policy.

Here is the shape of a useful comparison for a mid-size ecommerce site:

Read the middle column first. Competitor A ranks for everything you rank for, has a much smaller surface than you, and keeps 82% of it. That is a site that made hard decisions. Competitor B has fourteen times your surface and keeps 13% of it. That site is spending enormous resources generating and discarding URLs, and its rankings are propped up by brand and domain authority rather than by index hygiene.

The lesson is not "be like A". It is that A achieved its position with one seventh of your surface, which means your content plan is competing against a structural inefficiency of your own making.

How to estimate a competitor's surface​

You cannot crawl a competitor's full site, and you should not try. What you can do is sample patterns and scale up, which is the same method you should be using on your own site anyway.

The practical procedure:

1. Seed with the sitemap index. Read the root sitemap, note the child sitemaps, and record the file count and any lastmod dates. Sitemaps are a statement of intent, and the gap between stated and crawled is itself the finding.
2. Harvest URL shapes from the SERP and internal links. Take the top URLs for the site's main commercial queries and walk the internal link graph two levels. You are not trying to enumerate, you are trying to identify pattern families.
3. Sample each pattern. Take 30 to 50 URLs per pattern from the harvested set. Check status codes, canonical tags, robots meta, and whether the content differs between instances.
4. Scale by visibility, not by hope. For each pattern, estimate how many instances exist using available signals — sitemap file counts, filtered-navigation counts on collection pages, or your own site's known multipliers applied to comparable catalogue sizes.

The confidence interval matters more than the headline number. If you have crawled 40 URLs from a pattern and 38 returned 200 with unique content, you can say with reasonable confidence that the pattern is healthy. You cannot say how many instances exist, and you should not pretend otherwise.

A competitor surface estimate with a stated sample size is a useful artefact. The same estimate without one is a guess with a decimal point in it.

What you can legitimately learn from a competitor's robots.txt​

The robots.txt file is public, and it is often the most honest document a site publishes about its own indexation strategy. Fetching and reading a competitor's robots.txt is routine competitive analysis and tells you a great deal.

# Competitor robots.txt, abridged
User-agent: *
Disallow: /search/
Disallow: /*?sort=
Disallow: /*?view=
Disallow: /cart
Disallow: /account/
Disallow: /tag/
Allow: /blog/

User-agent: *
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*page=

What that tells you, pattern by pattern:

The interesting comparison is not "they block search, we do not". It is that they have made a decision about search, and you have made an accidental one. Most sites have never chosen what happens to internal search results; they simply never got round to it, and internal search URLs quietly become the largest crawl sink on the property.

A second-order observation is worth making: blocking a pattern in robots.txt prevents crawling but does not remove the URL from the index if it was already indexed. If a competitor blocks /tag/ they may still have thousands of thin tag pages sitting in the index. Robots and indexation are separate levers, and conflating them is the single most common misreading of a robots.txt file.

The comparison that is worth reporting​

If you do run a competitor analysis, the report should contain five things and not a sixth:

1. Surface and coverage per competitor, with sample sizes printed next to every estimate.
2. Verdict distribution by pattern, not by URL, so the reader can see which families are being wasted.
3. Routing decisions — canonical policy, facet handling, pagination, parameter policy.
4. Server and crawl hygiene signals — response times on uncached facet URLs, redirect depth, whether the sitemap matches the intended surface.
5. A ranked list of structural differences, each with an estimated size in URLs and an estimated effort.

What it should not contain is a traffic comparison. Competitor traffic estimates from third-party tools have error bars wide enough to drive any conclusion you want. Using them to decide a strategy is a category error, and it is the most common way competitor analysis turns into a budget justification exercise rather than an engineering one.

Turn the comparison into a backlog​

The output you want is a prioritised list where each item has a URL count attached. Something like:

That is a backlog a developer can work through. The ordering is not by URL count alone — a 190,000-URL pattern behind a single robots.txt line beats a 240,000-URL pattern requiring a template rewrite and a search-quality argument.

Where to start​

Run one comparison before you commission five. Pick your three closest competitors by query overlap, not by size, and measure surface, coverage and routing decisions for each. The finding that surprises people most often is not a keyword gap; it is that the competitor outranking them has a fraction of the surface and keeps almost all of it.

PruneSEO does the surface and clustering half of this. The User Guide covers connecting your property and running a first crawl, and URL pattern clustering explains how several hundred thousand URLs collapse into a handful of decisions. For the sampling discipline behind every number in the comparison, read sampling confidence and extrapolation before you present the results, because a competitor comparison without stated intervals is just a rumour with a table around it.

What each side can actually change​

Most comparisons flatten the two options into a list of tasks and then price the list. That approach misses the only thing that matters, which is that a large site's indexation problems are overwhelmingly execution problems rather than strategy problems. Almost nobody is short of ideas about which URLs are worthless. They are short of a mechanism that can remove sixty thousand of them safely and prove afterwards that nothing valuable was removed with them.

Read that as a whole rather than as a scoreboard. There is no row where agency is unambiguously better, and no row where in-house is unambiguously better. What the table describes is a division of labour, and most sites that get this wrong have hired an agency for the left column and expected results in the right column.

Where in-house genuinely wins​

The strongest argument for building the team in-house is latency. On a large site, most indexation wins come from a change to a template, a routing rule or a header, and each of those has to travel through a ticket. An in-house team with direct access to the repository makes that change the same week. An agency making the same change might make it in six weeks, and in those six weeks the crawl keeps hitting the same four hundred thousand unusable URLs.

That latency compounds in a specific way. Template fixes are high-leverage because one change affects every URL in a pattern. A pattern with eighty thousand URLs is not fixed by eighty thousand edits, it is fixed by one. If your path from decision to deployment is a month, you will only ever make a handful of those decisions a year, and you will spend them on the patterns that are easiest to explain rather than the ones that are largest.

The second genuine advantage is access to the people who know why the URL exists. Whether /search?colour=red&size=large should ever be indexable is a question for the merchandising team, and nobody outside the business can answer it from the outside. Agencies handle this badly when they lack a client-side champion, and they handle it well when they have one.

Where an agency genuinely wins​

The argument for agency is not expertise in the abstract. It is measurement discipline under load.

An experienced practitioner knows that "71% of your URLs are not indexed" is not a measurement, because it divides a surface you have not counted by an index count that Google reports as a sample. They know that the URL Inspection tool has a daily quota per property and that blowing through it produces a queue of failures rather than data. They know that a verdict drawn from six inspected URLs is an anecdote wearing a lab coat. None of that is difficult to learn and all of it is routinely forgotten by teams who are doing nine other things.

The second advantage is pattern recognition across sites. Indexation failures cluster into perhaps thirty archetypes: faceted navigation, expired campaign URLs, paginated archives, paginated product variants, internal search results, staged content leaking to production, locale fallbacks. A practitioner who has seen each of them four times recognises one from the URL shape before fetching a single page. That is not a research advantage, it is a recognition advantage, and recognition is bought much more cheaply by hiring than by developing.

The third is objectivity. A team that owns the site also owns the roadmap, the budget and the political position of everyone who created the URLs in question. That is a genuinely difficult position from which to say "delete sixty thousand pages somebody spent money building".

The two failure modes​

Both options fail in characteristic ways, and the failure modes are more predictable than the outcomes.

The one that costs the most money is the second. A well-researched agency report that does not survive a spot check destroys more credibility than no report, because the next three recommendations will be discounted too. Every finding should carry the URL pattern it came from, the sample size behind it, and the Search Console status that supports it.

Where the money actually leaks​

The cost of not fixing an indexation problem is rarely the server load, which is what most people estimate. The expensive parts are elsewhere.

Consider a retailer with 900,000 addressable URLs and 140,000 indexed. Assume the indexed set is broadly the right 140,000, and the remaining 760,000 are waste. What does that waste actually cost?

The dev-capacity line is the one that matters most and gets measured least. A site that has spent nine years generating URLs pays for them forever: every migration needs a redirect strategy for them, every crawl-log analysis is noisier, every incident review asks what was on that URL. Pruning is not primarily a search tactic. It is a maintenance strategy, and the return compounds.

The overlap that is actually worth paying for​

The useful answer is usually neither/or. The split that works:

1. In-house owns the surface inventory and the platform. Templates, routing, sitemaps, robots, headers and the change pipeline. These are engineering responsibilities and no one should rent them.
2. Agency or tool owns the measurement. Pattern clustering, sampling, verdict classification and the report. This is a measurement discipline and it is cheap to buy.
3. Joint owns the prioritisation. Which patterns to fix this quarter, ranked against dev capacity and revenue exposure.

If you take one practical step from this article, take this: the buying decision is really a question about where the bottleneck sits. If nobody internally can turn a finding into a merged pull request, buy execution. If the pipeline is fine and the diagnosis is weak, buy diagnosis. Most sites are surprised by the answer, which is itself worth knowing.

What a good handover looks like​

Whichever side you choose, insist on these four artefacts before the engagement ends:

• A runbook. The repeat procedure: how to crawl, how to sample, how to classify verdicts, how to generate the remediation files. Without it the knowledge leaves with the team.
• A pattern inventory. Every URL pattern on the site, with size, verdict and recommended action. This is the durable asset. Individual URLs are disposable.
• A prioritised backlog. Not "these 60,000 URLs are bad" but "these four patterns, in this order, because of these reasons".
• A regression check. A scheduled re-measurement so the next quarter knows whether the last one held.

The pattern inventory is the one teams underestimate. It is a spreadsheet today, but it is the artefact that lets a completely different set of people run the audit six months later and get comparable numbers.

Where to start​

Answer one question before you talk to any supplier: can your team turn a template-level decision into a deployed change this month? If yes, buy measurement. If no, buy the whole thing and plan to internalise it.

When you are ready to buy measurement rather than take it on trust, the User Guide covers connecting a read-only Search Console property and running a first crawl. The agency workspaces and permissions page covers running this across multiple client properties without cross-contaminating data, which is the part that determines whether an agency model is workable at all. For pricing and how many properties a plan covers, see plans and upgrading.

The conversation you will have in week two​

Most indexation engagements hit a predictable wall. The client asked for an SEO audit, expected a list of things to improve, and receives a statement that a large fraction of the site's addressable surface does not need to exist. The reaction is usually one of two, and both are reasonable.

The first is the most common and the most worth taking seriously. If the pattern the client wants to add content to is at 12 per cent indexed, then the content budget is being spent inside a family Google has already declined at scale. That is not an argument against writing content. It is an argument about the order.

Managing the developer, who is usually right about intent​

There is a structural asymmetry in most engagements. The developer who built the tag system in 2019 knows exactly why it exists and is usually correct about the reason. They are frequently wrong about the scale of the consequence, because they have never seen the total URL count across all nine years of tag combinations.

That combination — accurate on intent, uninformed on scale — is workable, and the way to work it is to move the conversation from whether to why.

Ask what the pattern was protecting. A faceted navigation exists because someone needed filters to work. A tag archive exists because an editor wanted visibility. A paginated listing exists because the CMS template was copied from a blog. Each has a legitimate origin, and each may no longer need one.

Bring the count, not the argument. "This pattern is 180,000 URLs and Google has read about 40 per cent of the ones we sampled" lands very differently from "your filters are bad". The first is a measurement the developer can use in a planning meeting; the second is an insult that has to be resisted.

Never frame removal as deletion. The distinction between retiring a URL and consolidating it into another is the difference between an engineering task and a business decision. Hand over redirects, retire the genuinely dead, and leave the editorial call with the person who owns the content.

The developers who fight an indexation audit hardest are usually the ones who were never shown the counts. Show them the counts and a surprising number of them become the strongest supporters of the change.

The stakeholder map​

Large organisations have more than one person who thinks they approved this, and finding out who they are late is expensive. Map four roles in the first fortnight and record them in the working agreement:

Most failures are a communication failure in one of the last two rows. The implementer received a report instead of files, or the day-to-day owner received a finding without a decision attached.

The number they will ask for​

Someone will ask for a single figure in the first month. It is a reasonable request and it is nearly always the wrong deliverable, because the honest answer is a set of numbers with different denominators and different confidence behind them.

What to do is give them three, in this order:

1. The surface. How many URLs the site generates, by pattern. This one is a census and is the easiest to defend.
2. What was assessed. How many patterns, how many URLs inspected, over what window. This is the credibility figure and it is the one to put on the slide.
3. The movement. Change in sampled indexed share on the same patterns, with the interval attached.

If they still want one number, give them the middle one. "We have measured 38 per cent of your site and the measurement is defensible" is a sentence an executive can use, and it is more persuasive than a percentage they cannot interrogate.

The month it does not move​

Every engagement has one. The campaign shipped, the files went in, four weeks later the sampled share is flat, and the client asks the question you were dreading.

The answer is a checklist, applied out loud in front of the client rather than privately, because the process is as reassuring as the conclusion:

• Was this measured with the same sample size and method as the baseline?
• Has enough time passed — four to six weeks for a large campaign — for crawl and index data to reflect it?
• Did anything else change on the site in the same period?
• Did the change actually reach production, or is something caching it?
• Was the campaign ever big enough to move this figure at all?

That last question is the one to ask out loud. A campaign touching 0.3 per cent of a surface cannot move a percentage measured on that surface, and saying so plainly converts a failure conversation into a scoping conversation. Nothing damages credibility faster than reporting a number that was never going to move.

Scope creep, and the audit that becomes a programme​

The predictable expansion is this: an audit is commissioned, a quarter of the waste is fixed, and then somebody asks for the same analysis again on the same site to confirm progress. That is a reasonable instinct and an expensive way to satisfy it, because a full re-crawl and re-inspection on a large site competes for the same 2,000 inspections a day that the remediation work needs.

Structure it so that progress measurement is cheap and separate:

Agreeing that at kick-off removes the argument later. The pattern-level re-measurement is cheap and answers the question the client actually has, which is whether the thing we shipped worked.

What to write down before the work starts​

A short working agreement, agreed in the first meeting, prevents most of the friction in this post. It does not need to be a contract; it needs to be a page both sides have read.

SCOPE
  Properties in scope: [list]        Each has its own 2,000/day inspection allowance
  Patterns to be measured: [n]       Remainder reported as not yet measured
  Inspection method: sampled per pattern, 95% confidence, plus/minus 10 points

DELIVERABLE
  One-page summary, findings by pattern, generated files, open questions
  Findings do not change with plan or scope; the deliverable is not reduced

OUT OF SCOPE
  Applying changes to the website. We produce files; your team ships them.
  Guaranteeing any indexation outcome. Google decides what it reads.
  Ranking or traffic targets, which belong to a different programme

COMMITMENTS
  Client: named implementer, release window, access for verification
  Us:  working days to turn around re-measurements; fortnightly updates

CADENCE
  Weekly    crawl and delivery status
  Monthly   report and review call
  Quarterly trend on the same patterns
  Ad hoc    after each campaign, on the date agreed before it ships

The out-of-scope block is the part that earns its place. Almost every difficult conversation in an indexation engagement is somebody asking for something that was never in scope, and the answer is much easier when the answer was agreed before anybody wanted it.

Where to start​

Ask your current client which percentage of their site you have actually measured. If nobody on your side can answer without looking it up, that is the conversation to have first, and it is a better one to have in week three than in month nine. The User Guide covers what a defensible audit contains, and health and reporting shows how the findings and their samples are structured for exactly this purpose.

The shape of the work​

An indexation engagement has four phases and they have very different economics:

The mistake most agencies make is billing the first four phases as one monthly figure. It looks tidy on the proposal and it destroys the relationship in month three, when the client asks why they are paying the same for a phase that is genuinely finished as they were for the phase that is genuinely running.

How the standard models fit​

The retainer is the default because it is easy to sell and easy to invoice. It works for the monitoring tail and it works badly for discovery. Charging a flat monthly fee for the measurement phase means the month where you spend 200 inspections a day on three patterns and the month where you spend 200 a day on thirty produce identical revenue.

The fixed project fee is the right shape for discovery and decision, provided the sample is scoped in the contract rather than discovered during it. This matters more than it sounds: on a large site, 97 inspected URLs per pattern at 95 per cent confidence and plus or minus ten points is a defined quantity. "We will audit up to 40 patterns" is not, because patterns are discovered during the crawl and the tail is unbounded.

Hourly is honest and is generally the wrong answer commercially. It prices the client's volume problem as your risk, and on a 400,000 URL site that conversation gets difficult in the second month.

Performance-based sounds appealing and is close to unusable here. The outcome you would be paid on — indexation of a set of URLs — depends on Google's decisions, which is exactly the sort of thing that made indexation work worth pricing separately in the first place. The one defensible variant ties payment to delivery: files produced, patterns classified, campaign targets specified. That is within your control and a client will accept it.

The real cost drivers​

Knowing where the hours go is what makes a price defensible rather than aspirational.

Inspection quota is a hard constraint and it is per property. Two thousand URLs per day, per property, on a Pacific-time reset. A full pass over a large site therefore takes weeks, not days, and ten client properties consume ten independent allowances rather than one shared pool. An agency that plans a month of audits across a book of business is planning against ten separate dailies, and the second client in a week is the one that slips.

Crawl runs are a lifecycle cost. A crawl takes time and cannot be undone, so a sensible commercial model treats them as a finite resource rather than a monthly subscription. PruneSEO's own allowance structure makes that explicit: crawl allowances are lifetime totals for the workspace rather than monthly figures, with one crawl on the free tier, 25 on the next and 200 on the agency tier. The reasoning is the argument — a crawler cannot undo what it already found, so a crawl earns its cost once you have acted on the previous one, and charging monthly for rediscovery is charging for a service the client cannot use.

Senior judgement is the scarce input. The mechanical parts — the crawl, the clustering, the artifact generation — should not consume senior hours. Where they do, the price is hiding an inefficiency rather than covering a cost.

Rework is usually a client delay, and it is usually unpriced. The audit finishes, the files are delivered, and the campaign sits for six weeks because the client is waiting on a release train. That gap is where indexation engagements quietly lose money, because the retainer keeps billing and the work is not progressing.

Price discovery and remediation separately​

The cleanest structure for most of these engagements is two fees and one retainer.

• A discovery and measurement fee, fixed, with the sample scope written in: number of properties, number of patterns, number of inspected URLs per pattern, and an explicit statement of what will be reported as not yet measured rather than estimated.
• A remediation delivery fee, per campaign or per band of targets, covering the classification, the destination decisions and the generated artifacts. Not the client's implementation.
• A small monitoring retainer, starting at a fixed date after the last campaign ships and running for a defined number of months.

That structure removes the two worst conversations. It cannot charge for a phase that is complete, and it cannot be blamed for a delay that happened after handover.

Something worth building into it: an implementation support line item, charged by the hour, for the period after handover. Not a retainer — a service the client can use when they are stuck on a specific file. It is cheap to provide, it converts the most common post-handover friction into paid work rather than into goodwill, and goodwill is where agency margin disappears.

What can legitimately be tied to performance​

Some outcomes are within reach and some are not. Being clear about the difference is worth more than a creative attempt to tie fees to something.

The third row from the bottom is the one clients ask for most and it is a trap. A pattern's indexed share is a function of Google's decisions on pages that also depend on the client's content and platform. Tying fees to it means carrying variance you did not create, and the variance is not symmetric: you get paid for noise on the way up and sued for it on the way down.

The clause that matters most​

Whatever the pricing model, one clause should be in the contract in plain language:

Remediation is complete when the specified files have been generated and handed over. Indexation outcomes are outside the scope of this engagement and are not a condition of payment.

That sentence does two things. It removes the expectation that the tool or the agency will change the website, which is true — no tool that produces files should be understood to apply them. And it protects the person who is paying you, who will otherwise be asked by their own leadership why the numbers have not moved yet, with no contractual ground to stand on.

Worth pairing with an implementation clause that names what you need from the client, with dates: access for the developer, a release window, and a named owner. Most stalled engagements are not disputes about scope, they are schedules nobody agreed.

Where to start​

Take the last indexation engagement you delivered and split the invoice into the four phases. Roughly half of the agencies that do this discover that discovery and decision were billed at monitoring rates. Rebuilding the proposal around that split is a day of work and it is the single highest-return change to pricing on this kind of engagement.

For what the delivery actually consists of, and what has to be in the handover for it to be accepted, campaigns and artifacts in the User Guide is the relevant section, and agency workspaces and permissions covers how the delivery and review split is handled across a team.

Name the jobs first​

Before looking at any vendor, write down the questions your team needs answered and how often. Six jobs cover almost all of the work on a large site:

Write these down before you look at a pricing page. It converts a shopping exercise into a specification, and it is the only reliable defence against buying a fourth tool that does a job two others already do.

The layers, and where they genuinely differ​

Discovery. A crawler. This is the layer most likely to be bought twice, because site audit tools bundle a small crawler that will cheerfully report that a 400,000 URL site has 2,000 pages on it. The test is simple: does it inventory by pattern, and does it say how much of the site it did not reach? On a large site the pattern grouping is the product, not a side feature — measured on three real sites, clustering collapsed 162,000 URLs into two to five clusters.

Measurement. Search Console inspection. Not optional and not replaceable, because it is the only source of a per-URL verdict from Google. It is also rationed: 2,000 inspections per property per day, 600 per minute, resetting on a Pacific-time day. Any stack whose design assumes unlimited inspection is a stack that will be wrong in month two.

Decision. This is the layer that most stacks are missing entirely. Tools that produce lists of findings do not produce judgements about them. The distinction matters because the work stops at the decision: a finding with no remedy attached is a complaint, and a complaint is not something a developer can action. Classification into ghost pages, rejected quality, suspected duplicates, other findings and not yet measured is what converts data into a plan.

Delivery. Artifacts. Redirect maps, robots fragments, noindex directive sets, canonical manifests, sitemap micro-sitemaps, internal-link plan CSVs. The requirement is that they are files the target system can read, because a report is read once and a file is applied.

Observation. Log analysis, or genuine crawl monitoring. This is the layer that distinguishes a stack built for a large site from one built for a brochure. A synthetic crawl tells you what a crawler could fetch; the logs tell you what Google actually fetched, how often, with what status, and at what cost in response time.

Ranking. Configured properly, with geography, device and volume bands stated. Covered separately, and the source of most disagreements between tools.

Where the overlap really is​

Two overlaps are worth naming because they are where budget goes.

Crawler versus audit tool. If your audit tool's crawl is capped in the low thousands, it is not a crawler for your site, it is a sample. Keep it for page-level checks if you like, and run a real crawler separately for inventory. Running three site audit tools because each finds something the others missed is expensive and produces three slightly different URL counts, which undermines all three.

Rank tracker versus indexation monitoring. They measure different systems on different cadences and cannot substitute for each other. What looks like redundancy in a stack audit is usually two tools being asked to answer the same question because nobody built the bridge between them.

Logs versus crawlers​

The cheapest upgrade available to most stacks is proper log analysis, and the reason is a structural one: a crawler tells you about the URLs it chose to fetch, and the interesting question is about the URLs it did not.

On a site where crawl budget is genuinely contested, the log file settles arguments that a crawl cannot. It is also the only place you will find the URL that is generated, linked from nowhere, and fetched twice a day.

Cost: what to buy, what to build, what to drop​

The build column is short on purpose. Building a crawler is a career; building a reporting layer over data you do not trust is a distraction. The one thing worth building is the thin integration that joins verdicts to patterns to artifacts, because that is the piece every vendor leaves to the customer.

Record the stack​

Write down what the stack is and why, once a year and whenever something changes. The artefact is boring and it prevents the two most expensive mistakes in this category: renewing a tool nobody uses, and losing the reason a decision was made three years ago.

layer,tool,purpose,renewal,owner,replacement_threshold
discovery,site crawler,URL inventory and pattern grouping,annual,technical lead,"cap below 50k URLs"
measurement,URL inspection tool,per-URL verdicts from Search Console,monthly,seo lead,"sample fidelity"
decision,indexation audit tool,verdict classification and remediation planning,annual,seo lead,"cannot distinguish inferred from observed"
delivery,artifact generator,files for web server and CMS teams,annual,technical lead,"PDF-only output"
observation,log analyser,actual crawler behaviour,annual,engineering,"sampled rather than full logs"
tracking,rank tracker,query-level positions,monthly,seo lead,"geography not configurable"

Eight questions before you renew​

1. What does it tell me that nothing else in the stack tells me?
2. Can I state the denominator behind its headline number?
3. Does it distinguish an observation from an inference?
4. Does it have a state for "not measured", or does it render absence as a zero?
5. What is the ceiling on volume, and what happens when I hit it?
6. Can it export the raw data, or is the dashboard the product?
7. Does anything in the stack depend on this tool's proprietary format?
8. If this were removed tomorrow, which number would I stop being able to produce?

Question seven is the one that determines whether you can ever leave. If the artefacts, the verdicts and the pattern map all live inside one vendor's format with no export, the tool is not a tool, it is a dependency, and it will be priced like one at renewal.

Where to start​

Write the six jobs down, map every current subscription to one of them, and cancel anything that maps to two. That exercise usually finds at least one redundant line item and at least one job nobody is doing at all, and both are useful. To see what a properly specified measurement layer looks like before comparing it to anything, the User Guide is the starting point and running and monitoring crawls covers the discovery layer in detail.

Four causes of apparent volatility​

Almost everything that looks like volatility is one of these, sometimes in combination.

The second row is the one that gets misdiagnosed most often, because a measurement change produces exactly the same picture as a real change and is usually caused by something innocuous: a tracker reconfiguration, a vendor migration, a device added to the set, a city added to a location list. The chart moves on the day of the change and never settles.

Before attributing a movement to Google, list everything that was deployed, reconfigured or re-scoped in the fortnight before it. On a large site that list is usually not empty.

The statistics of whether something moved​

Indexation work is measured as a proportion of a sampled pattern, which makes the right test a simple one: two proportions, two samples, same pattern.

The worked example below uses two samples of 97 URLs from the same pattern, five weeks apart.

The change looks like seven points of improvement. Whether it is real depends on the standard error of the difference:

standard error = sqrt( (0.608 x 0.392 / 97) + (0.680 x 0.320 / 97) )
               = sqrt( 0.002457 + 0.002244 )
               = 0.0685

observed difference = 0.072
ratio                = 0.072 / 0.0685 = 1.05

A ratio near one means the movement is entirely consistent with sampling noise. The correct report is "no measurable change at this sample size", and it is a genuinely useful sentence — it tells you to keep sampling rather than to claim a win.

The same arithmetic on a large change behaves differently. If a pattern moves from 12 per cent indexed to 40 per cent across two samples of 97, the standard error of the difference is about 0.060 and the difference is 0.28, a ratio of 4.7. That is not noise, and it is worth reporting as movement.

Two practical consequences follow, and both are routinely ignored:

• Sample size sets your detection floor. You will not see a five-point improvement on a pattern sampled at 97 URLs, no matter how long you wait. If small movements matter, the only answer is a larger sample, which means more of your 2,000 daily inspections going to a pattern that may not deserve them.
• One comparison is not a trend. A single interval either significant or not is weak evidence. Three consecutive intervals in the same direction is a claim you can defend in a meeting.

Index churn looks like volatility but is not about positions​

On content-heavy and ecommerce sites there is a third pattern worth naming separately: pages entering and leaving the index for reasons that have nothing to do with ranking. A product goes out of stock and the page is deindexed. A seasonal range lands and lifts a cluster. A publisher's archive ages out. Coverage figures move, positions fluctuate as the page mix shifts, and the whole chart looks volatile while the pages that matter are perfectly stable.

It shows up as a correlation between coverage and position movement, and it is worth testing explicitly because the two causes have opposite responses. Genuine ranking decline responds to content and competitive change. Churn responds to inventory and editorial behaviour. A team that misreads churn as a ranking problem will spend a quarter on content work when the cause was a catalogue feed.

Control groups: the cheapest way to tell change from drift​

The most reliable way to attribute a movement is to keep something that should not move, and compare.

Patterns that make good controls share a template, a platform and a technology stack with the patterns you changed, but were not themselves changed. If the product detail pattern was pruned and the comparable brand listing pattern was not, the brand pattern is a control. Its movement over the same window is a direct estimate of the background noise on that surface.

pattern,role,before_share,after_share,inspected
product-detail,changed,0.608,0.680,97
brand-listing,control,0.591,0.598,97
category-tree,control,0.932,0.941,97
legacy-tags,changed,0.124,0.402,97

Read that as two stories. The changed patterns moved by 7.2 and 27.8 points. The controls moved by 0.7 and 0.9. The background drift on this surface is small, which strengthens both results and, just as importantly, would have told you to distrust the first pair of numbers if the controls had moved too.

Controls are also how you detect that something site-wide changed — a platform release, a template deployment, a tracking change — because everything moves together. A control set that drifts with the treated set is not a control set, it is a shared environment, and that is worth knowing as well.

What to do during a volatile period​

Chaos is not a good time to make decisions, and indexation work is unusually susceptible to them because the changes are large. Four rules that hold up:

Freeze the sample allocation. Do not change which patterns get inspected in the middle of a comparison window. Reallocating invalidates the comparison more reliably than any site change will.

Widen intervals rather than narrowing claims. If the period is unstable, the honest response is to say the result is indicative and to sample again in four weeks, not to pick the reading you prefer.

Separate the campaign from the period. A change shipped on day 3 of a five-week window has a different measurement problem than one shipped on day 30. Note the ship date against the measurement window every time.

Look at the controls first. When a number surprises you, the control patterns are the fastest check on whether the surprise is about your site or about your observation.

Volatility that is worth acting on anyway​

Some movement should be responded to regardless of whether it is significant, because the underlying instability is the finding. A pattern that alternates between indexed and crawled but not indexed across successive samples is telling you something about the template, not about the sampling. Similarly, a pattern where verdicts come back mixed within a single sample usually has two sub-patterns hidden inside one, and the fix is to cluster more finely rather than to re-sample.

The rule is that uncertainty about which verdict applies is actionable, and uncertainty about how much applies is not. Investigate heterogeneity; measure magnitude.

Where to start​

Before attributing any movement, spend twenty minutes reconstructing what changed in your measurement setup over the preceding fortnight. Configurations, device sets, location lists, sample allocations, filter definitions. That check eliminates more false alarms than any amount of further analysis of the same data.

For the sampling discipline that makes these comparisons possible, sampling and confidence explains the interval arithmetic, and triage and pattern analysis covers how a mixed sample gets split into a judgement rather than averaged into one.

A tracker is a sampler, not an observer​

The single most important fact about a rank tracker is that it does not see the search results page a user saw. It requests a SERP, from one location, on one device, with one set of parameters, at a scheduled time, and reads a position out of the response. A real search is a different set of conditions every time, and Google varies them deliberately.

That is not a flaw in the tools. It is a property of the thing being measured, and it means the honest unit is not "the rank" but "the rank observed under stated conditions at a stated time".

Four sources of divergence account for nearly all disagreement between trackers:

Geography is the biggest single lever​

A query issued from a data centre in one country is a different query to a person in your market. This is not a subtle effect, and it is the reason two agencies can look at the same keyword set for the same client and reach opposite conclusions.

The failure mode is specific: a UK-focused client whose keyword set is being tracked from United States infrastructure will see local packs, map results and location-modulated results that the client's customers never see. Position three in that configuration is not position three in London, and a report built on it will recommend work aimed at a SERP that does not exist for the audience.

Fix the geography before you compare anything, and record it in the configuration rather than in somebody's memory. A reasonable minimum configuration:

{
  "domain": "example.com",
  "market": "gb",
  "language": "en",
  "device": ["desktop", "mobile"],
  "location_mode": "city_level",
  "cities": ["London", "Manchester", "Birmingham"],
  "personalisation": "anonymous",
  "schedule": "daily",
  "strip_features": ["answer_box", "knowledge_panel", "people_also_ask"]
}

Two of those settings need justifying. anonymous is not cosmetic: a signed-in session changes the SERP, and most tracking is anonymous by default for good reason. And stripping SERP features changes the position, which is why it must be decided and stated rather than inherited from whichever tool the account was set up in. If a competitor occupies the map pack above you on mobile, that is a local visibility problem and not a ranking problem, and a tracker that counts the map as position one will misdirect the work.

The sampling floor nobody checks​

Most keyword volume estimates are rounded to the nearest hundred and carry real uncertainty. If a query has, say, 100 searches a month and your tracker checks a fraction of searches, the number of actual observations behind a monthly average may be in single figures.

That produces a specific statistical artefact: the monthly position figure moves for reasons that have nothing to do with the site.

The practical consequence is that a low-volume keyword can look like it moved substantially in a month when the observed set simply did a good job of covering both weeks it was sampled from. If a keyword list is mostly terms under 1,000 searches a month, the aggregate movement across the set is substantially sampling noise and should be reported as such.

The correct habit is to separate the set into volume bands and report movement only for the bands where the observation count can carry it. This is unglamorous and it is the difference between a tracker as evidence and a tracker as decoration.

Personalisation, and the results nobody agreed on​

Search engines personalise, and rank trackers exist partly because that makes manual checking unreliable. The honest position is not that personalisation is negligible — it occasionally produces a four-position difference — but that it is unfalsifiable, so the standard response is to control for it rather than argue about it.

Anonymous sessions, a fixed location, a fixed device, and a consistent schedule will not eliminate variation, but they will keep it inside a band narrow enough that you can still see a real move when one happens. Trackers that offer recorded SERPs with the variation noted are more useful than ones that give you a clean number with no provenance, because they let you see when a keyword has become unstable.

That instability is itself a signal. A query whose SERP differs substantially between consecutive daily checks is usually one with a strong personalised or location-sensitive component, and it should not be part of a set you report movement on.

How to sanity check a tracker​

A tracker that has never been externally validated should be assumed wrong until tested. The test is cheap and takes an afternoon.

Pick twenty keywords across the volume bands. Search each one, manually, from your market, on a device matching the tracker's configuration, and record the observed position including or excluding features consistently with the configuration. Compare.

Expect agreement on high-volume, non-local, non-feature-dominated queries and disagreement everywhere else. The pattern of disagreement is diagnostic: systematic over-optimistic positions on one device point to a configuration problem, while random disagreement points to personalisation and to SERP features being counted differently.

Run it again after any configuration change, because configuration changes are where trackers quietly stop matching reality.

The complement that does not drift​

Rank trackers sample the results surface. They tell you nothing about the pages underneath it. Two questions that a tracker cannot answer at all:

Is the page indexed? A page that is not indexed cannot have a position, so the tracker either reports nothing or reports a stale last-known value. Both look like the same absence in a dashboard.

Why was it crawled or rejected? That requires a per-URL verdict from Search Console, which is a different request, a different quota and a different unit of analysis. Google returns 2,000 URL inspections per property per day, capped at 600 a minute on a Pacific-time reset, and nothing else in the toolkit produces a per-URL verdict.

So a sensible division of labour looks like this:

The failure mode of mixing these up is predictable and common. A rank dashboard shows 4,000 keywords tracked, 2,900 with positions, and no signal that the site just lost indexation on a third of its product pages — because a page that drops out of the index simply stops reporting, and a stop looks the same as a pause in a chart.

What rank tracking cannot tell you​

Three specific claims it does not support, all of which appear in reports every week:

• That a change in rankings caused a change in traffic. The causal arrow runs through pages and queries you would have to name, and the claim is usually made in the wrong direction.
• That a page is well optimised because its average position is high. Average position across a query set is dominated by the queries that convert nobody.
• That indexation is fine because positions are stable. Positions are a lagging surface metric about the pages Google already decided to read.

Where to start​

Check three things in your current tracker configuration: the geography, the device set, and the volume distribution of the keyword list. Any one of them can explain most of the disagreements you have had about performance. Once the tracker is trustworthy for what it does measure, the User Guide covers the layer it cannot see — connecting a property and inspecting real URLs — and search console inspection explains how the per-URL verdicts are gathered and rationed.

Three readers, three documents​

Almost every report written for a large site is really one document trying to serve three audiences with incompatible needs. Splitting them is the highest-leverage change available and costs no additional analysis, because the analysis has already been done.

The instinct is to write one document and let each reader find their section. That works until somebody forwards the whole thing to a board, or a developer reads the summary and acts on the least important line in it. Three documents, generated from the same findings, removes the ambiguity entirely.

The one-page summary​

The executive document has one job, which is to make the current state legible in under two minutes. It should contain five things and nothing else:

• The population. What was in scope, how many URLs it contains, and when the inventory was taken. Without this every other line is unanchored.
• What was assessed. How many patterns, how many URLs inspected, over what window. This is the line that determines whether the rest can be believed.
• The movement. Two or three figures against the same figures last period, on the same patterns, measured the same way.
• What was shipped. Campaigns deployed, their target counts, and the date each was applied.
• What was not measured. A list, delivered as a list.

That last item is the one teams skip and the one that protects the team later. When a number has not moved and the audience wants to know why, "this pattern was inventoried and not yet inspected" is an answer. Silence is not.

A real one-pager is short enough to paste into an email. Here is the shape:

INDEPENDXATION AUDIT — [site] — [period]

In scope            412,000 URLs inventoried, 61 patterns
Assessed            38 patterns, 3,214 URLs inspected, 12 May – 9 Jun
Not yet measured    23 patterns (38% of the surface)

Movement (same patterns, same method)
  Indexed share, product detail      61% (n=97)  ->  68% (n=97)
  Indexed share, category tree       94% (n=97)  ->  94% (n=97)
  Indexed share, legacy tags         12% (n=97)  ->  11% (n=97)
  Waste URLs in triaged patterns     149,400 est  ->  131,200 est

Shipped             3 campaigns, 41,180 targets, applied 2 Jun
Awaiting re-measure Legacy tags, due 7 Jul

Note that each percentage carries its sample size. That is not pedantry, it is what allows a reader to see that a move from 61 to 68 sits inside or outside the interval the samples support.

The delivery pack​

The practitioner document should be boring. It exists so that nothing depends on somebody remembering a conversation, and it should be reconstructable by a developer who was never in any of the meetings.

It contains:

• Campaign definitions. What kind of change each campaign applies, and the target list. Campaign kinds in PruneSEO are Prune, Noindex, Canonical, Recrawl nudge and Internal links, with 301 and 410 as the redirect directives, and a ceiling of 100,000 targets per campaign.
• Generated artifacts. An nginx map block, an Apache .htaccess file, a redirect CSV, a robots.txt fragment, a noindex directive set, a canonical manifest, a sitemap micro-sitemap or an internal-link plan CSV. Files, not screenshots of files.
• Platform constraints, stated plainly. Shopify cannot serve a 410 and its redirects are 301 only, so a Shopify "prune" hides a page from sitemaps, from search indexing and from storefront search while the URL still returns 200. Saying otherwise in a handover document is a promise you cannot keep.
• The sequence. Which campaign lands first, and what must be true before the next one is worth running.

And one thing it never contains: a rationale. Practitioners do not need the argument, they need the change. Keep the argument in the one-pager.

When a number does not move​

This is the situation that exposes weak reporting, because every instinct is to explain it away. The reliable approach is a fixed set of checks, applied in order, before any interpretation is offered.

The last row matters most and is the most often skipped. A campaign affecting 0.3 per cent of a surface cannot visibly move a percentage measured on that surface, and presenting it as a failed change is a category error rather than a disappointment. Say what the change was sized to do, say what it was not sized to do, and report the pattern-level figures where it can actually appear.

If after those checks the number genuinely has not moved, that is reportable, in the same register as a success. Pattern-level sampled share is the most honest lagging indicator available, and a flat line on a pattern you just repaired is information the client asked for.

Changing the story without changing the data​

The temptation in a difficult month is to reach for a different metric. Traffic was down, so lead with response time. Indexation was flat, so lead with crawl efficiency. This is not dishonest in itself, but it is unstable, because the metric of the month becomes whatever is most favourable, and a client who notices the change stops believing the whole report.

The rule that works: the metric set is fixed for a period, and it is changed deliberately, in writing, with the reason. If a metric is dropped because it is now tracked in the quarterly report, say so, and give the date it will next appear. If a metric is added, do not let it replace one — add it, and hold both for two periods so the reader can see whether it is actually more informative.

Consistency of the metric set is what allows a trend line to exist at all. A trend requires that the thing being trended has not changed definition between points.

The calendar​

Reporting problems are usually calendar problems wearing a reporting costume. This is the shape that tends to work:

The re-measurement interval on that last row is the one to hold. A large campaign needs roughly four to six weeks before crawl and index data reflects it, and measuring earlier produces a number that will fall, will be reported, and will mean nothing.

Where to start​

Take your most recent report and split it into three files without changing a word of it. Delete the rationale from the delivery pack, add the missing "not yet measured" line to the summary, and fix the reporting calendar before the next cycle starts. That is a day of work and it changes how the whole engagement is received. For how the underlying findings and their samples are produced, health and reporting is the relevant section of the User Guide, and campaigns and artifacts covers the files handed to whoever implements the work.

Start from the decisions, not the exports​

An indexation KPI is worth having only if some decision changes when it moves. That test eliminates most of what gets put on a dashboard. If a number moves and nobody would change what they do, it is not a KPI, it is a statistic.

On a large site there are really four decisions to support:

1. What to fix next, out of everything currently broken.
2. Whether last month's remediation worked.
3. Whether the site is getting structurally worse between audits.
4. Whether the reporting itself is trustworthy enough to be acted on.

Most metric sets cover the third, gesture at the second and ignore the first and fourth entirely. A set worth running covers all four, and it is not much larger than the one most teams already have.

The core set​

Six figures carry most of the weight, and each one has a stated denominator.

The first row is the one everybody quotes and the one that most needs the second and fourth rows next to it. A jump in indexed share is the intended outcome of a campaign, and it is also the easiest thing in the category to produce by accident, for reasons that have nothing to do with the work: a platform migration, a bot misconfiguration elsewhere, a seasonal crawl shift. This is why the sixth row exists as a separate measure rather than a continuation of the first.

The metric that is usually missing entirely​

If your set has one gap, make sure it is this one: the share of inventoried patterns that were never inspected.

Every site has a long tail of URL shapes that get inventoried and then dropped, because the sample was too small, because the verdicts came back mixed, or because the run was interrupted. That tail is neither a pass nor a fail. It is an open question, and its size is the honest measure of how much of your site any of the other numbers actually describes.

If 20 per cent of your patterns were never judged, then no indexation percentage in the report describes more than four-fifths of your site, no matter how precisely it is written.

Reporting that figure prominently is unusual and it costs almost nothing, because it is usually a fixed list. It is also the number that protects you when someone later points out the thing you did not check.

Leading, lagging and the lag that matters​

Metrics split into three groups, and the confusion between them causes most bad reporting.

Lagging outcome metrics. Organic sessions, indexed URL count, non-brand clicks. Real, but they respond slowly and they respond to everything, not only to you. On a large site, indexation changes take weeks to months to show anywhere except the pattern-level inspection data.

Leading technical metrics. Fetch error rate, crawl efficiency, depth distribution, orphan count, the share of a pattern that is indexed as measured by sampled inspection. These move within a crawl cycle, which makes them far more useful as evidence that something worked.

Reporting integrity metrics. Sample size per pattern, coverage of assessment, number of unmeasured patterns, the date the inventory was last refreshed. These describe the report rather than the site, and they are the ones that make everything else defensible.

Most teams report only the first group, which is the worst of the three options, because the group that moves slowest is the one with the most external noise. Move the leading group into the monthly report and the third group into every report, and the first group becomes a quarterly paragraph rather than a monthly headline.

Cadence matters more than the set​

A KPI measured weekly and reported monthly is fine. A KPI measured monthly and reported weekly is a chart that mostly shows sampling noise, and it does more damage than useless because people react to it.

The last row is the one teams get wrong. Measuring a pattern the afternoon you deploy a 40,000 URL prune produces a number that will fall, will be quoted, and will mean nothing. Set a fixed re-measure date instead — typically four to six weeks, because that is roughly how long a large campaign needs before the crawl and index data reflects it — and write it down before the campaign ships, not after.

Building the sheet​

Keep the definitions in the file so nobody reinterprets a column later. A small, boring KPI sheet looks like this:

metric,definition,denominator,source,frequency,caveat
indexed_share_pattern,share of sampled URLs returned as indexed,pattern URL count,URL Inspection sample,monthly,"n reported alongside; report lower bound"
waste_urls_estimate,projected non-viable URLs in triaged patterns,pattern URL count,sample projection,monthly,"range only, never a point estimate"
crawl_efficiency,2xx responses divided by total fetches,total fetches,crawl logs,weekly,"exclude monitoring and CDN paths"
assessment_coverage,patterns inspected this cycle,patterns inventoried,audit run,quarterly,"unmeasured patterns listed separately"
unmeasured_share,patterns inventoried and never inspected,patterns inventoried,audit run,quarterly,"report as a headline, not a footnote"

The caveat column is the one that stops the report being misread six months later, when the person reading it is not the person who wrote it and has no memory of the caveats. It takes two minutes per metric and prevents the most common form of accidental overclaiming.

Metrics to drop​

Some things on most dashboards should be removed rather than caveated.

Site health score as a single number. Covered at length elsewhere on this site, and in short: a single figure across a large site averages away the information it appears to summarise.

Average response time across all URLs. The mean is dominated by your slowest template and tells you nothing about the pages that matter. Use the slowest-decile figure per template, or the proportion of fetches exceeding a threshold.

Keyword rankings as an indexation KPI. Ranking movement and indexation are different systems with different cadences. Conflating them makes both harder to read. Rank tracking has its own set of accuracy limits, covered in a separate post.

Any percentage computed by dividing Google's index count by your sitemap count. That division has a real numerator and an invented denominator, and it is the single most quoted number in the category.

Where to start​

Write down the five decisions your monthly meeting actually makes, then check which of your current metrics would change one. If several do not, they can go, and the report gets shorter and more useful in the same afternoon. When you want the underlying measurement built properly, the User Guide covers connecting a property and running the first crawl, and health and reporting explains how the findings are structured, including what is deliberately labelled not yet measured.

Why impact-effort scoring fails here​

The standard tools score candidates on impact and effort, sometimes with a confidence multiplier. On an indexation backlog that produces confident nonsense, for three reasons.

Impact is unmeasured at the point of scoring. An analyst will happily score "fix the paginated archive" at impact 8 out of 10 from the URL count alone. If a sample of 97 of those URLs shows most are in fact indexed, the impact was 1 out of 10 and the team has spent three days on the wrong fix. The impact is not knowable until you have paid for the inspection, which means any framework that scores impact before sampling is scoring an estimate and labelling it a number.

Effort is dominated by one decision, not by URL count. Retiring 41,000 URLs sounds enormous and is one decision made in one template, shipped as one file. Rewriting 200 titles is small in scope and touches twenty-five templates and a content team. Effort in this category is better estimated as the number of distinct systems you have to convince, not the number of URLs affected.

Some findings are prerequisites for reading the others. You cannot sensibly prioritise "prune this pattern" until you know whether the URLs were blocked in robots.txt, because a blocked pattern may be excluded deliberately and permanently. Scoring them as peers produces work that has to be undone.

The four inputs​

Replace impact with something that is defensible before measurement, and add two things the standard frameworks omit.

Surface is the only one most teams already have, and it is the one most likely to be wrong. A pattern of 400,000 URLs at an average of eleven bytes of unique content is worth less attention than a pattern of 3,000 URLs where each one has distinct data, and surface alone cannot tell the difference.

Turning those into a score​

The ranking number does not need to be clever. It needs to be arithmetically honest, and the important move is to make the confidence term multiplicative rather than additive, because a large pattern you know nothing about should not outrank a small one you have measured.

priority = surface × confidence_share × prerequisite_gate ÷ effort_units

• surface is the count of URLs in the pattern, or better, an estimate of waste URLs rather than total URLs. For a pattern with 40,000 URLs where the sampled verdict is crawled but not indexed with a lower bound of 0.79, the waste estimate is about 31,600. That is the number the meeting cares about, not the 40,000.
• confidence_share is the proportion of the pattern actually inspected. A pattern of 250 URLs or fewer audited in full scores 1.0. A large pattern sampled at 97 out of 40,000 scores near zero, which correctly reflects that you are ranking a guess.
• prerequisite_gate is either 1 or zero. Zero when the fix depends on another landing first. This single term is what stops a team planning a prune on a pattern that is still blocked in robots.txt.
• effort_units is a count of distinct systems: one template, two templates, one platform setting plus a developer ticket, and so on. Not days. Days are a guess about people you have not met.

The result is not a universal truth. It is a number that everyone in the room can recompute, which is the actual requirement.

A worked example​

Four patterns from a mid-sized retail site, with the reasoning shown rather than just the score.

The facet pattern has the largest number in the entire audit and scores zero, because it is blocked in robots.txt and the decision about whether it should stay blocked has not been made. Until somebody decides that, any prune work on it is wasted. That is what the gate is for.

The out-of-stock pattern is the useful counter-example: 22,000 URLs, 97 sampled, and the verdict came back overwhelmingly indexed, which means those pages are doing something useful despite looking like classic waste. Scoring them high would have produced a large campaign to remove pages that were fine.

Order of operations beats order of value​

Beyond the score there is a sequence, and it is worth stating separately because it is the thing that most often gets ignored.

Doing stage five before stage two is the common mistake, and it is expensive in a specific way: you ship 150,000 URL changes based on a pattern that was never actually crawlable, discover that a third of them were deliberately excluded two years ago, and carry the rollback.

Keeping the backlog honest between audits​

A prioritisation framework that is only run at the end of an audit will be out of date within a month, which is how technical backlogs fill with findings nobody re-examined.

Three habits keep it current:

• Re-run the score when new samples land. A pattern sampled at 40 URLs and later at 200 has a different confidence term, and its rank should move accordingly. The sample is a purchase and it should change the plan.
• Keep a standing "not yet measured" list. Patterns inventoried but never inspected are open questions, and open questions do not get scores. They get a date and a name.
• Re-gate after every shipped change. Campaign five may have removed the prerequisite for campaign two. If the gate is never revisited, the ordering quietly stops being true.

Where the numbers come from​

The confidence term is the one worth being precise about, because it is where the honest claim and the convenient one differ. Sampling in PruneSEO targets 95 per cent confidence at plus or minus 10 percentage points, which needs 97 URLs per pattern; at plus or minus 15 points it needs 43. Below those numbers the correct phrasing is that the result is indicative. Forty-five indexed out of fifty inspected supports a statement of "at least 79 per cent" at the 95 per cent level, and does not support a clean 90 per cent, which is a different and stronger claim than the sample carries.

If the framework is going to be used in front of a client, that distinction is the whole credibility of it. Sampling and confidence sets out how the extrapolation works, and triage and pattern analysis covers where the verdicts behind these estimates come from.

Where to start​

Take your existing backlog and add two columns: the sample behind the impact figure, and the prerequisite. Most of the reordering will come from those two columns alone, and it costs an afternoon rather than a re-audit.

Links are the only part of the architecture you control directly​

Crawl budget is usually explained as Google's decision about how much it will fetch, which makes it sound like weather. Internal links are the part of that system you can actually edit, and they are edited more casually than any other technical element on a large site.

Most internal linking on a big site is a by-product. A footer renders the same forty links on every page because a designer built a footer. A sidebar shows "related products" from a component nobody configured. A navigation block includes a section that was added in 2019 and has since been folded into three other sections. None of these were decisions about importance. They were decisions about layout, and they have since become decisions about what Google spends its attention on.

A site with no internal linking strategy is not neutral. Its strategy is "whatever the templates happen to emit", and that is usually a weighting toward the newest and the most boilerplate-heavy.

The three things a link actually transmits​

Before changing a link, it helps to be precise about what changes when you do.

The first is the one that gets overlooked and it is the one that causes the most damage on large sites. Crawl depth is a hard structural constraint. Anything more than a few clicks from a linked entry point is effectively unreachable at any realistic crawl rate, and no amount of internal linking within the deep section will fix it, because the deep section is not being entered at all.

So the first question about any cluster of pages is not "how should they link to each other" but "what is the shortest honest path from something a crawler already visits to the top of this cluster".

Set entry points at template level​

Most crawl-depth problems are template problems, which means they are fixable once rather than link by link. The usual situation is a large section reachable only through a JavaScript-rendered menu, an XML sitemap, or a footer link that appears below forty thousand links of other footer links.

Fixing it means choosing where the entry point lives, and the choice is a genuine trade-off:

The table is deliberately blunt about the sitemap. Submitting a URL in a sitemap tells Google it exists; it does not tell the crawler it is interesting, and a URL whose only route to it is a sitemap is a page nobody has vouched for. Sitemaps are for discovery of the unknown, not for the pages you already know you want read.

Distribute links by decision, not by template convenience​

Once the entry points exist, the question is where internal authority lands. The default on a large site is that it pools in the template: the homepage links to eight categories, every category links to a hundred subcategories, and every product page inherits exactly the same footer.

The useful move is to make the distribution intentional. Three ways to do it, roughly in order of how much work they take:

• Template weighting. Decide which blocks render on which page types, and reduce the universal footer to links that genuinely apply everywhere. This is one change and it moves the whole graph.
• Contextual links in body copy. The strongest links on any site are the ones an editor placed because the paragraph needed them. They cannot be generated, and on large sites there are not nearly enough of them.
• Prioritised block insertion. Where an automated block does exist, decide what it points at. Showing the same forty URLs on every product page tells the crawler those forty are the site's most important destinations, and they are frequently the least important.

The judgement to apply: an automated link block should promote something because it is relevant to the page, not because it fills a slot. If a component cannot be configured per page type, that is the finding.

Find the pages that were never linked to​

The reverse problem is as common as the over-linked one. Orphan pages are ordinary in the output of any programmatic surface, and they are cheap to find once you have the crawl.

Take the site graph from your last crawl and count inbound internal links per URL. Then filter out the pages that are supposed to be orphans — login, legal, standalone campaign landing pages — and look at what is left. The results fall into two useful groups:

• Pages that genuinely deserve distribution and never got it. These are usually commercially valuable pages that were published and then forgotten, sitting in a category that has not been actively maintained since launch.
• Pages with nothing to distribute. If a page has no plausible inbound link from anywhere on the site, it may be a page that should not exist, or a section that never had a place in the information architecture.

The second group is the more valuable finding, because it points at a structural decision nobody has made rather than a gap somebody forgot to fill.

Turning the decisions into a work list​

The output of an internal linking review should be a file, not a set of intentions. The format is deliberately minimal, and the empty anchor column is not an unfinished feature — the tool does not know what your product is called or which of two phrasings your customers use, so it proposes a source and leaves the wording to the person who owns the page.

target,suggested_source,suggested_anchor
/guides/choosing-a-lens,/category/digital-cameras/lens-primes,
/guides/choosing-a-lens,/compare/lens-comparison-chart,
/products/lens-primes,/category/digital-cameras/lens-primes,
/products/lens-primes,/compare/lens-comparison-chart,

Four rows. The first two repair links pointing at a consolidated page. The last two are new links, and their count is the interesting part: if a page only has two plausible sources on a site this size, it is not under-linked by accident, it has never had distribution.

The rows where no plausible source exists are the ones to read first. An empty result is a finding about the information architecture, and it will not be fixed by generating more links.

What not to do​

Three common interventions look helpful and are not.

Adding a block of links to every page in a cluster to raise its prominence. If the added links point at each other rather than at anything indexed or externally linked, you have built a closed loop. Crawlers gain nothing and the pages still carry no external signal.

Rewriting anchors across the site to improve relevance. Bulk anchor rewriting is the kind of change that looks clever and reads as manipulation, particularly when applied to thousands of links at once. If you would not be comfortable seeing the before and after side by side in a review, do not ship it.

Automatically linking every brand or product mention to the newest page of that type. This is how cannibalisation gets manufactured at scale. The newest page is not usually the canonical one, and every internal link is a vote for which page you think should rank.

Measure the structure, not just the traffic​

Internal linking is one of the few technical changes whose effect can be checked without waiting months. Three measures, all available immediately after a crawl:

• Depth distribution. What share of your URLs sit within three clicks of a linked entry point, and has that moved?
• Orphan count. How many URLs have zero inbound internal links, excluding the exceptions you declared?
• Link concentration. What share of all internal links point at the top hundred URLs? A very high figure means the graph is a star and everything else is hanging off it.

These three are structural, they change within a crawl cycle, and they do not depend on rankings moving. For a post about how to read them against the rest of the health picture, Health and reporting is the relevant section of the User Guide, and crawl budgets and allowances covers how the crawl that produces this graph is bounded.

Where to start​

Run one crawl, build the inbound link counts, and list the zero rows. That list is either your distribution backlog or your architecture problem, and it is usually cheaper to produce than any argument about authority flow will be.

A brief is a set of decisions, not a document​

Look at what most briefs actually contain: an intent summary, a target query set, a competitor list, a word count, some tone notes, and a handful of internal links to include. Every one of those fields is about the page. None of them is about the neighbourhood.

That gap matters because the page is never the only artefact. On a site with a location service, a comparison tool, a category tree or a jobs board, publishing one new page changes the size and the shape of a pattern that may already contain thousands of URLs. The writer is handed a brief for item number 4,001 in a family, and has no way of knowing that items 1 through 4,000 were mostly crawled and rejected.

A brief that only describes the page is not incomplete. It is a brief that has quietly delegated the indexation decision to whatever the template happens to do.

The fix is small. Add a section to the template that describes the pattern the page belongs to: how many siblings exist, what their shared verdict is, whether this one is an addition or a replacement, and what the total count will be after publication.

Put the indexation verdict in the brief​

The single most valuable addition to a brief is one line of evidence about the pattern. Not an opinion, not a guideline. A measurement.

Suppose the location pattern has 12,400 URLs and a sampled inspection of 97 of them returns crawled but currently indexed for 71 per cent of them, with a 95 per cent Wilson lower bound of 0.61. That sentence tells an editor more than any content guideline, because it says the template is not the problem: Google is reading these pages and keeping most of them. It also tells the editor that a new page has a realistic chance of being indexed, which changes how much care the launch deserves.

Contrast it with the case where the same sample returns crawled but currently not indexed for 88 per cent of the family. Now the brief needs to say something very different: this page is a member of a pattern Google has already declined at scale, so publishing it is not a content decision, it is a decision to add one more URL to a group that needs fixing first.

The row that gets skipped is the last one. A heterogeneous sample does not average into a single editorial instruction, and a brief that tries to average produces a page written for a family that does not exist.

The template decides more than the page​

On any large site, most of what a new page inherits is decided before a word is written. The template supplies the title tag construction, the heading hierarchy, the number of links in the footer, the pagination, the breadcrumb, the schema markup and the analytics parameters. Writers adjust within that; they do not control it.

Which means an editor arguing about headline length is optimising the wrong end of the system. If 80 per cent of a pattern's problems come from the template — a self-referencing canonical on a paginated view, a filter that adds a query string to every link, a footer link injected into all four thousand pages — then the brief for page 4,001 cannot fix any of it.

The practical instruction in the brief is therefore about routing, not writing: which template, which pattern, and what the shared verdict is. Where a page does not need to exist, the strongest editorial decision available is sometimes to not commission it, and a brief is the cheapest place to make that call because it happens before production rather than after it.

What a working brief looks like​

The template below is plain markdown on purpose. It is meant to be read by an editor, dropped into a project tool, and argued over.

# Brief: [page title]

## Audience and intent
- Who is searching this and what are they trying to do
- The single job this page does that no existing page does

## Pattern
- Pattern name: [the pattern this URL belongs to]
- Current size: [n] URLs
- Sampled verdict: [n of m inspected returned X]
- Action: add to pattern / replace an existing URL / do not publish

## End state
- Once live this pattern will contain [n + 1] URLs
- Sitemap: include in [which sitemap file]
- Canonical: self-referencing
- Indexable: yes

## Links out
- Links in from: [target pages]
- Links out to: [priority pages]
- Breadcrumb trail: [path]

## Never
- Do not link to this page from the global footer
- Do not add it to the legacy navigation block

The last section is the one that does the work on a large site. Footer and navigation blocks are where orphan pages come from, and a brief that says where a page will be linked from — specifically — prevents the default outcome, which is that it gets linked from nowhere.

Volume is not a strategy​

A large publisher will eventually have a conversation where the answer to every indexation complaint is to publish more. More long-tail pages, more locations, more comparisons, more glossary entries. Sometimes that is right. Often it is a way of avoiding the measurement that would tell you whether the existing set is worth keeping.

The arithmetic is unforgiving. If a pattern is at 30 per cent indexed and you add 50 per cent more URLs to it, the indexed count rises and the indexed share falls, and the crawl budget that feeds the pattern gets divided across a larger set of pages that mostly will not be rewarded. Nobody gets a better outcome. You have simply made the problem larger and the reporting window harder to read.

Publishing more pages into a pattern Google is already declining converts a fixable problem into an intractable one.

If the brief genuinely needs volume — a programmatic surface that has earned its place, with unique data per URL and a distinct query behind each one — the brief should say so and prove it. Otherwise the honest recommendation is fewer pages, better linked.

Cannibalisation is a brief failure​

Two briefs written for the same query set is a planning failure that shows up months later as two pages competing. By then the fix is expensive, because you are no longer deciding what to write, you are deciding which page to retire.

The cheap check is a set intersection: does the planned query set for page B substantially overlap page A's? If it does, one of them needs a different job. Not a different phrasing of the same job. A different job.

That check belongs in the brief review, before production, and it is easier when briefs live in one place. It is also easier when the pattern view is available next to them, because overlap is rarely confined to a single pattern. Two teams, two templates, one query set, and nobody noticed, is the normal way this happens on a large site.

When the answer is not to write it​

The most useful brief on a large site is occasionally the one that concludes no new page should be produced. It is a legitimate output and it is almost never in anyone's reporting lines, which is part of why it is worth writing explicitly.

The justification is a measurement: this query is already served by a page that ranks; the pattern is at 94 per cent indexed and adding one more URL consumes crawl capacity; the same intent is reachable through an existing section. That is not a content failure. It is the brief doing the job it was commissioned for, one level higher than expected.

Where to start​

Take the last brief that was commissioned on your site and check whether it contains a pattern, a sampled verdict and an end state. If it does not, that is the gap, and it is a template change rather than a research project.

To see what a sampled verdict looks like before you write one, the User Guide covers connecting a property and reading the first results, and Triage and pattern analysis is where patterns, verdicts and the five result buckets are explained. For sites publishing at scale, URL pattern clustering sets out how a generated surface is grouped before anybody decides which part of it to keep.