--- type: Leaf title: Crawl health and Core Web Vitals as a standing measurement description: A launch audit catches launch day. A weekly crawl and field Core Web Vitals catch what changed since, as dated findings with a URL on each one. resource: https://nationaldigital.com.au/headless-cms/ai-visibility/crawl-health-and-core-web-vitals/ tags: - headless-cms - Technical SEO - Measurement - AI Search Visibility - Core Web Vitals - weekly site crawl - crawl health monitoring - redirect chains - canonical errors - field data CrUX - technical SEO monitoring timestamp: '2026-10-15T06:00:00.000Z' language: en-AU publisher: National Digital author: Cameron Young authorRole: CEO readingTimeMinutes: 6 wordCount: 1061 datePublished: '2026-10-15' dateModified: '2026-10-15' pillar: headless_cms canonical: https://nationaldigital.com.au/headless-cms/ai-visibility/crawl-health-and-core-web-vitals/ --- _Home — Content & Experience — AI visibility — Crawl health and Core Web Vitals as a standing measurement_ # Crawl health and Core Web Vitals as a standing measurement Redirect chains, canonicals, sitemaps and rendering all change after launch, and field Core Web Vitals slip when a script is added. A weekly crawl records each change as a dated finding. **Quick answer:** A weekly crawl diffed against the previous week records status codes, redirect chains, canonicals, sitemaps, rendering and structured data as dated findings, read beside field Core Web Vitals from Search Console, with fixes at the source of each mechanism. ## About this page - Written by Cameron Young, CEO - 6 min read - Updated 15 October 2026 _Technical SEO — Measurement — AI Search Visibility_ ## What does a weekly crawl of your own site catch that a launch audit misses? The things that change after launch: a redirect chain from a CMS edit, a canonical pointing at a staging host, a sitemap that stopped updating, a page that now needs JavaScript to show its text, and field Core Web Vitals that slipped after a new script. A weekly crawl dates each one against a URL. **Additional Context** Crawlers and assistant fetchers request the same HTML, so a loop or a challenge page fails all of them silently. _High confidence · Verified 15 Oct 2026_ *The site as served* ## What the crawl records A crawl fetches every URL the site links to and every URL the sitemap lists, and writes down what came back. It records the status code, the redirects and how many hops each took, the canonical each page declares and whether it points at itself, the indexability directives in the head and the headers, the titles, descriptions and headings, which pages carry structured data and whether it parses, the internal links in and out of each page, and the response time at the server. Run once, that is an audit. Run every week and diffed against the week before, it is a record in which every finding is a change with a date. The changes are rarely dramatic and they are always real. An editor saved a page under a new slug and the old URL now redirects through two hops. A plugin update rewrote the canonical to the staging hostname. The sitemap job stopped after a deploy and has listed the same pages for a month. A new component renders its text from script and the fetched HTML of that page is now empty where the content used to be. Google's documentation on canonicalisation and on sitemaps describes how each of these signals is read, and the crawl shows which ones the site is currently sending. ## Field data Core Web Vitals are assessed from field data, meaning the experience of real Chrome users as collected in the Chrome User Experience Report, and the Search Console report groups the site's URLs by that data. A lab score from a single test run is a diagnostic. The weekly record keeps the field status per URL group, so that a slip in Largest Contentful Paint the week a hero video was added, or a rise in Interaction to Next Paint after a chat widget shipped, is dated and attributable. The definitions and thresholds are published on web.dev, and [the weekly Search Console pull](/headless-cms/ai-visibility/search-console-weekly-record/) is where the field status is read from. ### Why this is visibility work Search crawlers and assistant fetchers request the same HTML the crawl does. A page that redirects in a loop or returns a challenge tells all of them the same thing, and a page that ships its text only after script runs fails the fetchers that do not render it. None of them reports back. The access logs show which agents were affected, which is what [auditing AI bot traffic](/headless-cms/omnichannel-content-delivery/auditing-ai-bot-traffic/) covers. The crawl shows what they were served. ## The engineering response Each finding names a URL and a mechanism, and the fix lives at the source of the mechanism: the redirect rules in the CMS, the template that generates the canonical, the sitemap job, the image pipeline, the order and loading of scripts. Fixing the instance without the source produces the same finding next week. The same discipline applies when a site moves platforms, which is why [SEO preservation in a migration](/headless-cms/wordpress-migration-to-headless/seo-preservation/) reads as a checklist of the things this crawl watches for afterwards. The weekly crawl is one of the sources in [AI Visibility Management](/services/ai-visibility-management/), where it sits beside the Search Console pull so that what Google reports can be compared with what the site served that week. *Key Takeaways* ## What a weekly crawl is for A weekly crawl diffed against the previous week turns status codes, redirects, canonicals, sitemaps, rendering and structured data into dated findings against URLs, read beside field Core Web Vitals. Each fix goes to the source of the mechanism, because the instance returns otherwise. - **An audit catches launch day and a weekly crawl catches everything since** Redirect chains, canonicals, sitemaps and rendering all change through ordinary editing and deploys, and the diff dates each change. _Critical_ - **Core Web Vitals are a field measurement and the lab is a diagnostic** The Search Console report groups URLs by real Chrome user data, and a weekly status per group makes a slip attributable to the week it happened. _Important_ - **Crawlers and assistant fetchers are served the same HTML** A loop or a challenge page fails every one of them silently, and script-only text fails the assistant fetchers that do not render JavaScript, so the crawl is part of the visibility record. _Critical_ - **Fix the mechanism behind the instance** A canonical corrected by hand returns next week if the template that generated it is untouched. _Important_ ## Questions about crawl health and Core Web Vitals ### We had a technical SEO audit at launch. Why crawl weekly? Because the audit describes the site on one day and the site changes every week through editing, plugin updates and deploys. The weekly crawl finds the redirect chain or the broken canonical the week it appears, with the date attached. ### Is a Lighthouse score the same as Core Web Vitals? No. Lighthouse is a lab test of one page load under fixed conditions and is useful for diagnosis. Core Web Vitals are assessed from field data collected from real Chrome users, which is what Search Console reports and what the weekly record keeps. ### Does the crawl affect the live site? It fetches pages at a modest rate under a named user agent and never submits anything. On a site that cannot absorb that, the rate is lowered and the crawl is scheduled off-peak. ### What usually turns up in the first crawl? Redirect chains from old slugs, canonicals pointing somewhere unexpected, sitemap entries that redirect or return errors, and components whose text is not in the fetched HTML are the common first findings. All of them are tickets with a URL. ## Want the first weekly crawl of your site, diffed against itself? - [Book a scoping call](https://calendly.com/cameron-at-national-digital/discovery) ## Related reading - [Recording what AI assistants say about your business](/headless-cms/ai-visibility/ai-assistant-answers/) - [The sources AI assistants cite when they describe you](/headless-cms/ai-visibility/sources-assistants-cite/) - [Testing whether an AI agent can use your website](/headless-cms/ai-visibility/assistant-task-tests/) - [Measuring your place in AI Overviews and search features](/headless-cms/ai-visibility/ai-overviews-and-serp-features/) - [Reading Search Console as a weekly record](/headless-cms/ai-visibility/search-console-weekly-record/) - [Keeping Business Profile, Bing and your site in agreement](/headless-cms/ai-visibility/business-profile-and-bing-consistency/) ## Related **Parent:** - [AI visibility](/okf/headless-cms/ai-visibility.md) **Related:** - [Recording what AI assistants say about your business](/okf/headless-cms/ai-visibility/ai-assistant-answers.md) - [The sources AI assistants cite when they describe you](/okf/headless-cms/ai-visibility/sources-assistants-cite.md) - [Testing whether an AI agent can use your website](/okf/headless-cms/ai-visibility/assistant-task-tests.md) - [Measuring your place in AI Overviews and search features](/okf/headless-cms/ai-visibility/ai-overviews-and-serp-features.md) - [Reading Search Console as a weekly record](/okf/headless-cms/ai-visibility/search-console-weekly-record.md) - [Keeping Business Profile, Bing and your site in agreement](/okf/headless-cms/ai-visibility/business-profile-and-bing-consistency.md) - [Turning visibility findings into engineering tickets](/okf/headless-cms/ai-visibility/findings-to-engineering-tickets.md) # Citations - [Google Search Central - Core Web Vitals and page experience](https://developers.google.com/search/docs/appearance/core-web-vitals) — Google's statement of what Core Web Vitals measure and that the metrics are assessed from field data. - [web.dev - Web Vitals](https://web.dev/articles/vitals) — Definitions of Largest Contentful Paint, Interaction to Next Paint and Cumulative Layout Shift and their thresholds. - [Google Search Central - Sitemaps overview](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) — What a sitemap tells Google about the pages on a site and when one helps. - [Google Search Central - What is URL canonicalization](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) — How Google chooses a canonical URL among duplicates and the signals that influence the choice.