--- type: Leaf title: Auditing AI bot traffic description: AI crawlers fetch server-side and never trigger analytics. Find them in your access logs, see what they are served, and set policy per agent. resource: https://nationaldigital.com.au/headless-cms/omnichannel-content-delivery/auditing-ai-bot-traffic/ tags: - headless-cms - Machine-Readable Delivery - AI Search Visibility - Measurement - AI bot traffic - GPTBot - ClaudeBot - PerplexityBot - AI crawler logs - robots.txt AI crawlers - server log analysis timestamp: '2026-09-01T00:00:00.000Z' language: en-AU publisher: National Digital author: Cameron Young authorRole: CEO readingTimeMinutes: 1 wordCount: 293 datePublished: '2026-09-01' dateModified: '2026-09-01' pillar: headless_cms canonical: https://nationaldigital.com.au/headless-cms/omnichannel-content-delivery/auditing-ai-bot-traffic/ --- # Auditing AI bot traffic AI crawlers fetch server-side and never trigger your analytics. Find them in the access logs, see what they are served, and set policy per agent. **Quick answer:** AI crawlers fetch server-side and never trigger analytics, so access logs are the only record of which assistants read a site and what they were served. ## About this page - Written by Cameron Young, CEO - 1 min read - Updated 1 September 2026 ## How do you find out whether AI crawlers are reading your website? Filter your server or CDN access logs by the published AI user agents - GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Applebot among them - then group the hits by URL and status code to see which sections they fetch and which they fail to reach. **Additional Context** Most teams have never separated this traffic from ordinary analytics, which miss it entirely: these are server-side fetches that never execute JavaScript, so no client-side analytics tag ever fires. _High confidence · Verified 1 Sept 2026_ *The Blind Spot* ## Why Your Analytics Cannot See This Client-side analytics fires when a browser executes JavaScript. An AI crawler requests the HTML and leaves, so the visit never appears in the dashboard the marketing team reviews. The only durable record is the access log kept by your server or CDN, which is why teams can be confident nothing is happening while a meaningful share of their pages is being fetched daily. This is why measurement belongs before protocol work rather than after it. Deciding whether to [restructure a site for AI search visibility](/headless-cms/omnichannel-content-delivery/machine-readable-delivery) is a different conversation once you can see which assistants fetch which sections, and which of them are hitting errors. ### What the Log Actually Tells You Three things, in order of usefulness. First, which agents appear at all - the difference between a training crawler and a live retrieval fetch matters, because the second means an assistant is answering a question about you right now. Second, which URLs they concentrate on, which is rarely the pages you would have guessed. Third, the status codes: a crawler collecting redirects, 404s or 403s from a misconfigured rule is being told your content does not exist, and that failure is completely silent from the inside. ### Separating Training From Retrieval The distinction worth drawing is between agents gathering material for model training and agents fetching a page to answer a live question. They warrant different decisions: the first is a licensing and policy question for the business, the second is a visibility question. Both are controlled from robots.txt, and a blanket block applied without that distinction is the common way a site accidentally removes itself from assistant answers while intending only to opt out of training. Getting that wrong is closer to [losing rankings in a migration](/headless-cms/wordpress-migration-to-headless/seo-preservation) than to a privacy control. ## How to audit AI bot traffic in your logs A first pass that takes an afternoon and usually changes how seriously the rest of the AI-visibility work gets taken. - Total time: Half a day for a first pass - Difficulty: intermediate - Audience: it_manager **Tools:** - CDN or origin access logs - A log query tool or spreadsheet - Your current robots.txt 1. **Get access to the right logs** _(An hour, mostly access requests)_: Use the CDN or edge logs if you have them, since they see requests your origin never does. Otherwise use origin access logs. Aim for at least a month so weekly patterns are visible. 2. **Filter by the published user agents** _(An hour)_: Search the user-agent field for the documented AI tokens rather than a generic 'bot' match, which will drown you in uptime monitors and scrapers. Keep the exact strings, because vendors publish them and they change. 3. **Group by agent, URL and status code** _(An hour)_: Three simple aggregations answer most of the question: which agents visit, what they fetch most, and what they are being served. Anything other than a 200 on a page you want quoted is a finding. 4. **Cross-check against robots.txt** _(Half an hour)_: Confirm the rules in place are the rules you intended. A directive added years ago, or a broad disallow inherited from a template, is a routine discovery at this step. 5. **Decide policy per agent, not in bulk** _(A conversation, not a task)_: Separate training crawlers from live retrieval fetches and decide each deliberately. Blanket blocks are how a site removes itself from assistant answers while meaning only to opt out of training. **Summary:** Filter access logs by published AI user agents, group by agent, URL and status code, cross-check robots.txt, then set policy per agent. *Key Takeaways* ## The Cheapest Useful Step on the List An afternoon in the access logs establishes whether the traffic justifying AI search work exists at all, and routinely surfaces silent failures that no dashboard reports. - **AI crawler traffic is invisible to client-side analytics** These are server-side fetches that never execute JavaScript, so no analytics tag fires. The access log is the only durable record of what was taken. _Critical_ - **Status codes matter as much as visit counts** An agent collecting redirects, 404s or 403s is being told your content does not exist. That failure produces no error anyone sees and can persist indefinitely. _Critical_ - **Training crawlers and retrieval fetches deserve different decisions** One is a licensing question for the business, the other a visibility question. A single blanket robots.txt rule collapses them and commonly removes a site from assistant answers by accident. _Important_ - **Measure before restructuring anything** Knowing which assistants fetch which sections changes which of the remaining protocol decisions are worth making, and in what order. _Important_ ## Auditing AI Bot Traffic: Common Questions ### Which user agents should we be filtering for? Filter for the tokens the vendors publish rather than a generic bot match: GPTBot and OAI-SearchBot alongside ChatGPT-User, ClaudeBot, PerplexityBot and Applebot among others. Vendors document these and add to them, so the list is worth re-checking rather than hard-coding once and forgetting. ### Why does none of this show up in our analytics? Client-side analytics runs when a browser executes JavaScript. AI crawlers request the HTML and leave without running scripts, so the visit never registers. Server or CDN access logs are the only place the request is recorded, which is why teams are often confident nothing is happening while their pages are fetched daily. ### Should we block AI crawlers? That depends which ones and why, and it is worth separating the two cases. Blocking agents that gather training material is a licensing and policy decision for the business. Blocking agents that fetch pages to answer a live question removes you from assistant answers. A single blanket rule collapses that distinction, which is the most common way a site opts out of visibility while intending only to opt out of training. ### What is a realistic first finding? Usually two things: the agents concentrate on sections nobody expected, and some of them are receiving redirects or errors on pages the business cares about. Neither shows up in any dashboard, and the second is generally fixable the same week it is found. ### Do we need a tool for this? Not for a first pass. Access logs, a query tool or spreadsheet, and your current robots.txt are enough to answer which agents visit, what they fetch and what they are served. Ongoing monitoring is worth automating once the first pass proves there is something to watch. ## Want someone to read the logs with you? - [Ask an engineer](https://calendly.com/cameron-at-national-digital/discovery) ## Related reading - [Web delivery best practices for Australian device and platform usage](/headless-cms/omnichannel-content-delivery/web-delivery/) - [Professional mobile apps solutions for Australian businesses](/headless-cms/omnichannel-content-delivery/mobile-apps/) - [Content adaptation best practices for Australian device and platform usage](/headless-cms/omnichannel-content-delivery/content-adaptation/) - [AI search visibility](/headless-cms/omnichannel-content-delivery/machine-readable-delivery/) - [JSON-LD for AI search](/headless-cms/omnichannel-content-delivery/json-ld-for-ai-search/) ## Related **Parent:** - [Omnichannel content delivery](/okf/headless-cms/omnichannel-content-delivery.md) **Related:** - [Web delivery best practices for Australian device and platform usage](/okf/headless-cms/omnichannel-content-delivery/web-delivery.md) - [Professional mobile apps solutions for Australian businesses](/okf/headless-cms/omnichannel-content-delivery/mobile-apps.md) - [Content adaptation best practices for Australian device and platform usage](/okf/headless-cms/omnichannel-content-delivery/content-adaptation.md) - [AI search visibility](/okf/headless-cms/omnichannel-content-delivery/machine-readable-delivery.md) - [JSON-LD for AI search](/okf/headless-cms/omnichannel-content-delivery/json-ld-for-ai-search.md) # Citations - [Google Search Central - Google crawlers and user agents](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) — Reference for Google's crawler user-agent tokens, including the Google-Extended control token.