• 1 min read

Auditing AI bot traffic

AI crawlers fetch server-side and never trigger your analytics. Find them in the access logs, see what they are served, and set policy per agent.

Quick answer: AI crawlers fetch server-side and never trigger analytics, so access logs are the only record of which assistants read a site and what they were served.

  • Machine-Readable Delivery
  • AI Search Visibility
  • Measurement
Jump to section
  1. Why Your Analytics Cannot See This
  2. Auditing AI Bot Traffic: Common Questions

Quick answer

How do you find out whether AI crawlers are reading your website?

High confidenceVerified 1 Sept 2026
Filter your server or CDN access logs by the published AI user agents - GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Applebot among them - then group the hits by URL and status code to see which sections they fetch and which they fail to reach.

Sources

The Blind Spot

Why Your Analytics Cannot See This

Client-side analytics fires when a browser executes JavaScript. An AI crawler requests the HTML and leaves, so the visit never appears in the dashboard the marketing team reviews. The only durable record is the access log kept by your server or CDN, which is why teams can be confident nothing is happening while a meaningful share of their pages is being fetched daily.

This is why measurement belongs before protocol work rather than after it. Deciding whether to restructure a site for AI search visibility is a different conversation once you can see which assistants fetch which sections, and which of them are hitting errors.

What the Log Actually Tells You

Three things, in order of usefulness. First, which agents appear at all - the difference between a training crawler and a live retrieval fetch matters, because the second means an assistant is answering a question about you right now. Second, which URLs they concentrate on, which is rarely the pages you would have guessed. Third, the status codes: a crawler collecting redirects, 404s or 403s from a misconfigured rule is being told your content does not exist, and that failure is completely silent from the inside.

Separating Training From Retrieval

The distinction worth drawing is between agents gathering material for model training and agents fetching a page to answer a live question. They warrant different decisions: the first is a licensing and policy question for the business, the second is a visibility question. Both are controlled from robots.txt, and a blanket block applied without that distinction is the common way a site accidentally removes itself from assistant answers while intending only to opt out of training. Getting that wrong is closer to losing rankings in a migration than to a privacy control.

How to audit AI bot traffic in your logs

Half a day for a first passIntermediate

A first pass that takes an afternoon and usually changes how seriously the rest of the AI-visibility work gets taken.

Tools Required

  • CDN or origin access logs
  • A log query tool or spreadsheet
  • Your current robots.txt

Steps

  1. 1

    Get access to the right logs

    Use the CDN or edge logs if you have them, since they see requests your origin never does. Otherwise use origin access logs. Aim for at least a month so weekly patterns are visible.

  2. 2

    Filter by the published user agents

    Search the user-agent field for the documented AI tokens rather than a generic 'bot' match, which will drown you in uptime monitors and scrapers. Keep the exact strings, because vendors publish them and they change.

  3. 3

    Group by agent, URL and status code

    Three simple aggregations answer most of the question: which agents visit, what they fetch most, and what they are being served. Anything other than a 200 on a page you want quoted is a finding.

  4. 4

    Cross-check against robots.txt

    Confirm the rules in place are the rules you intended. A directive added years ago, or a broad disallow inherited from a template, is a routine discovery at this step.

  5. 5

    Decide policy per agent, not in bulk

    Separate training crawlers from live retrieval fetches and decide each deliberately. Blanket blocks are how a site removes itself from assistant answers while meaning only to opt out of training.

Key Takeaways

The Cheapest Useful Step on the List

  • AI crawler traffic is invisible to client-side analyticsCritical

    These are server-side fetches that never execute JavaScript, so no analytics tag fires. The access log is the only durable record of what was taken.

  • Status codes matter as much as visit countsCritical

    An agent collecting redirects, 404s or 403s is being told your content does not exist. That failure produces no error anyone sees and can persist indefinitely.

  • Training crawlers and retrieval fetches deserve different decisionsImportant

    One is a licensing question for the business, the other a visibility question. A single blanket robots.txt rule collapses them and commonly removes a site from assistant answers by accident.

  • Measure before restructuring anythingImportant

    Knowing which assistants fetch which sections changes which of the remaining protocol decisions are worth making, and in what order.

An afternoon in the access logs establishes whether the traffic justifying AI search work exists at all, and routinely surfaces silent failures that no dashboard reports.

Auditing AI Bot Traffic: Common Questions

Which user agents should we be filtering for?
Filter for the tokens the vendors publish rather than a generic bot match: GPTBot and OAI-SearchBot alongside ChatGPT-User, ClaudeBot, PerplexityBot and Applebot among others. Vendors document these and add to them, so the list is worth re-checking rather than hard-coding once and forgetting.
Why does none of this show up in our analytics?
Client-side analytics runs when a browser executes JavaScript. AI crawlers request the HTML and leave without running scripts, so the visit never registers. Server or CDN access logs are the only place the request is recorded, which is why teams are often confident nothing is happening while their pages are fetched daily.
Should we block AI crawlers?
That depends which ones and why, and it is worth separating the two cases. Blocking agents that gather training material is a licensing and policy decision for the business. Blocking agents that fetch pages to answer a live question removes you from assistant answers. A single blanket rule collapses that distinction, which is the most common way a site opts out of visibility while intending only to opt out of training.
What is a realistic first finding?
Usually two things: the agents concentrate on sections nobody expected, and some of them are receiving redirects or errors on pages the business cares about. Neither shows up in any dashboard, and the second is generally fixable the same week it is found.
Do we need a tool for this?
Not for a first pass. Access logs, a query tool or spreadsheet, and your current robots.txt are enough to answer which agents visit, what they fetch and what they are served. Ongoing monitoring is worth automating once the first pass proves there is something to watch.

Want someone to read the logs with you?

Written by Cameron Young, CEO

  • 1 min read

Related reading