The sources AI assistants cite when they describe you
For a current fact an assistant fetches pages while it answers. The cited list shows which pages it trusted, and whether your own site was one of them.
Quick answer: AI assistants cite the pages they fetch at answer time. Tallying cited domains per question each week shows whose word they take, and a business's absence from its own answers traces to blocked agents, script-only text, PDFs, canonicals or thin pages.
- AI Search Visibility
- Measurement
- Machine-Readable Delivery
Quick answer
Where do AI assistants get their information about a business?
From what the model already holds and, for anything current, from pages it fetches while it answers. For a price, an opening hour or a licence the assistant usually fetches, and the list of sources it cites shows which pages it trusted and whether your own site was among them.
Additional Context
The cited list changes week to week. Tallied over time it shows the share of answers about a business that rest on the business's own pages, on directories, on review sites or on competitors.
Sources
- OpenAI - Overview of OpenAI crawlers
OpenAI's published user agents, separating the training crawler from the search crawler and the user-triggered fetcher.
- Google Search Central - Google crawlers and fetchers
The published user-agent tokens for Google's crawlers and user-triggered fetchers, including the Google-Extended control.
- Google Search Central - AI features and your website
Google's guidance that AI Overviews and AI Mode use the same eligibility rules as Search and need no special markup.
Where answers come from
Three kinds of fetch
The vendors publish their user agents, and the names matter because each one does a different job. One crawler gathers pages for training, and what it collects shapes what the model holds months later. One crawler builds a search index that the assistant consults when it answers. One fetcher goes to a page because a user asked a question right now and the assistant decided to read the page before replying. OpenAI documents these as three separate agents, and Google publishes its crawlers and its user-triggered fetchers separately, with the Google-Extended robots token controlling training use.
A site can allow one and block another, on purpose or by accident. A robots rule written years ago to keep training crawlers out can also keep the live fetcher out, and from then on the assistant answers questions about the business from everyone's pages except the business's own. The way to see which agents reach the site, and what they are served, is in the access logs, which auditing AI bot traffic covers.
Reading the cited list over time
Each weekly answer carries a list of sources. Record the domain of each one against the question it answered. After a month the tally shows, per question, how often the business's own site was cited, how often a directory or an aggregator was, how often a review site, a news article, a government register, or a competitor. That share is a dimension of the record in its own right. It moves when the pages change and it moves when the assistant changes how it searches, and the series shows both.
The tally also shows which pages on the business's site are doing the work. The tally shows which pages are cited, and they are often not the ones the business considers its most important. Pages whose facts sit in HTML text are cited more often in the record than pages whose facts sit in PDFs or behind script.
When the site is absent from its own answer
The reasons are mechanical and they can be checked. The live fetcher is blocked at the firewall or in robots.txt. The page returns its text only after JavaScript runs, and the fetcher does not run it. The fact sits in a PDF or an image. The page exists under two URLs and the canonical points at the wrong one. The page is thin on the question and the assistant found a fuller answer elsewhere. The page describing machine-readable delivery sets out how a site serves agents the same content a browser gets.
What to change
Allow the retrieval and search agents you want to be read by, and decide separately about the training crawler. Put every fact a customer asks about into HTML text on the page where a customer would look for it. Mark the page up with the structured data that describes the fact. Keep the fact identical on every surface the business controls, because an assistant that finds two values hedges or picks the wrong one. These are the changes that put the business's own pages back into the cited list. Whether the assistant then cites them is recorded, week by week, and never promised.
The work described here is the source dimension of AI Visibility Management. The adjacent dimension, what the assistants actually state once they have read the pages, is covered in recording what assistants say.
Key Takeaways
How to read the sources behind an assistant's answer
- Training crawlers, search crawlers and live fetchers are different agentsCritical
The vendors publish them separately. A rule meant for one commonly blocks another, and the live fetcher is the one that answers a customer's question today.
- The cited domains, tallied per question, are a measurementImportant
The share of answers resting on the business's own pages, on directories, on review sites or on competitors changes over time and the series shows it.
- A few pages carry most of the citationsImportant
The tally shows which of the business's pages are cited, and pages whose facts sit in HTML text are cited more often than pages whose facts sit in PDFs or behind script.
- Absence from the cited list has a mechanical causeCritical
Blocked agents, script-only text, facts in PDFs, wrong canonicals and thin pages are the usual ones, and each one is checkable and fixable on the site.
The pages an assistant cites about a business are a measurable dimension. Tallied weekly they show whose word the assistant takes, and the reasons a business's own site is missing are mechanical faults in delivery that engineering can correct.
Questions about AI assistant sources
Why does an assistant cite a directory about us and not our own site?
Usually because the directory's page states the fact plainly in HTML and yours does not, or because your site blocked the fetcher, or because the fact on your site sits in a PDF or behind a script. The access logs and a fetch of your own page with a plain client show which.
Should we block AI crawlers?
Decide per agent. Blocking the training crawler is a policy decision for the business. Blocking the search crawler or the live fetcher keeps the business's own pages out of the live fetch, so the answer rests on other sites. The published user-agent lists let robots.txt treat them differently.
Can structured data make an assistant cite us?
No one can promise that. Structured data states a fact in a form a machine reads without guessing, which removes one reason for an assistant to prefer another page. Whether the assistant then cites the page is what the weekly record shows.
How long before a changed page shows up in the cited list?
A live fetcher can read a changed page the same day. A search index may take longer, and what the model holds from training changes only when the model does. The record shows the date each assistant's answer changed, which is the timeline the record supports.
