Testing whether an AI agent can use your website
An agent test puts a scripted assistant to a customer's task on your site and on comparable sites, keeps the transcript, and shows where the site stopped it.
Quick answer: An agent test runs fixed customer tasks through a read-only scripted assistant against a site and comparable sites, keeps transcripts, and maps each failure to a site mechanism such as script-only text, bot challenges or PDFs.
- AI Search Visibility
- Measurement
- Machine-Readable Delivery
Quick answer
Can an AI agent actually complete a task on your website?
Give a scripted assistant a defined task, such as finding the fee for a service or checking availability for a date, let it browse the site under a named user agent, and keep the transcript. The transcript shows where it succeeded, where it guessed, and where the site stopped it.
Additional Context
Some customers now delegate the first pass of a purchase to an assistant. A site that a browser renders perfectly can still be one that an agent cannot read, and nothing in analytics reports it.
Sources
- Google Search Central - JavaScript SEO basics
How Google processes JavaScript in three phases, and why content that only appears after script runs can be missed.
- Cloudflare Docs - Verified bots
How Cloudflare verifies known crawlers and fetchers and how a site's bot rules treat them.
- OpenAI - Overview of OpenAI crawlers
OpenAI's published user agents, separating the training crawler from the search crawler and the user-triggered fetcher.
The assistant test
What the test is
An agent test is a fixed set of tasks, written as a customer would phrase them, run by a scripted assistant against the business's site and against a benchmark set of comparable sites. The assistant reads pages the way an assistant does: it fetches HTML, follows links, and tries to extract the answer. Every run keeps the transcript, the pages fetched, the status codes returned, and whether the final answer was correct against the business's own published facts. The same tasks run on the same day each week, so the result is a series beside the answer record.
The benchmark set is the point. A site that passes four tasks in ten is a different story when the comparable sites pass eight, and a different story again when they pass two. The task set is written for the industry, because the tasks a customer gives an assistant about a clinic are not the tasks they give it about a lender or a resort.
Where agents fail
The same kinds of failure recur. Text appears only after JavaScript runs, so the fetched HTML is a shell. A consent or cookie interstitial cannot be dismissed by the agent. A bot management rule challenges every unfamiliar client and serves the agent a challenge page. A fee table is published as a PDF or an image. A price appears only after a date picker is used. A tab component leaves its other tabs out of the HTML. Infinite scroll hides the second half of a list. A form is the only route to an answer the site could have stated.
Each of these is invisible from inside the business, because the people who work there use a browser, accept the cookie banner once, and never see the challenge page. Google's own documentation on JavaScript explains the three-phase processing that lets its crawler eventually render script, and most assistant fetchers do less than that. The practical rule is that the fetched HTML has to carry the answer. The page on machine-readable delivery describes what that takes.
The test never submits a form
The scripted assistant reads. It submits nothing. It runs under one named user agent so that it can be found in the access logs and, if the business wishes, allowed or blocked as a known client. Cloudflare and other edge providers publish how they verify known bots, and the test client is configured to be identifiable in the same spirit.
Reading the transcript as an engineering ticket
A failed task maps to a change in the site. A challenge page maps to a bot rule. A script-only tab maps to server rendering. A PDF fee table maps to an HTML table with structured data. A wrong canonical maps to the CMS template that generates it. The ticket names the transcript as its evidence and the mechanism as its diagnosis. The method for writing those tickets is set out in turning findings into engineering tickets, and the agent test is one of the measurements inside AI Visibility Management.
What the test does not do is say what the site should offer. If the task cannot be completed because the business has never published the fact, that is recorded as unverifiable and handed back as a question for the business. The engineering begins once the fact exists.
Step-by-step
How to run an agent task test
A day for the first passAdvanced
A first pass that shows whether an assistant can get a customer's answer from your site at all.
Tools Required
- Real customer enquiries
- An agent framework with HTML fetching
- Access logs to confirm the agent's requests
- A benchmark list of comparable sites
Steps
- Write the tasks in customer language(Two hours)
Take real enquiries and turn each into one task with one checkable answer: the fee for a named service, whether a date is available, which location serves a suburb.
- Choose the benchmark sites(An hour)
Pick comparable businesses a customer would also ask about. The task set must make sense on every site, so keep the tasks generic to the industry.
- Run a scripted assistant under a named user agent(Half a day to set up)
Use an agent framework that fetches HTML and follows links, set a user-agent string you can find in the logs, and never let it submit a form.
- Score against published facts and keep the transcript(An hour per run)
Mark each task as correct, wrong or unverifiable against the site's own pages. Keep every fetched URL, status code and the full transcript.
- Trace each failure to a mechanism(As failures arise)
Challenge page, script-only text, PDF, interstitial, canonical, missing fact. Write the mechanism beside the failure, because that is the ticket.
Summary: Write customer tasks with checkable answers, run a read-only scripted assistant under a named user agent against your site and comparable sites, and trace each failure to a mechanism.
Key Takeaways
What an agent test shows that a browser cannot
- A site a browser renders perfectly can still be one an agent cannot readCritical
Script-only text, consent interstitials, bot challenges, PDFs and tab components all look fine to staff and stop a fetcher cold.
- The benchmark set gives the result its meaningImportant
A pass rate only reads against comparable sites doing the same tasks, so the task set is written for the industry and run on every site alike.
- The test reads and never submitsImportant
One named user agent, no forms, no bookings, so it can be found in the logs and treated as a known client.
- Each failure maps to a site change, and a missing fact maps to a questionCritical
Challenge pages, rendering, PDFs and canonicals are tickets. A fact the business never published is handed back before any engineering starts.
A scripted assistant running fixed customer tasks against a site and its comparables, with transcripts kept, shows where the site stops an agent. Every failure traces to a mechanism the site controls, and the transcript is the evidence the ticket carries.
Questions about agent task tests
Is this the same as testing the site in ChatGPT?
No. Asking an assistant a question measures what it says. The agent test measures whether a scripted assistant can get the answer from your site by reading it. A site can be quoted correctly from a directory while being unreadable itself, and only the second test shows that.
Will the test trip our bot protection?
It may, and that is a finding. The test runs under one named user agent so the business can see it in the logs and decide whether to allow it. Whether the assistants customers use are also challenged is checked in the access logs against their published user agents, because an edge provider may treat a verified fetcher differently from an unknown client.
Does the test touch our booking or enquiry systems?
No. It reads pages, follows links and extracts answers. It does not submit forms, create bookings or place orders.
What if the task cannot be done because the fact is not on the site?
The result is recorded as unverifiable and handed back as a question for the business. Whether to publish the fact is the business's decision. Publishing it in a form an agent can read is the engineering that follows.
