Can AI Assistants Actually Read Your Website? How to Check

An AI assistant can only describe a business it can fetch, parse and attribute. Checking that means confirming three things: crawler access is permitted, the meaningful text is present in the served HTML, and the pages state plainly who and what they are about.

Key takeaways

• Most AI visibility problems are access problems, not writing problems. Fix reachability before rewriting anything.
• Everything in this diagnostic is inspectable from your own machine — no vendor dashboard is required.
• Text that appears only after JavaScript runs is text a fetch-only reader may never see.
• Consistent naming matters more than clever phrasing. A system cannot attribute a claim to an entity it cannot pin down.
• Nothing here controls what an assistant says about you. It removes the reasons it cannot say anything accurate.

Why this check comes first

Discussions about AI visibility usually start with tone and structure. That is the second question. The first is whether a system fetching your URL receives your content at all — because if it does not, the finest answer-first paragraph ever written is invisible.

This ordering matters because the two failures look identical from outside. A business that is never mentioned by an assistant cannot tell whether its pages were read and judged unremarkable, or never read. The checks below separate those cases.

Step 1: confirm crawler access

Open /robots.txt on your own domain and read it as a rulebook rather than a formality. Google’s introduction to robots.txt explains the mechanics: the file tells crawlers which paths they may request, and rules are matched per user-agent.

Two failure modes are common. The first is a file copied from another project years ago that disallows a path which now holds real content. The second is a rule aimed at one user-agent that a site owner assumes covers all of them; it does not. If you intend assistants to read the site, the agents you care about need to be permitted explicitly rather than by accident.

Decide this deliberately. Allowing AI crawlers is a business choice, not a default — and it is entirely reasonable to decide otherwise. What is not reasonable is discovering by accident that a stale rule made the decision for you.

Step 2: read what a fetch actually returns

Request one important page and look at the raw response body, not the rendered browser view. The browser runs your JavaScript; a fetch-only reader may not. If your product description, pricing or answer text is absent from that response, treat it as absent for the reader.

Check three pages rather than one: the homepage, one commercial page and one article. They are usually rendered by different templates, and it is common for exactly one of them to be the problem.

Step 3: make the page say what it is about

A reader that has your HTML still has to work out what the page concerns and who is responsible for it. Structured data is the least ambiguous way to say so. The Schema.org vocabulary provides shared types for exactly this, and Google’s introduction to structured data describes how the markup is meant to correspond to what a visitor actually sees.

That correspondence is the rule that matters. Markup describing content that is not on the page is worse than no markup: it is a machine-readable inaccuracy, and it is trivially detectable.

Naming discipline does the rest. Use one exact form of your company name, one exact form of each product name, everywhere. A system reconciling "the platform", "our tool" and three spellings of a product cannot confidently attribute anything to any of them.

Step 4: publish a plain-language summary file

A short llms.txt at the site root, listing your main pages with one honest sentence each, gives a reader a curated map instead of leaving it to infer structure from navigation. It is a convention rather than a standard, and no engine is obliged to fetch it, so treat it as cheap insurance rather than a mechanism.

The discipline of writing it is itself useful: describing every page in one sentence tends to expose the pages that do not have a clear purpose.

Step 5: audit how consistently you name things

A system that has read your pages still has to decide what a claim should be attributed to. That decision is made on names, and most sites make it harder than it needs to be by using several names for the same thing.

Run the audit by listing, in one place, every way the site refers to the company, the product, each paid plan and each service. Then read the list looking for drift rather than for style. A product that appears under its full name in the hero, an abbreviation in the navigation and an internal shorthand in the documentation is three different things to a reader that cannot ask you which one you meant.

Three checks are worth doing by hand:

• Does the company name appear in exactly one form, including punctuation such as a trailing Inc. or Ltd.?
• Does every plan and product name match across the pricing page, the capability pages and the articles that mention it?
• Where a page names both the company and the product, is it clear which is which to someone who does not already know the business?

This is unglamorous work with no dashboard attached to it, and it is the cheapest item on this list. It guarantees nothing about how an assistant describes you. What it removes is the specific reason a system would attribute your claim to the wrong entity, or decline to attribute it at all.

How to verify each fix

• Access: request /robots.txt and confirm the rules say what you intended for each agent you care about.
• Content: fetch each key page and search the raw body for a distinctive sentence you expect to be there.
• Markup: parse each JSON-LD block and confirm it parses cleanly, then confirm every value it asserts appears on the visible page.
• Naming: grep the site for every variant of your own names. One form should survive.
• Summary file: confirm it returns 200, is plain text, and lists only URLs that exist.

Every one of these produces a yes or no answer, which is the point. Re-run them after any redesign, because a template change is the most common way a passing site quietly starts failing.

What you can measure, and what counts as evidence

The checks above share a useful property: each has a definite answer that does not depend on anyone’s judgement. Whether a path is disallowed, whether the text is present in the served response, whether a page carries structured data — those are facts you can record with a date and re-check later.

Treat that record as the measurement. Note the date, the URL checked and the result, then repeat after every significant change to the site. A regression on any of them is a real finding, because it means content a reader could previously fetch is now unreachable.

Asking an assistant whether it knows your business is a much weaker test, and it is worth being precise about why. Answers vary between systems, between sessions and with how the question is phrased, so a change of wording can produce a different answer with no change to your site at all. A system may also describe a business from sources other than its website. An assistant naming you correctly is therefore encouraging but proves little about your pages, and an assistant getting you wrong is a prompt to re-run the mechanical checks rather than a measurement in itself.

We publish no expected result for any of this, because there is none to publish honestly. The claim these checks support is narrow: your content is reachable and legible. What any system then does with it is outside your control and ours.

What this cannot do

None of this determines whether an assistant mentions you, how it characterises you, or whether it cites you at all. Those systems change, and their behaviour is not a contract. Anyone selling a guaranteed placement in an assistant’s answer is describing something outside their control.

It also cannot compensate for a page with nothing specific in it. Retrievability is a precondition, not a substitute for having something worth quoting — which is the subject of the article on what actually makes a site quotable.

Finally, a passing diagnostic is a snapshot. It describes today’s site and today’s crawler behaviour, and both move.

Where this fits

The capability page for this work is AI visibility, which covers the artifacts we ship on every site and, just as importantly, what we decline to promise about them.

Written by Vincent Chen, who is accountable for the claims on this page.

Sources

• Google Search Central — Introduction to robots.txt
• Google Search Central — Introduction to structured data
• Schema.org — Shared vocabulary for structured data