Can AI assistants actually read your site? Four checks

Four checks tell you whether an AI assistant can read your site at all. Run them before you write a single new page, because content strategy is wasted effort on a site that crawlers cannot parse. Each check takes minutes. None of them requires a paid tool.

Check 1: read robots.txt the way the crawlers read it

Open yoursite.com/robots.txt and look for five user agents specifically: GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot and Google-Extended.

These are not interchangeable. GPTBot, ClaudeBot and Google-Extended are bulk crawlers that gather content for training and retrieval indexes. ChatGPT-User is different: it fetches a page because a live person just asked a question that needs it. Blocking the first group is a policy decision with real tradeoffs. Blocking ChatGPT-User means that when a user asks about you by name, the assistant cannot open your site to answer.

Check for wildcard blocks too. A `User-agent: *` with `Disallow: /` in a staging config that shipped to production blocks everything.

Check 2: does your content exist before JavaScript runs

The fastest test in this entire list:

curl -s https://yoursite.com/your-key-page | grep -i "a distinctive phrase from the page"

If that returns nothing, your content is client-rendered. Some crawlers execute JavaScript. Several do not, and the ones that do treat it as optional work under a time budget.

We took on an AI learning platform whose entire site rendered as a client-side single-page app. Crawlers could not see it at all. Not partially, not slowly. The HTML response was a div and a script tag. Every argument about keyword targeting on that site was moot until the rendering changed.

Server-side rendering or static generation solves this. If a full migration is not on the table, prerendering the highest-value routes gets you most of the way.

Check 3: is structured data present, and is it valid

Structured data is how you state facts about yourself in a form that does not depend on a parser interpreting prose correctly. At minimum: Organization on the site, Article on posts, FAQPage where you genuinely answer questions, BreadcrumbList for hierarchy.

Validate it rather than assuming. Two failure modes are common and both are quiet. The first is schema that throws errors and is therefore ignored entirely. The second is worse: schema that parses fine but contradicts the visible page, listing a service the page does not describe or an author who is not credited anywhere in the copy. That mismatch is a signal you do not want to send.

Check 4: can an answer be lifted out without reading the whole page

Assistants extract passages. They do not read your page top to bottom the way a prospect does.

So put the answer in a block that survives being removed from its context. Under a question-shaped H2, write 40 to 60 words that answer it completely, with the subject named rather than implied. Then expand underneath. Comparisons go in tables, because a table is unambiguous when parsed. Definitions go in their own short paragraph, not buried in the third sentence of a section about something else.

The test: copy any single paragraph out of your page, paste it somewhere with no surrounding context, and read it. If it still makes sense and still names what it is about, it is extractable. If it starts with "This means that" or "As we mentioned above", it is not.

The CDN default that causes most of this

On an audit of a US retention-marketing agency, robots.txt was blocking the major AI crawlers through a default CDN setting that nobody had chosen. Only 5 of 52 pages were indexed. There was no schema markup anywhere and the title and meta description were identical on every page. Form submissions had fallen to zero across twelve months while their paid traffic doubled.

Nobody on that team made a bad decision. A bot-management default shipped on, the marketing team never saw a robots.txt, and the developers never saw the traffic reports. That gap is where most of these problems live, which is why the Growthym team runs the crawl and the log review together rather than handing marketing a PDF.

Run the curl check on your five most important pages this afternoon. If any of them come back empty, that is your entire roadmap for the next sprint, and their seven-day diagnostic covers the rest of the stack while you fix it.