Contents8
An AI assistant that cites your website first had to read it. That step, the simplest on the face of it, is where many sites fail without knowing: a crawler turned away by an option nobody switched on, text that only appears once JavaScript runs, an answer buried under three paragraphs of introduction. Our article on getting cited by ChatGPT covers content and measurement; this one looks at what the site’s build decides. It belongs with our guide to migrating to a static site with Astro.
In short
AI assistants’ crawlers fall into three families: training, search, and visits on behalf of a user. They can be treated differently.
The robots.txt that counts is the one the site actually serves, not the one in the repository: a network layer can rewrite it.
A Vercel study from December 2024 found no major AI crawler that executes JavaScript, apart from Gemini’s, which goes through Googlebot.
A citable page answers a question in its first lines, with dated and sourced facts.
Google states that no special file or markup is needed for its AI features: an indexed, well-built page is enough.
Three families of crawlers, three decisions
Each AI vendor sends several crawlers, which do not serve the same purpose. OpenAI describes its own in its documentation on bots: GPTBot is used to train models, OAI-SearchBot to surface websites in ChatGPT’s search answers, and ChatGPT-User visits a page when a user asks. Anthropic follows the same split in its help page with ClaudeBot, Claude-SearchBot and Claude-User.
What each family of crawlers does with your page
- Training
The page may be used to train future models.
- Search
The page is indexed so it can be suggested and cited in answers.
- On-demand visits
A user asks the assistant to read the page.
Refusing training while accepting search is a valid choice, and often the one made by companies that want to be cited without feeding the models. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. For Google, the rule is simpler: its page on AI features says a page must be indexed and eligible to be shown with a snippet, with no additional technical requirements.
The robots.txt served is not always the one in the repository
Watch out
A client found out at the audit
At Palmora Property, a real estate agency in Phuket that we support, the robots.txt in the repository allowed everyone. The one actually served started with a block added by the network layer, absent from the code, which refused GPTBot, ClaudeBot, CCBot and Google-Extended. The audit of 22 July 2026 caught it, and the setting was switched off the same day. The episode is told in detail in our article on getting cited by ChatGPT.
The lesson applies to every site: a content delivery network, a web application firewall or a “bot protection” option act before your site does. So you check the production URL, not the file you wrote. Our article on what a proxy like Cloudflare protects comes back to these settings made on your behalf.
Text present in the HTML
A crawler reads what the server sends it. If the page’s text only appears after JavaScript runs in a browser, a crawler that does not execute JavaScript sees an empty page. That is the case for most AI crawlers, according to the study Vercel published on 17 December 2024:
The results consistently show that none of the major AI crawlers currently render JavaScript.
The study notes one exception: Gemini relies on Googlebot’s infrastructure, which renders JavaScript. Its recommendation fits in one sentence: server-render any important content. That is exactly what a static site does: each page is built at publication and sent complete, title, text, lists and links included. JavaScript remains possible for what is not content, such as a form or a counter.
The test takes a minute: view the page source (not the browser inspector) and search it for a sentence from the text. If it is not there, crawlers that do not execute JavaScript cannot see it either.
One page, one question, the answer up front
An assistant cites a passage that answers, not a page that circles the subject. Structure helps: a title that asks the question, a self-contained answer in the first lines, subheadings that split the sub-questions, lists where there are steps or criteria.
From the field
What we do on this blog
Every article on this blog carries, under its title, a direct answer that reads on its own. It is a required field in the article schema: an article without it does not build, and a check flags any answer outside 40 to 60 words. Links to sources become numbered notes, listed at the end of the article.
This discipline serves the busy reader first. That it also serves assistants is no accident: both look for the same thing, a clear answer they can verify.
Dated, sourced and consistent facts
A claim with no date or source is hard to reuse, for an assistant as for a journalist. Three habits make a text reusable:
- Date it. The publication date and the last update date, visible on the page.
- Source it. A figure points to its source, where it appears.
- Stay consistent. Google reminds site owners that structured data must match the visible text on the page, and that no special schema.org markup is required for its AI features.
One stable URL per topic
An assistant that has read a page must be able to find it again. A URL that changes without a redirect loses what had been read and cited; two pages covering the same topic compete with each other. Vercel’s study also recommends clean URLs and an up-to-date sitemap. On a static site, the sitemap is generated at every build with real modification dates, as on this site since August 2026, and each language can have its own URL, as explained in our article on multilingual websites.
- Read the robots.txt served
At the production URL, and decide crawler by crawler.
- Search for a sentence of text
In the source code of three key pages.
- Reread the first lines
Of those pages: do they answer the question in the title?
- Check the dates and sources
Visible on the page.
- Check the sitemap
Every useful page, no redirected URL.
An llms.txt file can come on top, once the structure is stable; our article on llms.txt says what to expect from it.
The rest of the project
Readability by AI depends on the whole build of the site. The tool you choose decides what is served as HTML (Webflow, Framer or Astro), and speed matters for crawlers as for visitors (Core Web Vitals). An empty page stays empty for an assistant, whether it targets a city or an industry (when those pages help), and a redesign is prepared page by page (keep, merge or delete) with its 301 redirects. Hosting (Cloudflare, Netlify or a VPS, hosting in Europe), the form (on a static site) and the budget (how much a website costs) follow. On the writing side: Markdown or a CMS, headless Astro and Sanity and disclosing that a text was written by AI. An accessible page is also easier for a machine to read (the 2025 EU obligation), a site without useful content generates no enquiries (the five blocks), and a compromised WordPress puts at stake what visitors and crawlers read (hacked site).
Our static site migration with Astro delivers complete HTML pages, URLs kept, with a sitemap generated at every publication.
Frequently asked questions
Should we allow every AI crawler?
No, you should decide. Training crawlers, such as GPTBot or ClaudeBot, are used to train models; search crawlers, such as OAI-SearchBot or Claude-SearchBot, are used to cite pages in answers. Many companies refuse the former and accept the latter. What matters is that this choice is written in the robots.txt actually served, and checked at the production URL.
Is my React site invisible to AI?
If it only displays its text after JavaScript runs in the browser, most AI crawlers do not see it, according to Vercel’s December 2024 study. Server-side rendering or static generation of the pages solves the problem without changing how the site looks. Check by searching for a sentence of text in the page source, which shows what a crawler actually receives.
Does structured data help get cited?
It helps engines understand the page, provided it describes what is visible. Google states that no special schema.org markup is needed for its AI features, and that structured data must match the text displayed. It replaces neither a clear answer at the top of the page nor dated and sourced facts, which remain what an assistant can quote.
Is an llms.txt file enough to make a site readable by AI?
No. It is an optional summary at the root of the site, which guides agents that read a site on demand. Google writes that no special file is needed for its AI features. An llms.txt does not make up for a refused crawler, text invisible without JavaScript, or a page that does not answer its own question.
Sources
- OpenAI, crawlers and user agents (new tab)developers.openai.com
- Anthropic help centre: Anthropic’s crawlers (new tab)support.claude.com
- Google Search Central, AI features and your website (new tab)developers.google.com
- Vercel, The rise of the AI crawler, 17 December 2024 (new tab)vercel.com
Jim · founder of Brain Plus AI
Also in websites, migration and astro
Understand8 min The topic guide
Static site with Astro: why migrate, without losing your SEO
A static site generated by Astro replaces an ageing WordPress or Drupal without changing a single address: pages in plain HTML served by a CDN, no plugin to patch, and a publishing chain with rollback. What it changes, how the migration goes, and when not to do it.
Choose and price8 min
City or industry pages: when they help, when they hurt
One page per city or per industry can bring qualified enquiries, or make the whole site look like a keyword farm. What Google calls a doorway page, the signs of a useful page, and the order in which to write them.
Understand8 min
Multilingual website with Astro: one language, one URL
A multilingual website rests on one simple rule: each language has its own URL, and each page knows its translations. What the rule protects, what Astro does to keep it, and the mistakes that cost money when it is forgotten.