GPTBot Doesn't Render JavaScript, and Neither Does Claude or Perplexity: What That Means for How a Site Gets Built

Does GPTBot actually render the JavaScript on a page before deciding what a business's site says, the way Googlebot does? No. OpenAI's own crawler documentation describes GPTBot's purpose as crawling content that may be used in training its generative AI foundation models, and a large-scale analysis of its live traffic found no evidence that it executes any of the JavaScript it encounters along the way. The same holds for Anthropic's ClaudeBot and Perplexity's PerplexityBot. That single architectural fact, whether a crawler runs a real browser engine or just an HTTP client, decides whether a modern JavaScript-heavy website is actually visible to the systems that increasingly decide what gets cited in an AI-generated answer.

Vercel's December 2024 traffic study, The Rise of the AI Crawler, remains the most detailed public breakdown of what these bots actually do on arrival, and the architecture behind its central finding, a plain HTTP fetch with no browser engine attached, has not changed since. The report states plainly that none of the major AI crawlers currently render JavaScript, naming OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, Meta's Meta-ExternalAgent, ByteDance's Bytespider, and Perplexity's PerplexityBot explicitly. Analyzing traffic across its own hosting network in a single month, Vercel recorded 569 million GPTBot fetches and 370 million Claude fetches; GPTBot downloaded JavaScript files in about 11.5% of its requests and ClaudeBot in about 23.84%, and in neither case did the crawler ever execute what it downloaded. These bots request a URL, read whatever HTML comes back in that first response, and move on.

Google's own Search Central documentation describes a fundamentally different process for its own crawler. Googlebot processes a page in three phases, crawling, rendering, and indexing, and after the initial crawl a page queues for rendering, where, per Google's documentation, a headless Chromium renders the page and executes the JavaScript, with Google then using that fully rendered HTML, not the raw server response, to index the page. A React or Vue single-page application that assembles its content through a client-side fetch call can look complete to Googlebot once that rendering pass finishes, while showing GPTBot, ClaudeBot, and PerplexityBot nothing but an empty shell, because none of them run the browser step that makes the content appear.

The crawlers behind that empty shell are not interchangeable, and treating every one of them as one generic "AI bot" misses a real strategic choice in how a site's robots.txt gets written. OpenAI documents three separate agents: GPTBot, used to crawl content for training its models; OAI-SearchBot, used to surface a site in ChatGPT's search results; and ChatGPT-User, triggered live when a person asks ChatGPT or a Custom GPT to visit a specific page, a case where OpenAI's own documentation notes that robots.txt rules may not apply because the fetch is user-initiated rather than automated crawling. Anthropic runs the same three-way split: ClaudeBot for training, Claude-User for live user questions, and Claude-SearchBot to improve search result quality. Perplexity runs a single PerplexityBot that, per Perplexity's own documentation, is used to surface and link websites in Perplexity's search results and is not used to crawl content for AI foundation models. A robots.txt file that blocks GPTBot wholesale in an attempt to opt out of AI training does nothing to stop OAI-SearchBot from indexing the same site for ChatGPT search, and a business trying to appear in Perplexity's answers has to allow PerplexityBot specifically rather than assuming a blanket AI-crawler block already covers it.

None of this is a copywriting problem. A page can be built with an entity-explicit direct answer, clean structured data, and every other GEO best practice applied, and still be invisible to GPTBot if the actual words only appear after client-side JavaScript finishes running. Next.js's own documentation states this plainly for exactly the pattern that produces the failure: client-side rendering "can impact SEO" because "some search engine crawlers might not execute JavaScript and therefore only see the initial empty or loading state of your application." That warning was written with traditional search engines in mind, but it describes the AI-crawler problem with equal precision, since the underlying mechanism, a crawler that fetches HTML once and never runs a script, is the same one behind both risks.

The fix is a rendering decision made at build time, not a retrofit applied after launch. Server-side rendering and static site generation deliver a page as fully formed HTML the moment a crawler's first request lands, whether that crawler is Googlebot's headless Chromium instance or GPTBot's plain HTTP fetch, because the content was already assembled before any bot arrived. Next.js's hybrid rendering model lets each page make that choice independently: static generation for content that does not change often, like a service page or a blog post, server-side rendering for anything that needs request-time data, and client-side rendering reserved for the interactive pieces that only ever need to be read by a human's browser.

This is the same distinction that separates a site actually built for Generative Engine Optimization from one that is merely optimized for a human reader: it is not enough for the right words to exist somewhere in a component's source code, they have to be present in the actual bytes a crawler receives on its first and only request. The same logic that makes a website the most controllable GEO signal a business owns applies with more force once the rendering layer is understood, because no amount of on-page structure recovers content that a crawler never received in the first place.

Italian DesAIgns builds every web design project on server-side rendering and static generation as the default, not an optimization applied after launch, specifically because the rendering layer decides whether GPTBot, ClaudeBot, PerplexityBot, and Googlebot all receive the same fully formed content on first contact. A quick AI visibility check shows whether a business's existing site is already serving that content correctly, or whether its actual words are sitting behind a JavaScript step that most of the crawlers deciding what gets cited will never run.