Guide

ChatGPT Can't Read Your Site? The curl Test and the Raw-HTML Fix

Summary

Usually because the page's words arrive by script. A browser runs the JavaScript a site builder ships and paints the name, the fee and the states served onto the screen; the fetcher an assistant sends reads only the HTML the server returns, and at the major AI companies it executes none of that script, so the page reads as blank. A robots.txt line or a firewall rule produces the same message. One curl command tells them apart, and the fix puts the practice facts in the server's own response.

By Gale Editorial · Updated 2026-09-15. Every figure cited to a dated source. How we write.

Why does an assistant say it cannot access a site that opens fine in a browser?

Because the browser and the fetcher read two different pages. The browser downloads the HTML, then runs the scripts a site builder attaches, and those scripts paint the name, the fee and the states served onto the screen. The fetcher an assistant sends downloads the same HTML and stops. In December 2024 Vercel and MERJ measured the crawlers crossing Vercel's network and reported that none of the major AI crawlers rendered JavaScript, OpenAI's, Anthropic's, Meta's and Perplexity's among them 1.

The crawlers do download the script files. ChatGPT's fetches were 11.5 percent JavaScript files and Claude's 23.8 percent, and neither executed what it downloaded 1. Googlebot did render, and Gemini rides on Googlebot's infrastructure, which is why one page can be readable to Google's AI features and blank to ChatGPT.

Three other things produce the same sentence: a robots.txt line that turns a search crawler away, a firewall rule that answers with a challenge page, and an assistant that never fetched at all. The test below sorts them. Which of the five sources an assistant leans on once it can read the site is the next question; a readable page is the entry fee to all five.

The raw-HTML test: one curl command

Open a terminal (Terminal on a Mac, PowerShell on Windows; both ship with curl), ask the server for the home page the way a fetcher does, and search the reply for the facts a patient wants. The command below fetches the page, follows redirects, identifies itself with the user-agent string OpenAI publishes for ChatGPT-User 2, and saves the reply to a file. Replace the address with your own.

``` curl -sL -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" "https://www.yourpractice.com/" -o raw.html ```

Open raw.html in a text editor, never a browser, and search it for four things: the practice name, the phone number, the states you see patients in, and the sentence that says who the practice is for. A served page has them inside ordinary paragraph and heading tags. A script-painted page is short: one empty container (a div with an id of root or app), a few script tags pointing at files, and none of the facts.

Run it a second time without the -A flag and its quoted string. If the two files differ, the server answers the fetcher's name differently from a browser's, which points at a bot rule, taken up below. Without a terminal, the view-source: prefix in Chrome or Edge shows the served HTML.

Reading the result: what the file tells you

The file you saved lands in one of four states, and each one names its own fix. The first two are settled by the file alone, and the third by the second run without the -A flag. A fifth case, no fetch at all, is settled by a server log, if the host exposes one, and is worth ruling out first.

What raw.html containsWhat it meansWhere the fix lives
The four facts, as textReadable; the trouble is a block, a missing fact, or no fetchThe log, robots.txt, then the facts
A near-empty shell of script tagsScript-painted; the fetcher read a blank pageServer-render the facts
A challenge page or a 403, and a fuller file without -AA firewall or bot rule turned the fetcher awayThe crawler allowlist and the host's bot settings
Nothing, after a long waitThe server did not answerHosting, then PageSpeed Insights

For the log: search the host's access log for ChatGPT-User or Claude-User at the minute you asked. OpenAI lists ChatGPT-User as the agent behind user actions in ChatGPT 2, and Anthropic says Claude may access websites under a Claude-User agent when a person asks it a question 3. No such line means no fetch happened and the sentence was about the assistant itself. Turn its web search on and ask again.

Blocked is different from blank: robots.txt and the firewall

A fetcher that was turned away reports much the same sentence as one that read a blank page, and the fixes are different. Two things turn fetchers away: a line in robots.txt, and a rule at the host or the content-delivery network that challenges whatever does not look like a browser. The second can arrive as a default setting nobody at the practice chose.

OpenAI runs its search crawler and its user fetcher under separate names. A Disallow for OAI-SearchBot keeps a site out of ChatGPT search answers, and OpenAI recommends allowing it; ChatGPT-User, the fetcher behind a pasted link, may not honor robots.txt and is not used to decide what appears in search 2. Anthropic's Claude-User honors robots.txt, and Anthropic says disabling it may reduce a site's visibility for user-directed web search 3. So a robots.txt line explains a practice absent from search answers better than a pasted link that fails. The crawler allowlist covers which names to allow.

The firewall case is newer. In July 2024 Cloudflare added a one-click block for AI scrapers and crawlers to every plan, the free one included, after finding AI bots on about 39 percent of the top million properties it fronts while 2.98 percent had blocked or challenged the requests 4. A web person who turned the switch on, or a host that does so by default, leaves a challenge page in raw.html. Cloudflare also notes that user agents are trivial for bot operators to change 4, so a rule keyed on something else may pass the test and stop the fetch. Ask the host which rule fired.

The raw-HTML fix: put the facts in the server's response

Move the facts a patient asks about out of the script and into the HTML the server sends. Builders differ in how, and three routes cover most of them: a plain static page, a text block above any script-driven widget, or a builder setting that server-renders text. Run the curl command again after each change; the fix is finished when raw.html contains the facts.

A static about page is the surest route. Most builders allow a plain text page with headings, and a page with no widget has nothing to paint by script. State the practice name, the credential, the states you see patients in, the fee, the plans you take and how to book, in sentences, with the date they were last checked. The practice website needs only this page to pass, and the five-page website is built from pages of this kind. Second, on a home page whose booking widget paints by script, add a text block above it that states the same facts. Third, some builders offer a setting that renders page text on the server; the name varies by product, and the web person knows it.

Bing's webmaster guidelines describe what its index, Copilot and grounding look for in these terms: key information that surfaces early, entities defined clearly, content that can be verified on the page itself, and an accurate title, meta description and heading structure 5. A page that passes the curl test is one where those things exist in the response. A JSON-LD block survives the test too, but it restates facts the page should state in words. The snippet controls and llms.txt are separate levers; neither puts a sentence into a response that has none.

Gale publishes a provider's gale.care page as server-delivered HTML, so the practice facts sit in the raw response a fetcher reads.

After the fix: speed, Search Console, IndexNow and the monthly re-run

A page that passes the curl test can still be slow to paint, and a fixed page has to be re-read by the engines before an answer changes. Four checks close the loop: PageSpeed Insights for the paint time, Search Console's URL Inspection to see how Google received the page and request a re-crawl, IndexNow for Bing and Copilot, and the same prompt a month later.

Google's Web Vitals set the loading target at a Largest Contentful Paint within 2.5 seconds, measured at the 75th percentile of page loads, and PageSpeed Insights reports it from field data collected by the Chrome User Experience Report 6. That is a threshold for people; no fetcher publishes the seconds it waits. But a home page whose largest paint arrives late because scripts run first is the page the curl test flagged, and the render-blocking scripts PageSpeed lists are the ones to defer.

In Search Console, URL Inspection's live test shows a screenshot and the HTML Google's tool received; compare that HTML with raw.html, since Google renders and the fetchers above do not. Then request indexing. Google says indexing typically takes a day or so, can take much longer, and that a request is no guarantee the page will appear in the index 7.

For Bing and Copilot, one crawling and indexing foundation serves both, and Bing's guidelines list IndexNow, XML sitemaps and crawlable links as the ways a URL is discovered 5. The IndexNow setup sends the fixed URL the day it changes. Then ask the assistant the same question next month, in the monthly prompt panel. A changed answer after a re-crawl is the only receipt this fix produces.

Common questions

Less often. Vercel's measurement found that Googlebot renders JavaScript and that Gemini uses Googlebot's infrastructure, so a script-painted page Google has indexed can still surface in its AI features. OpenAI's, Anthropic's, Meta's and Perplexity's fetchers read the raw response. A page that passes the curl test is readable to every one of them, which is why the fix is worth making even when Google already shows the site.

Yes. In Chrome or Edge, type view-source: directly in front of the address and press Enter, then search the page that opens for the practice name and the states served. That view shows the HTML as the server sent it, before any script ran. It sends a browser's name, so it cannot reveal a rule that fires on a fetcher's; for that, ask your host whether a bot-management or AI-crawler setting is switched on.

No. That file is a separate document at the site root, and it puts no words into the home page's response. A fetcher that skips your scripts reads the page a person asked about, and if that page is blank in the raw HTML, a second file does not fill it. Fix the page first; whether to add the file afterward is a separate question with its own answer.

A different one, and the test tells you which. If raw.html holds the practice name and the states but no fee, the page is readable and the fee is missing or painted by a script-driven pricing widget. Put the fee in plain text on the about page, with the date it was last checked, and run the command again until the number appears in the file.

Because a score built on Google's crawler measures the page after its scripts have run, and the assistant's fetcher reads it before they do. Send the agency raw.html and ask where the practice name appears in it. If it appears nowhere, the request is specific: a static about page, or a text block above the widget, with the facts in sentences. Ask for the curl output after the change as the receipt.

Usually no. OpenAI runs GPTBot, which collects content for its models, separately from OAI-SearchBot, which surfaces sites in ChatGPT search, and from ChatGPT-User, which fetches a page a person pasted. Each has its own robots.txt token. A Disallow aimed at GPTBot alone leaves the other two unaffected, so check the file for the other two names and run the curl test before undoing the training block.

Run your practice on Gale

The software is free. Gale earns one flat 3.5% all-in per paid transaction — only on transactions that actually pay. No subscription, no setup fee, no network cut.

Start or manage a practice →

References

  1. 1.Giacomo Zecchini, Alice Alexandra Moore, Malte Ubl, Ryan Siddle (Vercel and MERJ) (2024). The rise of the AI crawler. Vercel Blog. link ✓Vercel and MERJ's December 2024 measurement that none of the major AI crawlers (OpenAI, Anthropic, Meta, Perplexity) render JavaScript though they download script files (ChatGPT 11.5 percent, Claude 23.8 percent of fetches), that Googlebot and Gemini do render, and the recommendation to server-render content that must be readable by AI crawlers.
  2. 2.OpenAI (2026). Overview of OpenAI Crawlers. OpenAI Developer Platform documentation. link ✓OpenAI's own description of ChatGPT-User as the fetcher for user actions in ChatGPT, its published user-agent string, that robots.txt rules may not apply to it and that it is not used to determine whether content may appear in search; and that disallowing OAI-SearchBot keeps a site out of ChatGPT search answers while OpenAI recommends allowing it.
  3. 3.Anthropic (2026). Does Anthropic crawl data from the web, and how can site owners block the crawler?. Claude Help Center (Anthropic). link ✓Anthropic's statement that Claude may access websites under a Claude-User agent when a person asks it a question, that the bot honors robots.txt, and that disabling it may reduce a site's visibility for user-directed web search.
  4. 4.Alex Bocharov, Santiago Vargas, Adam Martinetti, Reid Tatoris, Carlos Azevedo (Cloudflare) (2024). Declare your AIndependence: block AI bots, scrapers and crawlers with a single click. Cloudflare Blog. link ✓Cloudflare's July 2024 one-click block for AI scrapers and crawlers on every plan including free, its June 2024 measurement that AI bots reached about 39 percent of the top million Cloudflare-fronted properties while 2.98 percent had blocked or challenged them, and its note that user agents are trivial for bot operators to change.
  5. 5.Microsoft Bing Webmaster Tools (2026). Webmaster Guidelines - Bing Webmaster Tools. Bing Webmaster Tools Help Center. link ✓Bing's statement that one crawling, indexing and ranking foundation serves Bing, Copilot and grounding; that a citable page surfaces key information early, defines entities, can be verified on the page and carries accurate title, meta and heading structure; and that IndexNow, XML sitemaps and crawlable links are the discovery routes.
  6. 6.Philip Walton, web.dev (Google) (2024). Web Vitals. web.dev Articles (published 4 May 2020, last updated 31 October 2024). link ✓The Largest Contentful Paint target of 2.5 seconds, the 75th-percentile measurement across mobile and desktop, and that PageSpeed Insights reports field data from the Chrome User Experience Report.
  7. 7.Google (2026). URL Inspection tool. Search Console Help (support.google.com). link ✓Google's stated turnaround for a Request Indexing submission in Search Console (typically a day or so, sometimes much longer) and that a request does not guarantee the page will appear in the index.

https://www.gale.care/for-providers/aeo-raw-html-readable · 7 sources. Competitor details are cited to dated public sources and maintained as they change; figures are estimates, not commitments. Synthetic demonstration.

← Back to for providers

Findability, by specialty

How practices like yours get found in local search and AI answers — the honest playbook, per specialty.

SEO for private practices · SEO for AI search / answer engines (all verticals)