Guide

Which AI Bots to Allow on a Practice Site: A Copy-Paste robots.txt

Summary

Allow the agents that fetch a page to cite it (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Googlebot) and decide the training agents (GPTBot, ClaudeBot, Google-Extended) on your own terms; the two groups carry different names, and one line does not govern both. Before editing anything, request the home page as each agent with curl and read the status code, because a host's block-AI toggle can already be returning 403 to every one of them. Intake, portal, booking and upload paths go behind a login, with noindex, and out of every group.

By Gale Editorial · Updated 2026-09-15. Every figure cited to a dated source. How we write.

Should a practice block GPTBot?

Blocking GPTBot is a decision about training, and it changes nothing about whether ChatGPT can cite the practice. OpenAI runs GPTBot to collect content that may train its foundation models and a separate agent, OAI-SearchBot, to surface sites in ChatGPT search; each answers to its own line in robots.txt 1. So the question splits in two: whether the pages may train a model, which is yours to decide, and whether an assistant can fetch them to cite.

The same split runs through the other vendors. Anthropic names ClaudeBot for material that may contribute to training, and Claude-SearchBot and Claude-User for search and for fetching a page when a person asks; it says disabling the last two may reduce a site's visibility 2. Perplexity says PerplexityBot surfaces and links sites in its results, is not used to gather content for foundation models, and recommends allowing it 3. Google's training switch, Google-Extended, is a robots.txt token with no user-agent string of its own; it governs whether crawled content may train future Gemini models, and Google states that it does not affect a site's inclusion in Search or its ranking 4.

A practice that blocks the training agents and allows the citing fetchers gives up nothing it was counting on.

Which agents fetch to cite and which collect to train

Sort the tokens by what the vendor says each agent does, and the file writes itself. The citing fetchers and the training collectors come from the same companies and share nothing else, so the vendor's own page is the record for each row. The table carries only the agents whose vendor pages this article cites; other vendors publish pages of the same kind, and a line for their agents belongs in the file only once that page has been read.

TokenVendorWhat it doesrobots.txtIf disallowed
OAI-SearchBotOpenAISurfaces sites in ChatGPT searchRespectedNot shown in ChatGPT search answers 1
GPTBotOpenAICollects content that may train modelsRespectedSignals the content should not train 1
ChatGPT-UserOpenAIFetches a page for a user actionMay not applyNot used to decide Search appearance 1
Claude-SearchBotAnthropicIndexes to improve search resultsHonoredMay reduce visibility and accuracy in search 2
ClaudeBotAnthropicCollects content that may train modelsHonoredFuture material excluded from training sets 2
PerplexityBotPerplexitySurfaces and links sites in resultsHonoredPerplexity recommends allowing it 3
Perplexity-UserPerplexityFetches a page when a person asksGenerally ignoredFetch proceeds because a user asked 3
Google-ExtendedGoogleTraining and grounding token, no user agentHonoredNo effect on Search inclusion or ranking 4

ChatGPT-User and Perplexity-User fetch a page because a person asked for it, and both vendors say robots.txt may not apply, or is generally ignored, for that reason 13. A line naming them records a preference; stopping those fetches is firewall work.

How to tell whether the host is already blocking them

Ask the site, as each agent, and read the number that comes back. A request that identifies itself as OAI-SearchBot and receives 200 is being served; one that receives 403 is being refused somewhere between the network edge and the page, whatever robots.txt says. Cloudflare has offered a one-click block for AI scrapers and crawlers on every plan, including free, since July 2024 5, and site builders carry settings of the same kind under names that change.

Run this from any terminal with your own address, then repeat it with Claude-SearchBot, PerplexityBot, GPTBot, ClaudeBot, bingbot and Googlebot in place of the first agent:

``` curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://www.yourpractice.example/ curl -s -A "OAI-SearchBot" https://www.yourpractice.example/robots.txt ```

Write the code for each agent in a note with the date. A 200 means served. A 403 means refused, and the place to look is the host or the CDN, since the file cannot produce that code. The last line shows the robots.txt the fetcher is handed, and its status matters on its own: under RFC 9309 a crawler may access any resource when robots.txt comes back unavailable with a 4xx, and must assume complete disallow when it is unreachable with a server error 6. A missing file is an open door; a file that returns 500 is a silent total block.

In Cloudflare the toggle sits under Security, then Bots, labeled AI Scrapers and Crawlers 5. Cloudflare's June 2024 count found AI bots reaching about 39 percent of the top million properties it fronts, while 2.98 percent of those properties blocked or challenged them 5. A web person who switched it on in good faith may have taken the citing fetchers with it, which is what the curl lines are for.

The copy-paste robots.txt, in three groups

The file has three kinds of group: one for the citing fetchers, allowed on public pages; one for the training agents, set the way you decide; and one for every other agent, which is where the private paths live for the crawlers nobody named. Under RFC 9309 a crawler obeys the group that matches its product token and falls back to the star group only when none does 6, so the private paths repeat inside every named group.

``` # Citing fetchers: allowed on public pages. Replace the four prefixes with your own. User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: bingbot User-agent: Googlebot Disallow: /intake/ Disallow: /portal/ Disallow: /book/ Disallow: /upload/ Allow: /

# Training agents: your decision. Keep this group to block them, delete it to allow them. User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended Disallow: /

# Everyone else User-agent: * Disallow: /intake/ Disallow: /portal/ Disallow: /book/ Disallow: /upload/

Sitemap: https://www.yourpractice.example/sitemap.xml ```

Matching is case-insensitive against the product token, and where an allow and a disallow both match a URL the more specific rule, measured in octets, wins, with the allow winning a tie 6. The training group is the only opinion in the file. Anthropic says disallowing ClaudeBot signals that a site's future material should be excluded from its training sets 2, OpenAI says disallowing GPTBot indicates content should not train its models 1, and nothing on either page ties that choice to citations. Blocking Google-Extended blocks nothing that reaches Search; Google says the token neither affects inclusion nor serves as a ranking signal 4. The snippet controls, which govern what Google shows from a page it has crawled, are a separate set of tags and a separate decision.

Which paths stay out of every crawler

Intake forms, the patient portal, booking and document upload stay out of every group, and robots.txt is the weakest of the three locks on them. RFC 9309 states that its rules are not a form of access authorization 6, and Bing's documentation places a stronger control in the page itself: a noindex directive keeps a page out of Bing's index and out of generative-AI training 7. Authentication is the wall. The other two are requests.

Listing a path in robots.txt exposes it publicly and makes it discoverable, in the RFC's own words 6, so the file names a short prefix such as /portal/ and never the address of a specific form. And a page the practice wants cited must not carry the wrong tag: at Bing, noarchive means the page will not be linked in Chat and Copilot, and nocache limits Copilot to the URL, snippet and title 7. Read the head of the home page, the services page and the insurance page for those two words; a site-wide template carries its tags to every page.

The list of private paths is the same list the tracking-pixel problem is about; write it once, with a login column, and use it for both audits.

What the file cannot do, and what to re-check

The file is a request read by cooperative software, and its vocabulary is still being written. Cloudflare's engineers put it plainly in 2024: user agents are trivial for bot operators to change, so a user-agent rule cannot be relied on against a bot that chooses to spoof 5. The tokens above are the vendors' own as of September 2026, and each vendor's crawler page is the record when one changes.

Re-run the curl lines after any host, plan or CDN change, since each can put a new toggle between the page and the fetcher. Reread the vendor pages quarterly; Anthropic's alone names three agents, each with its own line 2. And keep the file in proportion: the practice website is one of the five sources an assistant can draw practice facts from, and robots.txt decides only whether that one is readable.

The effect shows up in the monthly prompt panel, where a silent 403 looks like an assistant that names the practice from a directory and never from its own site. Once each agent gets a 200, the raw-HTML test is the next check, because a served page still has to carry the facts in its HTML. For Bing, the IndexNow setup announces the changed page once the block is lifted. Whether llms.txt is worth adding is a separate question; none of the vendor pages cited here mentions it. Gale keeps a second first-party copy of the practice's public facts on its gale.care page and announces that page to Bing through IndexNow, so the facts stay reachable while a provider's own site is being unblocked.

Common questions

No. OpenAI runs GPTBot for content that may train its models and OAI-SearchBot for ChatGPT search, and each reads its own line in robots.txt. A site that disallows GPTBot and allows OAI-SearchBot can still be shown in ChatGPT search answers. A site that disallows OAI-SearchBot will not be shown there, whatever it does about GPTBot. Write the two lines separately and decide each on its own.

The request was refused before it reached the page, which robots.txt cannot do; a disallow in that file produces no status code at all. Look at the host's crawler or bot setting and at any CDN in front of the site, where a one-click block for AI crawlers may be switched on. Ask whoever manages the site whether a security setting changed in the past year, and record the date.

No. The Robots Exclusion Protocol states that its rules are not a form of access authorization, and it warns that listing a path in the file makes that path discoverable. A portal, an intake form, booking and document upload belong behind a login, with a noindex directive on the page. The robots.txt line is the third lock, naming only a short prefix, never a specific form's address.

Only if you want to control whether Google may use crawled pages to train future Gemini models or ground its answers. Google-Extended is a robots.txt token with no user-agent string; crawling happens under Google's existing user agents either way. Google says the token has no effect on inclusion in Search and is not a ranking signal, so leaving it out changes nothing about search, and adding a disallow changes only the training use.

Often enough that the file needs a date. Anthropic documents three agents and OpenAI four, and each vendor keeps a page that is the reliable record. Reread those pages quarterly, re-run the curl check after any change of host, plan or CDN, and add a token only after reading the vendor's description of what it does and whether it honors robots.txt.

Run your practice on Gale

The software is free. Gale earns one flat 3.5% all-in per paid transaction — only on transactions that actually pay. No subscription, no setup fee, no network cut.

Start or manage a practice →

References

  1. 1.OpenAI (2026). Overview of OpenAI Crawlers. OpenAI Developer Platform documentation. link ✓OpenAI's four tokens: GPTBot collects content that may train models, OAI-SearchBot surfaces sites in ChatGPT search and sites that disallow it are not shown in ChatGPT search answers, ChatGPT-User fetches for user actions and robots.txt may not apply to it.
  2. 2.Anthropic (2026). Does Anthropic crawl data from the web, and how can site owners block the crawler?. Claude Help Center (Anthropic). link ✓Anthropic's three agents: ClaudeBot for content that may train models, Claude-SearchBot and Claude-User for search and user-directed fetches; disabling the latter two may reduce visibility; all three honor robots.txt.
  3. 3.Perplexity (2026). Perplexity Crawlers. Perplexity Docs. link ✓PerplexityBot surfaces and links sites and is not used to collect content for foundation models, Perplexity recommends allowing it, and Perplexity-User generally ignores robots.txt because a user requested the fetch.
  4. 4.Google (2026). List of Google's common crawlers. Google for Developers (Crawling infrastructure documentation). link ✓Google-Extended is a robots.txt token with no separate user-agent string that governs training and grounding use of crawled content and does not affect inclusion in Google Search or ranking; Googlebot is listed as a token.
  5. 5.Alex Bocharov, Santiago Vargas, Adam Martinetti, Reid Tatoris, Carlos Azevedo (Cloudflare) (2024). Declare your AIndependence: block AI bots, scrapers and crawlers with a single click. Cloudflare Blog. link ✓Cloudflare's one-click AI Scrapers and Crawlers block, available on every plan including free since July 2024, its dashboard location, the June 2024 figures (about 39 percent reached, 2.98 percent blocking or challenging), and that user agents are trivial to change.
  6. 6.M. Koster, G. Illyes, H. Zeller, L. Sassman (2022). RFC 9309: Robots Exclusion Protocol. RFC Editor (IETF, Standards Track). link ✓The Robots Exclusion Protocol's rules: not a form of access authorization; a crawler obeys its matching group and falls back to the star group; case-insensitive token matching; most-specific match wins with allow on a tie; 4xx robots.txt permits access and 5xx means complete disallow; listing paths exposes them publicly.
  7. 7.Microsoft Bing Webmaster Tools (2026). Robots meta tags and attributes that Bing supports - Bing Webmaster Tools. Bing Webmaster Tools Help Center. link ✓At Bing, noindex keeps a page out of the index and out of generative-AI training; noarchive means the page will not be linked in Chat and Copilot; nocache limits Copilot to URL, snippet and title; the bingbot name scopes a directive to Bing.

https://www.gale.care/for-providers/aeo-robots-txt-ai-crawlers · 7 sources. Competitor details are cited to dated public sources and maintained as they change; figures are estimates, not commitments. Synthetic demonstration.

← Back to for providers

Findability, by specialty

How practices like yours get found in local search and AI answers — the honest playbook, per specialty.

SEO for private practices · SEO for AI search / answer engines (all verticals)