Do SaaS Sites Block AI Crawlers? We Checked 192
Short answer: almost none of them. On November 20, 2026 we fetched the robots.txt files of 218 B2B SaaS and marketing-technology websites; 192 returned a readable file. Only 15 of those 192 (8%) fully disallow any AI crawler, and only one blocks a crawler that powers AI search answers. Meanwhile 129 (67%) already publish an llms.txt file, a format no major AI provider has said it reads.
The takeaway for marketers: if your brand is missing from ChatGPT, Gemini or Perplexity answers, a robots.txt block is rarely the reason. Check it anyway, because when it is the reason, it is the easiest fix you will ever ship.
Published November 21, 2026. Raw data: CSV of per-site results and the domain list.
Key findings
Finding | Sites | Share of 192 |
|---|---|---|
Serve a file at /llms.txt | 129 | 67% |
Name at least one AI crawler in robots.txt | 51 | 27% |
Fully disallow at least one AI crawler | 15 | 8% |
Fully disallow GPTBot (OpenAI training) | 6 | 3% |
Fully disallow an AI search or user agent | 1 | 0.5% |
"Fully disallow" means the rule group that applies to that crawler contains Disallow: / with no Allow that reopens the site, following the matching rules in the robots.txt standard (RFC 9309).

Training crawlers get blocked; search crawlers almost never do
AI companies now run separate crawlers for separate jobs, and they document which is which:
- Training crawlers collect pages to train models: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (a control token for Gemini training, not a separate crawler), CCBot (Common Crawl), Bytespider (ByteDance), Applebot-Extended and meta-externalagent.
- Search and user agents fetch pages to answer a question right now: OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot.
See OpenAI's crawler documentation, Anthropic's, Perplexity's and Google's list of common crawlers.
All 15 sites that block anything block at least one training crawler. Bytespider (13 sites) and CCBot (8) are the most blocked; GPTBot is blocked by 6: Canva, Figma, Harvest, Loom, Medium and Vidyard. Only one site, Harvest, also blocks an AI search agent. In other words, SaaS companies that block AI crawlers are mostly opting out of training, not out of being recommended.
That distinction matters. Blocking GPTBot does not stop ChatGPT search from reading and citing your pages, because OpenAI's search uses OAI-SearchBot, and OpenAI states the two are controlled separately.
Most sites have no explicit AI policy
141 of the 192 sites (73%) never mention an AI crawler by name. Those crawlers fall back to the User-agent: * group, which usually allows the whole site apart from a few admin or search paths. That is a fine default for most marketing sites. It is still worth making the choice on purpose: name the search agents you want, and decide separately about training.
llms.txt is popular but unproven
129 sites (67%) return a text file at /llms.txt, and 121 of those follow the proposed format, which starts with a Markdown heading (llmstxt.org). Adoption this high in a marketing-tech sample is not surprising; several SEO plugins now generate the file automatically, and one of the files we fetched says so in its first line.
We found no documentation from OpenAI, Anthropic, Google or Perplexity saying their crawlers use llms.txt to decide what to read or cite. It costs little to publish one, but don't expect it to change your AI visibility on its own.
Some sites refuse AI crawlers at the edge
Six large sites (Salesforce, Zoho, Xero, WordPress.com, Rippling and ChurnZero) served their homepage normally to a browser user agent, but returned 403 or 503 to the same request carrying a GPTBot, ClaudeBot or PerplexityBot user agent, even though their robots.txt does not disallow those crawlers.
There is an important caveat: our requests came from our own network, not from OpenAI's, Anthropic's or Perplexity's published IP ranges. Bot-management services commonly block traffic that claims to be a known crawler but comes from the wrong network (Cloudflare's verified bots). So these sites may only be blocking impostors. What the result does show is that robots.txt is not the only gate: a firewall or bot-management rule can refuse a crawler that robots.txt allows. If you use one, check its AI crawler settings, not just robots.txt.
What to do with this
- Read your own robots.txt. Search it for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended. If you never named them, they inherit your
*rules. - Separate training from search. If you want to opt out of training, block the training crawlers by name and leave the search agents allowed. Blocking everything also removes you from AI search answers.
- Check your edge. If you use Cloudflare, Akamai or another bot manager, look at its AI crawler settings. A rule there overrides a friendly robots.txt.
- Make the content readable without JavaScript. Several AI crawlers do not run JavaScript, so key facts (what you sell, pricing, who it's for) should be in the HTML.
- Then work on being cited. Access is the floor, not the ceiling. Our AI citation playbook covers what actually earns mentions, and the GEO guide explains the basics.
Septra's free GEO scan checks AI crawler access in your robots.txt and whether your homepage text is readable without JavaScript, then asks ChatGPT, Gemini and Perplexity real buyer questions in your category. If you'd rather keep checking by hand, see our comparison of GEO tools.
Method
- Sample: 218 B2B SaaS, marketing-technology and adjacent software companies, chosen by us from well-known vendors in CRM, email, SEO, analytics, ads, project management, finance and AI-agent categories. It is a convenience sample of established brands, not a random sample of the web, so the results describe that group only.
- Collection: on November 20, 2026 (UTC) we requested
https://<domain>/robots.txtand/llms.txtonce each with a standard browser user agent, following redirects. 192 domains returned a valid robots.txt; the other 26 returned an error (mostly 403 from bot protection) and are excluded from the robots.txt percentages. - Parsing: rules were grouped by user agent as defined in RFC 9309. A crawler with its own group uses that group; otherwise it falls back to
*. We report a crawler as fully disallowed only when its group disallows/without an allow rule for the root. - Edge test: we requested each homepage once with a browser user agent and once each with GPTBot, ClaudeBot and PerplexityBot user agent strings, from our own network.
- Limits: robots.txt is a request, not an enforcement mechanism; crawlers decide whether to honor it. A single crawl is a snapshot, and sites change their rules.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT answers?
No, not by itself. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler used for ChatGPT search results. Blocking GPTBot opts you out of training; to stay out of ChatGPT search you would also have to block OAI-SearchBot.
Is Google-Extended a crawler?
No. Google documents Google-Extended as a robots.txt token that controls whether content crawled by Googlebot can be used for Gemini model training. Blocking it does not affect Google Search or AI Overviews eligibility.
Should I publish an llms.txt file?
It is cheap and harmless, so there is little reason not to. Just don't treat it as a GEO strategy: we found no major AI provider documenting that it reads the file.
Why are some percentages based on 192 sites and not 218?
26 sites refused our robots.txt request or returned an error, so we couldn't read their rules. Counting them as "allowed" or "blocked" would be a guess.
Sources
- IETF, RFC 9309, Robots Exclusion Protocol: rfc-editor.org
- OpenAI, Overview of OpenAI crawlers: platform.openai.com/docs/bots
- Anthropic, Does Anthropic crawl data from the web: support.anthropic.com
- Perplexity, Perplexity crawlers: docs.perplexity.ai
- Google Search Central, Google's common crawlers: developers.google.com
- The llms.txt proposal: llmstxt.org
- Cloudflare, Verified bots: developers.cloudflare.com

Founder of Septra. Previously engineered core compute and auto scaling systems at AWS and Stripe.
Set your business goal. Put your AI team to work.
Seven autonomous AI employees plan, execute, and deliver results on a predictable weekly cadence.
Build my goal plan


