AI Crawlers on WordPress: robots.txt, llms.txt and Cloudflare Settings Explained

Cover image for the developer guide to AI crawlers and robots.txt on WordPress: training vs search bots, Cloudflare, llms.txt

We went through a month of our own server logs to see how often AI crawlers read deviart.agency. The answer was interesting, but the bigger surprise was who else showed up. About a third of the requests that called themselves GPTBot, OAI-SearchBot or ChatGPT-User came from IP addresses OpenAI doesn’t use. Every single “PerplexityBot” was fake. Most of the fakes were scanners looking for .env files and private keys.

That changes how you should read any report about AI crawlers and robots.txt, including the ones you build for clients. This guide starts with what we found in our logs, then covers the parts you need on client sites: which bots to allow, where WordPress builds robots.txt, what Cloudflare and the host do before WordPress sees a request, and what llms.txt is worth. It’s the technical half of our AI search optimization for WordPress work.

A month of AI bots on our own site

The data: Apache access logs from deviart.agency, October 17 to November 16, 2026. The site runs on Cloudways behind Cloudflare. Requests that Varnish or Cloudflare answered from cache never reach these logs, so the totals are lower than the real traffic. The split between real and fake isn’t affected by that.

We checked each request’s IP against the lists that OpenAI, Perplexity, Google and Microsoft publish for their crawlers. A user agent is just a text field anyone can set; the IP is much harder to fake.

Terminal screenshot of verify_bots.py run on deviart.agency logs: real versus fake requests for GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Googlebot and Bingbot, and the config files the fake bots requested
The real output on our own logs. Every PerplexityBot request was fake.
Claimed botRequestsFrom the vendor’s IPsFrom somewhere else
GPTBot15411143
OAI-SearchBot1379938
ChatGPT-User592732
PerplexityBot36036
Perplexity-User45045
Googlebot383277106
Bingbot41537738
deviart.agency, 31 days. Anthropic and Meta don’t publish IP lists for their crawlers, so ClaudeBot and Meta’s bots aren’t in this table.

What we took from it:

  • The real OpenAI crawlers behave. GPTBot spent most of its visits on sitemap_index.xml (29 times) and then fetched pages from it. Real OAI-SearchBot got a 200 or a redirect on every request. Nothing on our side blocks them.
  • Real ChatGPT-User requests are the interesting ones. Each one is a person asking ChatGPT to open one of our pages. There were 27 in a month, mostly the homepage. Nine minutes after we published our AI search pages, ChatGPT-User fetched all three in the same second. Someone had pasted the links into ChatGPT, quite possibly one of us.
  • The fake ones aren’t researchers. Their top requests were /docker-compose.prod.yml, /ssl/server.key, /railway.json and /__vite_rsc_findSourceMapURL?filename=file:///root/.ssh/id_rsa. Someone has figured out that site owners are told to let AI bots through, and borrows their names.
  • Fake Googlebot is the old version of the same trick: 106 of 383 requests.

If you report “AI crawler visits” to a client from user agents alone, you’re reporting scanners too. On our site that would have inflated the OpenAI numbers by almost half.

How to check which bots are real

OpenAI, Perplexity, Google and Bing publish JSON files with their crawlers’ IP ranges. This script downloads them, reads an access log and splits each bot’s requests into real and fake. Standard library only, so it runs anywhere Python 3.8 or newer is installed:

#!/usr/bin/env python3
"""Split AI and search bot hits in an access log into real and fake.
Usage: python3 verify_bots.py access.log [more.log ...]"""
import collections, ipaddress, json, re, sys, urllib.request

RANGES = {
    "GPTBot": "https://openai.com/gptbot.json",
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
    "Googlebot": "https://developers.google.com/static/search/apis/ipranges/googlebot.json",
    "bingbot": "https://www.bing.com/toolbox/bingbot.json",
}
LINE = re.compile(r'^(\S+) .*?"[A-Z]+ (\S+)[^"]*" (\d{3}) .*"([^"]*)"\s*$')


def load(url):
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (bot IP check)"})
    data = json.load(urllib.request.urlopen(req, timeout=20))
    return [ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"]) for p in data["prefixes"]]


nets = {bot: load(url) for bot, url in RANGES.items()}
stats = collections.defaultdict(collections.Counter)
fake_paths = collections.Counter()

for path in sys.argv[1:]:
    with open(path, errors="ignore") as f:
        for line in f:
            m = LINE.match(line)
            if not m:
                continue
            ip, url, status, ua = m.groups()
            bot = next((b for b in nets if b.lower() in ua.lower()), None)
            if not bot:
                continue
            try:
                real = any(ipaddress.ip_address(ip) in n for n in nets[bot])
            except ValueError:
                continue
            stats[bot]["real" if real else "fake"] += 1
            if real:
                stats[bot]["real " + status] += 1
            else:
                fake_paths[url] += 1

print(f"{'bot':16} {'real':>6} {'fake':>6}   statuses of real hits")
for bot in nets:
    s = stats[bot]
    codes = ", ".join(f"{k[5:]}: {v}" for k, v in sorted(s.items()) if k.startswith("real "))
    print(f"{bot:16} {s['real']:6} {s['fake']:6}   {codes}")
print("\nMost requested URLs by fake bots:")
for url, n in fake_paths.most_common(10):
    print(f"{n:5}  {url}")

Before you run it: behind Cloudflare, the log needs the visitor’s real IP, not Cloudflare’s; Cloudways and most managed hosts restore it, but check that the first column of your log isn’t always a Cloudflare address. And on Cloudways the logs live in the application’s logs folder, rotated daily, so zcat -f the .gz files into one file first.

For bots without published IPs (ClaudeBot, Claude-SearchBot, Meta’s crawlers), look at what they request. Real ones fetch sitemaps and pages. Fake ones go after config files.

Training bots, search bots and user fetchers

The rule for client sites stays simple once you know which bot does what. AI companies run separate crawlers for training, for search, and for fetching a page a user asked about:

User agentOwnerJobBlocking it means
GPTBotOpenAITrainingContent not used to train OpenAI models
OAI-SearchBotOpenAISearchSite not shown in ChatGPT search answers
ChatGPT-UserOpenAIUser fetchOpenAI says robots.txt may not apply
ClaudeBotAnthropicTrainingFuture content left out of training
Claude-SearchBotAnthropicSearchContent not indexed for Claude’s search
Claude-UserAnthropicUser fetchClaude can’t open pages users ask about
PerplexityBotPerplexitySearchSite not shown in Perplexity answers
Perplexity-UserPerplexityUser fetchPerplexity says it generally ignores robots.txt
Google-ExtendedGoogleGemini training and groundingNo effect on Google Search or AI Overviews
Applebot-ExtendedAppleTrainingContent not used for Apple’s models
CCBotCommon CrawlOpen datasetLeft out of a dataset many models train on
Meta-ExternalAgentMetaTrainingContent not used for Meta’s AI
Checked against the OpenAI, Anthropic, Perplexity and Google docs on November 16, 2026. Google-Extended and Applebot-Extended are robots.txt tokens only and never appear in logs.
Diagram of three groups of AI crawlers: training crawlers like GPTBot and ClaudeBot, search crawlers like OAI-SearchBot and PerplexityBot, and user fetchers like ChatGPT-User
Block training crawlers if the client wants to. Keep search crawlers and user fetchers allowed.

Our default for agency clients: allow every search crawler and user fetcher, and block the training crawlers only if the client asks. Note the Google row. AI Overviews and AI Mode are part of Google Search and use Googlebot, so blocking Google-Extended doesn’t take a site out of them.

What 33 agency sites have in robots.txt

We also fetched robots.txt from 33 white label WordPress agency sites on November 16. Only one blocked any AI crawler, and it blocked training bots only. None blocked a search bot. Nine named AI bots in their file, almost all to allow them explicitly.

So on sites run by people who know WordPress, robots.txt is rarely what keeps AI search out. When a bot can’t get in, the cause is usually one layer up: Cloudflare or the host. The full results are in how to get a WordPress site cited by ChatGPT.

If the client does want to opt out of training, this is the block we use:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /

Several User-agent lines can share one group under RFC 9309. Don’t give search bots a group of their own unless you need one: a bot with its own group ignores User-agent: * completely, so any Disallow rules you rely on would have to be repeated there.

Where robots.txt comes from on WordPress

WordPress has no robots.txt file. It answers /robots.txt from PHP, until someone puts a real file in the site root; from then on the PHP version is never used. Yoast’s file editor (Yoast SEO → Tools → File editor) writes that physical file, so check with ls -la robots.txt in the WordPress root before editing anything.

Without a physical file, add rules with the robots_txt filter. We keep it in a mu-plugin so a theme change can’t remove it:

<?php
/**
 * Plugin Name: AI crawler rules
 * Description: Opts the site out of AI training crawlers. Search crawlers stay allowed.
 */

add_filter( 'robots_txt', function ( $output, $public ) {
	// Staging, or a site hidden from search engines: leave robots.txt alone.
	if ( ! $public ) {
		return $output;
	}

	$training_bots = array( 'GPTBot', 'ClaudeBot', 'Google-Extended', 'Applebot-Extended', 'CCBot', 'Meta-ExternalAgent' );

	$rules = '';
	foreach ( $training_bots as $bot ) {
		$rules .= "User-agent: {$bot}\n";
	}
	$rules .= "Disallow: /\n\n";

	return $rules . $output;
}, 20, 2 );

We tested it on WordPress 7.1 with Yoast SEO active. The training group comes first and Yoast’s block follows unchanged. Yoast swaps WordPress’s default Disallow: /wp-admin/ for an empty Disallow:, so don’t be surprised when that line is gone.

One thing that has changed and still catches people: Settings → Reading → “Discourage search engines” doesn’t touch robots.txt in current WordPress. It adds a noindex robots meta tag to every page. A staging copy pushed live with it ticked drops out of Google and Bing, and with them out of AI Overviews and Copilot, while robots.txt looks perfectly fine. Look in the page source.

Cloudflare and the host decide first

Two things in our own logs couldn’t be explained by robots.txt.

Bytespider, ByteDance’s crawler, got a 403 on 44 of its 89 requests. Our robots.txt allows it, Cloudflare passed the requests through, and there’s no rule for it in .htaccess. The block comes from the server stack, most likely the host’s own bot protection, and nobody at our end set it up. That’s fine for Bytespider, but it shows how a crawler can be blocked without anyone on the project knowing.

Perplexity is the open question. Not one request from Perplexity’s published IPs reached our server in 31 days, while 81 fakes did. Our robots.txt allows PerplexityBot. In August 2025 Cloudflare publicly accused Perplexity of crawling with undeclared user agents and dropped it from its verified bots list, so Cloudflare’s managed rules are our first suspect. We’re checking the AI Crawl Control log for the domain and will update this section with what we find.

Where to look on a client site behind Cloudflare:

  1. AI Crawl Control. It lists every AI crawler that tried to reach the domain, with Allow or Block per crawler. Since mid-2025 Cloudflare has offered blocking AI crawlers as a default when a domain is added, so check every site, including the ones you didn’t set up.
  2. Security settings, bot traffic. The AI crawler access policies and Bot Fight Mode. WAF skip rules can’t bypass Bot Fight Mode, so if it gets in a crawler’s way, the only fix is turning it off.
  3. Managed robots.txt. Cloudflare can prepend Content Signals (search, ai-input, ai-train) to your file. If the live robots.txt has lines nobody wrote, they come from here.
  4. WAF custom rules. Old rules that block user agents containing “bot” catch search crawlers too.

Then ask the host. On five of the 38 agency sites we tried to check, a plain request with a normal browser user agent got a 403 or timed out. Bot protection that strict may well stop real crawlers too.

Want us to run the same log and settings check on a client site? Our free AI readiness audit covers robots.txt, Cloudflare, real versus fake bot traffic and what’s in the HTML, as a report you can forward under your brand.

llms.txt: what our logs say

llms.txt is a 2024 proposal: a Markdown file at the site root that lists a site’s most useful pages for language models. Yoast generates one if you switch it on, and 22 of the 33 agency sites we checked have one.

Ours is a physical file Yoast wrote in November. Here’s who requested it in the 31 days of logs:

WhoRequests
Google Lighthouse (it now checks for the file)45
SEO tools: SEOJuice, Semrush, BuiltWith, Dataprovider and others13
Browsers, curl and our own checks20
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot0
deviart.agency static file logs, October 17 to November 16, 2026.

Not one AI crawler asked for it. Google says its AI features don’t need special files, and no AI search product has said it reads llms.txt when answering. Our logs agree.

We also noticed what Yoast put in it: Privacy Policy and Contact near the top, a handful of blog posts, and none of our service pages, because they’re a custom post type. If you keep the file, use Yoast’s option to pick pages by hand. We’d spend the time on the HTML and the schema first.

Checklist

  1. Run the bot check on a month of logs before reporting “AI traffic” to anyone.
  2. Find out whether the site serves a physical robots.txt or WordPress’s virtual one.
  3. Training bots blocked only if the client wants it; search bots allowed.
  4. No noindex in the page source on live.
  5. Cloudflare AI Crawl Control, Bot Fight Mode and WAF rules checked.
  6. Host bot protection checked if a real crawler gets 403s.
  7. llms.txt only if the client wants it, with pages picked by hand.

AI crawlers and robots.txt: FAQ

No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI treats the two settings separately. Block GPTBot to opt out of training and keep OAI-SearchBot allowed.

Check its IP against the ranges the vendor publishes. OpenAI, Perplexity, Google and Bing all publish JSON lists. On our own site about a third of the requests using OpenAI’s bot names came from other IPs, mostly scanners looking for config files.

No. AI Overviews and AI Mode are part of Google Search and use Googlebot. Google-Extended covers Gemini training and grounding in Gemini apps and Vertex AI. Keeping a page out of AI Overviews takes nosnippet, max-snippet or noindex, which also affect the normal result.

Not on our site. In 31 days of logs, our llms.txt was requested 78 times, mostly by Lighthouse and SEO tools, and never by GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or Googlebot.

OpenAI and Perplexity say about 24 hours. Anthropic and Google don’t give a fixed time. Test the live file, then check the logs a week later.