robots.txt Wildcard That Blocks AI Citations

The robots.txt file is two lines on most property management websites: allow everything, point to the sitemap. When something goes wrong with AI citations, robots.txt is rarely the first place anyone looks. It should be.

A single wildcard rule, one that looks completely reasonable, can silently block every AI citation crawler on the web from accessing your pages. ChatGPT, Perplexity, Claude, Google AI Mode: all of them, at once, without a single error message.

How does a robots.txt wildcard rule work?

The User-agent: * directive is a catch-all: any rule under it applies to every crawler that doesn’t have its own explicit rule elsewhere in the file. Most sites use it simply, User-agent: * followed by Allow: /, which lets every crawler reach every page.

The problem starts when a site adds a specific crawler block, usually GPTBot, using a pattern that looks like this:

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

This is the intended structure: block GPTBot specifically, allow everything else. And it works, for a plain crawl-all-or-block-one setup.

But many sites that have been through a “block AI training crawlers” guide don’t stop there. They end up with something like:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Disallow: /

That second configuration blocks everything by default, then allows Googlebot back in. The intent is usually to limit crawling to Google only. The effect is that OAI-SearchBot, ClaudeBot, PerplexityBot, and Googlebot-Extended are all blocked, the wildcard Disallow: / catches them all, and they don’t have explicit allow rules.

The site looks fine in Google Search. AI citations stop without any visible signal.

Which AI crawlers does a restrictive wildcard block?

Four crawlers do the work of getting you cited, and a Googlebot-only configuration blocks all four:

• OAI-SearchBot, ChatGPT live-search citations (distinct from GPTBot, which is training only)
• ClaudeBot, Claude web search citations
• PerplexityBot, Perplexity inline citations
• Googlebot-Extended, Google AI Mode

None of these are Googlebot. Allowing Googlebot does nothing for any of them, because each identifies itself under its own name and falls through to the wildcard when no rule matches it.

Why doesn’t Search Console warn you about a blocked AI crawler?

Because Google Search Console only reports on Google’s own access. If Googlebot can reach your site, the Coverage report is clean. There is no equivalent report for OAI-SearchBot, no dashboard showing “ChatGPT attempted to access 47 pages and was blocked on all of them.”

The block is silent in every direction that would normally catch it. Rankings don’t change. Traffic doesn’t drop. The only signal is that AI citations don’t appear, and an absence is easy to attribute to something else entirely, usually to the content rather than to access.

How do you check whether your robots.txt blocks AI citation crawlers?

Open your file at yourdomain.com/robots.txt and work through three questions. The whole check takes about five minutes and needs no tooling.

Is there a wildcard rule that disallows everything?

Look for User-agent: * followed by Disallow: /. If it’s there, every crawler not explicitly named with its own rule is blocked, regardless of what else the file says further down. This is the single most common cause, and it usually arrives with a well-intentioned edit meant to keep training crawlers out. The citation crawlers need their own Allow: / rules to escape it.

Are the citation crawlers named anywhere in the file?

Search the file for OAI-SearchBot, ClaudeBot, PerplexityBot, and Googlebot-Extended. Most robots.txt files I open in audits name GPTBot and nothing else, because GPTBot is the crawler the “block AI” guides were written about. If the four above are absent and the wildcard is restrictive, they inherit the block by default and nothing in the file records that decision.

Does rule order put the specific crawlers above the wildcard?

More-specific rules take precedence, but only when the crawler matches its own User-agent block. The wildcard applies whenever no specific block matches. So a citation crawler with no rule of its own will always fall through to User-agent: *, no matter where in the file that wildcard sits. Order matters less than presence: what protects a crawler is having a rule at all.

How do you fix a robots.txt that blocks AI citation crawlers?

Add an explicit allow rule for each citation crawler. Even when the wildcard already allows everything, naming them makes the intent readable and stops a future edit from recapturing them by accident:

User-agent: *
Allow: /

# AI citation crawlers, explicitly allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot-Extended
Allow: /

# Training crawlers, block separately if desired
User-agent: GPTBot
Disallow: /

This allows all standard crawling, whitelists the citation crawlers by name, and keeps the training-crawler decision as a separate, deliberate line rather than a side effect.

Does allowing the crawlers guarantee you get cited?

No. robots.txt controls access, not selection. Opening the door is necessary and it is not sufficient.

Once a crawler can reach the page, three other things decide whether it gets cited: the page has to rank in the top results for the query, the answer has to be structured so it can be lifted out cleanly, and the entity signals have to be complete enough that the engine can attribute the citation to your firm rather than to a directory listing you appear in. None of those are solved by adding an llms.txt file, which is the move most often suggested in their place. Fixing the access problem is the cheapest of the four, which is the only reason to do it first.

Technical audit of AI crawler access is included in every site review I run. If you’d like to know which crawlers can reach your pages, and what’s preventing AI engines from citing you, a free audit shows you. The audit is the next step.