Is your WooCommerce store blocking Claude, Perplexity and Meta AI? I tested 634 stores

On 10 November I asked 634 live WooCommerce stores for their homepage under ten different AI crawler names from OpenAI, Anthropic, Perplexity and Meta. 193 refused at least one name on three separate tries. Most of that is training bots: GPTBot 146, ClaudeBot 128, Meta-ExternalAgent 109. The number worth your time is PerplexityBot. Perplexity says it’s a search bot that doesn’t collect training data, and 63 stores refused it, more than twice as many as refused OpenAI’s search bot. 60 of those 63 have a robots.txt that doesn’t block it. If you have a blanket “block AI” setting on, it may be turning away a search engine you never meant to turn away.
What I tested
This is the same random draw of 700 WooCommerce stores I used for the GPTBot test in October. 66 didn’t load for a normal Chrome browser this time, which leaves 634. For each store I asked for the homepage eleven times, once as Chrome and once under each AI name, in random order and 1.5 seconds apart.
Any refusal got tested twice more, with the last round 33 to 55 minutes after the first. Those rechecks added two controls: a fake Googlebot and a made-up bot called ExampleBot. Every number below counts only stores that refused a name all three times.
Two limits on the names. Anthropic publishes its crawler names but not full user-agent strings, so for Claude I put the documented name inside a standard wrapper. Meta’s I used in the short form its documentation prints. I left Bingbot out because Bing’s crawler page wouldn’t load for me, so this post says nothing about Copilot.
Three jobs, three very different refusal rates
These names do different work, and the vendors say so. Anthropic says restricting ClaudeBot signals that “future materials should be excluded from our AI model training datasets”. Blocking Claude-SearchBot “may reduce your site’s visibility and accuracy in user search results”, and blocking Claude-User “may reduce your site’s visibility for user-directed web search”.
The stores mostly drew that line, whether or not they meant to. 114 stores refused ClaudeBot but not Claude-User. Not one refused Claude-User while allowing ClaudeBot. The three live-fetch names I tested stay small: 30, 21 and 14. And 12 of those 14 Claude-User refusals are the same 12 stores on Automattic’s network, which refused all ten names and the fake Googlebot too. That’s a crawler check rather than an AI policy, and it may let the real bots through. I can’t tell from outside.
Refusing a training bot is a reasonable choice, and OpenAI and Anthropic both describe those bots separately from their search bots. Meta’s description is broader: Meta-ExternalAgent covers “training foundation AI models” or improving products “by indexing content directly”, so that one isn’t purely about training. The bigger surprise is one row down.
Why PerplexityBot stands out
Perplexity’s documentation says PerplexityBot “is designed to surface and link websites in search results on Perplexity” and is explicit that “It is not used to crawl content for AI foundation models.” It also says: “To ensure your site appears in search results, we recommend allowing” it. How products reach Perplexity’s shopping answers is a separate question, which I covered in this post on its merchant program.
Yet 63 stores refused it, against 29 for OAI-SearchBot and 23 for Claude-SearchBot. 34 of the 63 are behind Cloudflare, and 33 of those 34 also refuse at least one training bot. 25 of those 34 answered with the same short page: “Your request was blocked.”
My inference, and it is only that: these stores switched on something aimed at AI training, and PerplexityBot got swept up with it. A store can also block Perplexity on purpose. But only 3 of the 63 say so in robots.txt, which is the place you’d expect a deliberate decision to live.
Who was doing the refusing
Cloudflare. 214 of the 634 stores answered from Cloudflare’s network. 80 of those refused ClaudeBot, 71 GPTBot, 34 PerplexityBot and 31 Meta-ExternalAgent. On 4 November Cloudflare announced that its setting is now called “AI Crawler and Agent access policies”, with per-agent choices, including “Block on pages with ads”. Its docs say each policy applies to “Verified bots with that behavior and matching unverified bots”, and that for new domains, the three behaviours “use Allow (do not block) by default”. The presets are “available as managed presets you can act on across all plans”, though the announcement doesn’t say which plans get the per-agent overrides. Nor can I see which setting each store used.
Hostinger. 32 of the 104 Hostinger stores refused GPTBot and 28 refused Meta-ExternalAgent, nearly all with “429 Too Many Requests” on the first request. ClaudeBot: 1. Every Perplexity name: 0. So this isn’t a general AI rule. It’s those two names. Separately, Hostinger’s rate-limit help page says it limits “traffic originating from specific network ranges, such as Meta, AWS, and Microsoft”. That page is about network addresses and my server is on none of those networks, so the real Meta crawler may face both rules.
You can’t block Google-Extended at the server, and that’s fine
People often lump Google-Extended in with these bots. It doesn’t work the same way. Google’s crawler docs say: “Google-Extended doesn’t have a separate HTTP request user agent string.” It’s a robots.txt setting and nothing else, so a firewall rule can’t see it. 45 of the 612 stores with a readable robots.txt block it by name, and Google says doing so “does not impact a site’s inclusion in Google Search”.
These settings don’t drift
Of the 159 stores that refused GPTBot in October, 155 still loaded and 142 still refused it, with 4 new ones. For OAI-SearchBot, 27 of the 29 still refused, with 2 new. Two weeks isn’t long, but these look like standing settings rather than temporary rate limits. If yours is wrong, it probably stays wrong until someone looks.
What I can’t see from here
Every request came from my server, not from these companies’ addresses. A rule that matches the name catches the real bot too, but a rule that checks the sender’s address might treat the real bot differently from my copy of it. Meta’s short strings carry one more risk: a server that refuses anything that doesn’t look like a browser would refuse them regardless of name. That can explain at most the 15 stores that refused all three Meta names; 74 stores refused Meta-ExternalAgent alone.
Only your server logs can show what the real crawlers got. Search them for the names in Figure 1.
Test your own store
Run these from a terminal and compare the status codes. The first is your baseline:
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://yourstore.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://yourstore.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; ClaudeBot/1.0)" https://yourstore.com/
Swap in any name from Figure 1. Add -I to see headers: cf-ray means Cloudflare answered, and x-hcdn-request-id means Hostinger’s CDN, which tells you where to look for the setting.
Then decide on purpose. Refuse training bots if you like, and say so in robots.txt, where you can see it and reverse it. Check separately that the search names, especially PerplexityBot, are getting through if you want to be found there. For a wider look at what AI tools can read on your store, there’s a free scan at pollen.elytron.in.