Is your WooCommerce host blocking ChatGPT’s crawlers? I tested 670 stores

I sent OpenAI’s three crawler names to 670 WooCommerce stores on 27 October. 159 of them, nearly one in four, refused GPTBot on three tries about a quarter of an hour apart, and 133 of those 159 have a robots.txt that doesn’t block it. The file says come in; the server says no. GPTBot only collects AI training data, so for most of these stores nothing is lost in ChatGPT itself. The bot that decides whether ChatGPT search can fetch your pages is OAI-SearchBot, and 29 stores refused that one. The same test on 199 Shopify stores: zero refusals. Here’s what was doing the refusing, how to run the same test on your store, and what I can’t see from where I sit.
What I tested
OpenAI publishes the exact user agent for each of its bots in its crawler documentation. I used those strings, word for word, plus a normal Chrome browser as the baseline. For each store I asked for the homepage four times, once per name, in random order and 1.5 seconds apart. 30 of the 700 WooCommerce stores I drew didn’t load for Chrome either, which leaves 670. The stores are a random draw from my own crawl corpus of public WooCommerce and Shopify shops, so they’re mostly small and mid-size stores, not big brands.
A single refusal can be a fluke, so I tested every refusal twice more, the last round 13 to 16 minutes after the first. On those rechecks I added two controls: a fake Googlebot and a made-up bot called ExampleBot. That’s how you tell “this server blocks GPTBot” apart from “this server blocks anything that looks like a bot”. The numbers in this post count only stores that refused all three times.
Why GPTBot is the one that gets blocked
OpenAI runs separate bots for separate jobs. GPTBot “is used to crawl content that may be used in training our generative AI foundation models”. OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features”, and OpenAI is blunt about what blocking it costs: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers”. ChatGPT-User is the one that visits a page live when someone asks ChatGPT about it.
So a block on GPTBot is a decision about AI training, not about selling in ChatGPT. Products reach ChatGPT shopping through a feed, not a crawler; I covered that split in what each OpenAI crawler actually does. Saying no to training is a perfectly reasonable choice. The problem in my data is that most of these stores never wrote that choice down anywhere they’d see it: their robots.txt doesn’t block GPTBot, but the server in front of the store does.
What was doing the refusing
The 159 GPTBot refusals fall into three groups.
95 stores block the GPTBot name and nothing else. The fake Googlebot and ExampleBot got the homepage; GPTBot got a 403. 65 of these sit behind Cloudflare, where 45 answered with the same short page: “Your request was blocked.” Cloudflare offers a switch for exactly this. Its AI bot policies let a site block bots by behaviour (Search, Agent, Training), and each policy blocks “Verified bots classified with that behavior, plus additional unverified bots”. My data can’t say which Cloudflare setting each store uses, or whether a custom firewall rule does the work. But the same Cloudflare network let all 199 Shopify stores answer GPTBot, so it’s a per-site setting, not a Cloudflare-wide rule.
37 stores on Hostinger answer GPTBot with “429 Too Many Requests” from its first request, so this isn’t a crawler hitting a real limit, and 26 of them refuse it robots.txt as well. That’s 37 of the 104 Hostinger-hosted stores in my sample. The same stores let OAI-SearchBot and ChatGPT-User straight in, and gave the fake Googlebot a different answer, a 403. 20 of the 37 responses carried Hostinger’s CDN header (x-hcdn-request-id); the other 17 came from the LiteSpeed web server. If GPTBot can’t fetch robots.txt, it can’t even read your own rules.
27 stores check any famous crawler name. They refused the fake Googlebot as well as GPTBot. 12 of them are on Automattic’s network and showed a page titled “Checking search engine crawler…” that needs JavaScript to pass. These systems may check where a request comes from, so the real GPTBot, arriving from OpenAI’s addresses, might get through. I can’t tell from outside.
What I can’t see from here
Every request in this test came from my server, which is not one of OpenAI’s addresses. That matters, and I won’t pretend otherwise. A rule that matches the name “GPTBot” catches the real bot too. A rule that checks the sender’s address might treat the real bot differently from my copy of it.
Hostinger adds one more twist. Its help page on server-level rate limits says it limits “traffic originating from specific network ranges, such as Meta, AWS, and Microsoft”, and that these limits “cannot be disabled for individual websites or hosting plans”. I looked up the owner of every address block in OpenAI’s published GPTBot and OAI-SearchBot lists: all 57 are registered to Microsoft. Hostinger doesn’t name OpenAI, so this is my inference, but it means OpenAI’s real crawlers can hit a 429 on Hostinger that no test from your own laptop will ever show. Only your server logs can.
The one that matters for ChatGPT search
29 stores refused OAI-SearchBot, about 4%. 16 of them blocked the name outright while letting the fake Googlebot and ExampleBot in, and 12 were the Automattic crawler check again. Only one of the 29 blocks OAI-SearchBot in its robots.txt. If you’re one of these stores, ChatGPT search can’t fetch your pages, and OpenAI’s advice is to allow the bot “in your site’s robots.txt file” and allow “requests from our published IP ranges”. A robots.txt that says yes doesn’t help if the firewall says no.
Shopify stores in my sample showed none of this: 0 refusals for all three bots across 199 stores. WooCommerce gives you a choice of host, CDN and security plugin, and every one of them can carry its own bot rule. That flexibility is the reason to run the test.
Run the test on your own store
From any terminal, ask for your homepage with each name and compare the status codes. The first line is the baseline; the second uses OpenAI’s GPTBot string from its documentation:
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://yourstore.com/curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://yourstore.com/
Repeat with OpenAI’s OAI-SearchBot string and with a fake Googlebot (Googlebot/2.1), then read the result off the figure above. Add -I to see the response headers: cf-ray means the answer came from Cloudflare, and x-hcdn-request-id means Hostinger’s CDN, which is where you look for the setting. If your host checks crawler names, a test from your computer can’t settle it; search your server’s access log for “GPTBot” and “OAI-SearchBot” and look at the status codes they got.
Then decide on purpose. If you don’t want your content used for training, say so in robots.txt, where it’s visible and reversible. Hostinger’s own guide to AI crawlers tells you to do it there. If you do want ChatGPT search to see you, make sure nothing between the internet and your store is quietly refusing OAI-SearchBot.