Is Your Store Accidentally Blocking AI Crawlers?

Here's a quiet little problem that bites a lot of store owners. You've spent months getting your product pages right. Reviews are coming in. People love the stuff. But when a shopper opens ChatGPT and types "best dark roast coffee for a French press," your brand never comes up. Not once. Your competitor down the street does, though. Every time.

It's easy to assume the AI just doesn't know you exist yet, or that you need more press, or better SEO. Sometimes that's true. But there's a much dumber and much more common reason your store might be missing: you're blocking the AI crawlers from reading your site at all. And usually nobody did it on purpose.

What's actually happening here

ChatGPT, Perplexity, Gemini, and Google AI Overviews don't pull recommendations out of thin air. To know your store exists, their systems first have to crawl your pages, the same way Google's search bot has done for years. Each AI company runs its own bot to do this. The big ones you care about are GPTBot (OpenAI, the company behind ChatGPT), PerplexityBot (Perplexity), Google-Extended (which controls whether Google can use your content for Gemini and AI Overviews), and ClaudeBot (Anthropic).

Every website has a small text file called robots.txt that tells these bots where they can and can't go. It lives at one fixed spot: your domain plus /robots.txt. Think of it as the bouncer at the door. If that file says "GPTBot, you're not coming in," then OpenAI's crawler turns around and leaves, and ChatGPT never learns a thing about your products.

The frustrating part is how this rule shows up. A developer you hired two years ago might have added it. A "privacy" or "SEO" plugin might have switched it on by default during an update. Some hosting platforms and themes ship with AI bots blocked out of the box because, for a while, blocking AI scraping was the trendy thing to do. You never touched it. You probably don't even know it's there. Meanwhile your visibility in AI tools just quietly went to zero.

How to check your own store in two minutes

You don't need a developer for this part. Open a browser tab and go to your store's address followed by /robots.txt. So if your shop lives at brightbeancoffee.com, you'd visit brightbeancoffee.com/robots.txt. You'll see a plain page of text. It might be short, it might be long. Don't panic at the formatting.

Now use your browser's find function (Ctrl+F on Windows, Cmd+F on a Mac) and search the page for these names one at a time:

• GPTBot
• PerplexityBot
• Google-Extended
• ClaudeBot

For each one you find, look at the line right under it. This is the whole game. A blocking rule looks like this:

User-agent: GPTBot
Disallow: /

That "Disallow: /" with a lone forward slash means "you are not allowed anywhere on this site." That's the line shutting the door. If you see Disallow: / under any of those bot names, that crawler is locked out. If instead you see "Allow: /" or no Disallow line at all, or the bot name doesn't appear in the file, you're generally fine for that one.

One more thing to watch for. Some files have a catch-all section that starts with "User-agent: *" (the star means "every bot"). If that block has "Disallow: /" under it, you may be blocking everything, search engines included. That's a bigger problem and worth a closer look.

How to let the AI crawlers back in

The fix depends on how your robots.txt got that rule in the first place, so start by figuring out the source.

If a plugin or app added it

This is the most common case for WooCommerce and Shopify stores. Go through your installed apps and plugins and look for anything about SEO, privacy, "AI bot blocking," or "content protection." A lot of these tools have a simple toggle labeled something like "Block AI crawlers" or "Allow AI training bots." Flip it to allow them. Many WordPress SEO plugins manage robots.txt directly inside their settings, so you can often edit the rule there without touching code.

If it's in the file itself

On a self-hosted WooCommerce site, robots.txt may be a real file you can edit through your host's file manager or an SEO plugin. To allow a bot, you either remove its blocking block entirely, or change the Disallow line. Allowing GPTBot looks like this:

User-agent: GPTBot
Allow: /

Shopify is a bit different. It generates robots.txt automatically, and historically it didn't block AI bots by default, so if yours is blocked it's almost certainly from a theme tweak or an app. Shopify lets you customize the file through a template, so check there or with whoever set up your theme.

After you make a change, go back and reload your /robots.txt page to confirm the blocking line is actually gone. The bots won't recrawl you the same day, so give it a few weeks before you expect to show up in answers.

Be honest with yourself about the tradeoff

I'm not going to pretend blocking these bots is always wrong. Some owners do it on purpose, and that's a real choice. The argument goes: AI companies are using your content, your photos, your product descriptions, sometimes to train models, and they're not paying you for it. If that genuinely bothers you, blocking is a fair stance to take.

But you have to be clear about what you're trading. If you block GPTBot, you're not just opting out of "AI training." You're also opting out of being recommended when someone asks ChatGPT what dog food to buy for a senior Labrador, or which fragrance-free moisturizer is good for sensitive skin. For most small stores trying to get found, that's a bad trade. The shoppers asking those questions have their wallets out. You want to be in the room when the AI answers. My honest take: unless you have a specific reason to keep them out, let them in.

Don't stop at robots.txt

Clearing a blocking rule is step one, and it's the easiest win because it's a yes-or-no fix. But it's not the whole story. You can have a perfectly open robots.txt and still get ignored by AI because your product pages are thin, your store never gets mentioned on other sites the AI trusts, or a competitor simply has a stronger footprint. Crawler access gets you in the door. It doesn't guarantee the recommendation.

This is the gap Zerogaps was built to close. The free audit checks the actual questions shoppers ask the AI tools about your category, shows you whether you come up, names the competitors getting recommended instead, and yes, it flags when your store is blocking AI crawlers so you're not guessing about that robots.txt line. If you want to see where you stand right now, you can run a free AI visibility audit and start from real data instead of a hunch. From there you can track it over time and watch whether your fixes actually move the needle.

Go check your robots.txt first, though. Seriously, do it before you read another word about AI strategy. It takes two minutes, and if you find a "Disallow: /" sitting under GPTBot, you may have just discovered why those AI answers have been quietly skipping you this whole time.