How to Check if AI Crawlers Can Access Your Website

ChatGPT can only recommend a website it can reach. Before any AEO work, confirm that AI crawlers are allowed in. The check takes about ten minutes, and if your site runs through Cloudflare there is a date to know about: from September 15, 2026, Cloudflare applies new default rules for AI crawlers, so it is worth seeing what your account is set to.

Know which bots matter

OpenAI runs four crawlers, and each one does a different job:

Anthropic (ClaudeBot) and Perplexity (PerplexityBot) work the same way, and every check below covers them too.

Step 1: read your robots.txt

Open yourdomain.com/robots.txt in a browser and look for Disallow: / under any of the user agents above. A safe baseline for AI visibility looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

GPTBot is a separate decision. Blocking it keeps your content out of training data and has no effect on whether ChatGPT search shows you.

Step 2: check Cloudflare

If your DNS or CDN runs through Cloudflare, open your domain, go to Security, then Settings, and find the AI traffic controls. Cloudflare sorts AI crawlers into three groups: Search, Agent and Training. From September 15, 2026, domains newly added to Cloudflare get Training and Agent blocked by default on pages that display ads, while Search stays allowed. Existing domains keep whatever they have set, which is exactly why you should look.

Keep Search allowed. And be careful with Training: Cloudflare states that crawlers with more than one purpose, such as Bingbot and Googlebot, get blocked too when you block the Training group, because the most restrictive rule wins. Blocking training can take you out of Bing, and out of ChatGPT search with it.

Step 3: check your host and firewall

Some hosting providers and security plugins block bots at the server level, before robots.txt is even read. Check your WAF rules and any security plugin's bot list for the user agents above.

Step 4: confirm with a real request

Ask your developer to run one line, or run it yourself in a terminal:

curl -A "OAI-SearchBot" -I https://yourdomain.com

A 200 response means the door is open. A 403 means something in the stack still blocks the bot, and the usual suspects are the firewall or the CDN.

Once the door is open, every prompt where ChatGPT currently recommends a competitor is a prompt where it can recommend you instead. To see how often that happens today, run the free AI visibility and SEO audit on your domain.

No sign-up needed: https://www.contentmonk.io/ai-visibility-audit