domains / families / security / llm-crawler-access-check

llm-crawler-access-check

separate

Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers from the search crawlers that actually control citations.

When to use

• Read `https://<domain>/robots.txt` and nothing else. Do not crawl the site, do not attempt to access disallowed paths, and do not bypass any access control. This is a read of one public file.

When not to use

• Crawl the site
• Access disallowed paths
• Bypass access control
• Recommend whether to block AI crawlers
• Check server-side WAF blocking
• Check CDN bot rules
• Check IP reputation
• Check Cloudflare bot management

Skill page

Source