domains / families / marketing / robots

robots

separate

When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.

When to use

• configure robots.txt
• audit robots.txt
• optimize robots.txt
• block crawlers
• AI crawler strategy
• crawl directives
• user-agent rules
• fix robots.txt

When not to use

• indexing
• page-level exclusions
• noindex directives

Skill page

Source