# --- AI crawlers: ALLOW (customer discovery surfaces) --- User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Amazonbot Allow: / User-agent: Google-Extended Allow: / # --- Low-value AI scraper: block (no customer discovery) --- User-agent: Bytespider Disallow: / # --- SEO competitor-intel scrapers: block --- User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: spyfu Disallow: / User-agent: similarweb Disallow: / # --- Everything else: allow --- User-agent: * Disallow: