Decision: robots.txt AI-crawler policy (Cloudflare-managed) #9633

Medium#8 from the 2026-07-08 audit, re-verified live 2026-07-09: https://anyplot.ai/robots.txt still serves Cloudflare Managed Content with Disallow: / for ClaudeBot, GPTBot, CCBot, Google-Extended, Amazonbot, Applebot-Extended, Bytespider, meta-externalagent.

This overrides the site's own use=reference content signal and blocks all AI retrieval/citation crawlers. llms.txt (#9618) helps user-directed agent fetches but does not lift robots blocks for compliant crawlers.

Maintainer decision needed (Cloudflare dashboard action, not repo-fixable):

• Allow retrieval/citation crawlers (ClaudeBot, GPTBot retrieval, etc.) while optionally keeping training-specific blocks (Google-Extended, CCBot)?
• Or keep the full block as a deliberate policy?

Either way, document the decision in the repo (docs/reference/seo.md) so the next audit doesn't re-flag it.

Audit: agentic/audits/2026-07-08-product-ux.md (Medium#8).