# robots.txt for https://www.erikkangas.com/ # Everything here is public and intended to be crawled, indexed, and cited. # # Content-Signal is Cloudflare's machine-readable permission vocabulary. The values # below affirmatively GRANT what a restrictive policy would withhold: indexing, # retrieval into AI answers, training, and full use of the content. # # NOTE: Cloudflare can prepend a "Managed robots.txt" block ahead of this file that # Disallows the AI training crawlers, which would contradict everything below. If # that block reappears, turn it off under AI Crawl Control in the Cloudflare # dashboard — it cannot be overridden from this repo. User-agent: * Content-Signal: search=yes,ai-input=yes,ai-train=yes,use=full Allow: / User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: meta-externalagent Allow: / User-agent: Bytespider Allow: / User-agent: Amazonbot Allow: / User-agent: CCBot Allow: / User-agent: Diffbot Allow: / User-agent: cohere-ai Allow: / Sitemap: https://www.erikkangas.com/sitemap.xml # Structured summary for LLM crawlers: https://www.erikkangas.com/llms.txt