AI Crawlers and robots.txt: Which Bots to Allow AI Crawlers and robots.txt: Which Bots to Allow The training, search and user-fetch crawlers run by OpenAI, Anthropic, Perplexity, Google and others, and how to set robots.txt rules for each. September 18, 2026