AI Crawlers
Automated agents (GPTBot, ClaudeBot, PerplexityBot) that index web content for AI training and retrieval.
Major AI providers deploy their own crawlers to index the public web. GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot all follow robots.txt rules but serve different purposes — some for training data collection, others for real-time retrieval augmentation. Allowing these bots access to your content is a foundational GEO infrastructure step.