AI crawler
An AI crawler is an automated bot that fetches web pages for an AI company, either to collect training data or to retrieve pages in real time to answer a person's question.
Example
A site's robots.txt contains a rule that disallows GPTBot. Separately, its content delivery network has a bot setting that challenges unfamiliar crawlers. Either one can stop an AI crawler from reading the site's pages.
Why it matters
AI companies run crawlers with published user agents, such as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot, and some separate training crawlers from the ones that fetch pages to answer questions. Blocking a crawler that retrieves pages for answers can keep your content out of that engine's responses, while blocking a training crawler is a different decision with different trade-offs.
Access can be blocked in more than one place. A robots.txt that allows a crawler does not help if a firewall or CDN rule turns it away, so it is worth checking both.
How Stellarcast handles it
Stellarcast checks your robots.txt for which AI crawlers it blocks and requests your site as each crawler's user agent to see whether it gets through. If you connect Cloudflare, it can also read the crawl activity that actually reached your site. The site also offers a free AI crawler checker.
See how AI describes your brand
Get a free AI visibility audit across ChatGPT, Gemini, Perplexity and Google AI Overviews and see where you stand against your competitors.
Get your free visibility audit