SeoGeoMap
My project
GEO and AI answers

What is AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot)?

Updated on November 6, 2026

AI crawlers are the robots the assistant companies use to read websites, either to train models or to fetch sources for answers. Each has a name a site can allow or block in robots.txt.

OpenAI runs GPTBot for training data and OAI-SearchBot for the sources of ChatGPT search; Anthropic runs ClaudeBot; Perplexity runs PerplexityBot; Google uses Google-Extended as the switch for training, while its search crawler, Googlebot, also feeds AI Overviews and AI Mode. Each publishes its user agent and its address ranges.

A site that blocks OAI-SearchBot cannot be cited by ChatGPT search; a site that blocks Googlebot disappears from Google altogether. Many sites blocked GPTBot in 2023 to stay out of training data without realising that the search crawlers are separate. The robots.txt file is where this is decided, robot by robot.

Frequently asked questions

Should a company block AI crawlers?

Only if it accepts not being cited by those assistants. Blocking the training crawlers (GPTBot, Google-Extended) and allowing the search ones (OAI-SearchBot, Googlebot) is a common middle ground.

How do you know which AI crawlers visit a site?

In the web server's access logs, by their user agent, or in a crawler report of the hosting platform. Each vendor publishes the IP ranges to verify that a visitor is really its robot.

Related terms

robots.txt

robots.txt is the text file at the root of a site that tells crawlers which parts they may fetch, robot by robot. Honest crawlers obey it; it is not a security measure.

AI citation

An AI citation is a link an assistant places next to its answer to show where a statement comes from. It is the unit GEO work is measured in.

Crawling

Crawling is the way a search engine discovers and reads pages: a robot follows links and sitemaps, fetches each page, and hands its content to indexing. What the crawler cannot reach does not exist for the engine.

Back to the definitions