My project
Technical SEO

What is Crawling?

Updated on November 6, 2026

Crawling is the way a search engine discovers and reads pages: a robot follows links and sitemaps, fetches each page, and hands its content to indexing. What the crawler cannot reach does not exist for the engine.

Googlebot starts from pages it knows, follows every link it finds, and comes back at a rhythm set by how often the site changes and how much its servers can take. A page reachable only through a form, a script that runs after a click, or a link hidden to robots is rarely discovered. Internal links and a sitemap are how a site tells the crawler where its pages are.

Crawling also renders JavaScript, with a delay and a cost: a page whose text appears only after scripts run is read later and sometimes incompletely. Serving the text in the HTML itself is the safe choice, for search crawlers and even more for AI crawlers, several of which do not run scripts at all.

Frequently asked questions

How often does Google crawl a site?

It depends on the site: large, frequently updated sites are crawled many times a day, small static ones every few days or weeks. Search Console's crawl stats report shows the actual rhythm.

Can crawling hurt a server?

A badly configured site with endless filter combinations can be crawled into slowness. Blocking those parameter URLs in robots.txt and keeping one canonical URL per page keeps the crawler on useful pages.

Related terms