Journey to the Center of Googlebot: How Google Search’s Web Crawler Really Works

Google has sought to shed light on the internal workings of Googlebot, whose crawler is responsible for exploring the web in order to locate, process, and index content in Search. This is highly valuable information for SEO professionals, as well as for anyone interested in understanding how Google’s mechanisms operate and how to make the most of them.

The individual responsible for dispelling myths and providing clarity regarding crawling, data retrieval, and the processing of bytes has been Gary Illyes, an analyst on the Google Search team, through an official post on the Google Search Central blog.

Googlebot is not “the crawler”; it is one of the various Google crawling identifiers

One point the technology giant wished to clarify is that Googlebot does not comprise the entirety of its crawling infrastructure. As Illyes explains: “In the early 2000s, Google had only one product, which meant we had a single crawler. The term ‘Googlebot’ became popularized. However, nowadays, Googlebot is merely a user of something resembling a centralized crawling platform.”

This indicates that, at present, Googlebot is not the only identifier that Google employs. While Googlebot is the name used by the platform when operating for the Search team (Search), that same infrastructure uses other crawler names for services such as Google Shopping or AdSense. You may access additional information about Google’s most relevant crawler names here.

Understanding the limits of crawling

In his post, Gary Illyes explained the extent to which Googlebot is capable of crawling a website, outlining the byte limitations that this crawler can process according to file type and format. Each client (crawler) within Google’s crawling infrastructure has a distinct configuration profile and, as such, different limits.

“Each client within the crawling infrastructure must configure certain settings for their requests. These configurations include the user agent string, which user agent tokens to look for in robots.txt, and how many bytes they will retrieve from a single URL.”

At this time, Googlebot has the following limits:

• Googlebot crawls up to 2 MB of any individual URL. In other words, it retrieves only the first 2 MB of a resource, including the HTTP header. Therefore, if a webpage is 5 MB in size, Googlebot will stop downloading upon reaching 2 MB and will not read the remainder of the content.
• In the case of PDF files, it downloads the first 64 MB.

It should be noted that for those crawlers that do not specify a limit, this is set at 15 MB by default, regardless of content type. Additionally, “image and video crawlers typically have a wide range of threshold values, which depend heavily on the product being sought. For instance, the search for a favicon could have a very low limit, whereas image search would be different.”

How this impacts the bytes your server sends across the network

Once Googlebot begins to crawl your content from an individual URL, the following occurs:

• Partial retrieval: If your HTML file exceeds 2 MB, Googlebot does not reject the page; instead, it simply ceases retrieval precisely at the 2 MB limit. You should consider that this limit includes the HTTP request headers.
• Bytes left behind: Any byte beyond that 2 MB threshold is completely ignored. That is, these bytes are neither retrieved, rendered, nor indexed.
• Fragment processing: The downloaded data (the first 2 MB of bytes) are transmitted to indexing systems and to the Web Rendering Service (WRS) as if it were the entire file.
• Resource inclusion: All resources referenced in the HTML (excluding media files, fonts, and certain special files) will be processed by WRS with Googlebot, in the same manner as the main HTML. Each resource has its own per-URL byte counter and is not counted against the main page’s size.
• Rendering the bytes: WRS processes JavaScript and executes client-side code to understand the final visual and textual state of the page. Rendering retrieves and executes JavaScript and CSS files, and processes XHR requests to gain a better understanding of the textual content and structure of the page (it does not request images or videos). For each requested resource, the 2 MB limit is also applied. In this regard, Illyes notes: “WRS is stateless: it clears local storage and session data between requests. This may have specific implications for how our systems interpret dynamic elements that rely on JavaScript.“

While Illyes explains that a 2 MB HTML payload is extremely large, and it is highly unlikely you would ever reach that limit, he also cautions of the importance of not allowing excessively heavy code to push your useful content further down: “If your page contains excessively large embedded base64 images, massive blocks of CSS or JavaScript inline in the code, or starts with megabytes of menus, you could inadvertently push your actual textual content or critical structured data beyond the 2 MB mark. If those crucial bytes are not downloaded, then for Googlebot, they simply do not exist.”

Best practices recommended by Google

Finally, Illyes has compiled a series of recommendations you should implement to ensure that Googlebot can efficiently retrieve and understand your content. These are as follows:

• Optimize your HTML: Move heavy CSS and JavaScript to external files, as external scripts and style sheets are downloaded separately (subject to their own limits) and thus will not excessively interfere with the 2 MB limit for the initial HTML document.
• Place the most important information first: Position the most relevant elements (meta tags, <title> elements, <link> elements, canonical tags, and essential structured data) at the top of the HTML document. In this manner, you will facilitate their discovery before reaching the limit.
• Monitor your server response times: If your server has difficulty processing data, Google’s crawlers will automatically reduce their activity to avoid overloading the infrastructure, which will decrease the crawl frequency.

“Crawling is not magic; it is a highly orchestrated and scalable exchange of bytes. By understanding how our core data retrieval infrastructure acquires and limits those bytes, you can ensure that the most important content on your site is always included,” Illyes concludes.

Photo: generated with Nano Banana 2