Table of contents of the article:
One of the key aspects of your website's visibility involves interaction with Google crawlers, the bots that crawl your content to index it in search results. Two key elements of this interaction are Google's "Crawl Statistics" and "Time to First Byte" (TTFB). Both of these factors can significantly impact the frequency with which Google visits your site and how quickly your content is indexed.
Google crawl statistics are a set of data describing how Google crawlers interact with your site. This data can include the number of crawl requests made, the time it takes to download a page, the success or failure of requests, and more. In general, your goal should be to have the lowest crawl time possible. A lower crawl time means Google's bots can crawl more pages in less time, which in turn can increase the frequency with which your site is indexed.
Time to First Byte (TTFB) is another key SEO metric, and we've covered it extensively in this post . It's the time between a client (such as a web browser or Google crawler) making an HTTP request and receiving the first byte of data from the server. Again, a lower TTFB is generally better: it means data is transmitted more quickly, which can improve both the user experience and the effectiveness of Google's crawling.
A common misconception is that the Google bots crawling your site come from European data centers, but a closer look reveals a different reality. By looking at your web server's access.log files, you can geolocate the crawlers' IP addresses and discover that many of them originate from the United States, specifically Mountain View, California . This adds an additional latency of about 100 ms, as can be seen by pinging the IP address.
For example, let's consider this Google Bot IP 66.249.64.239 that we retrieved from a recent log file:
66.249.64.239 - - [14/May/2023:03:37:25 +0200] "GET /wp-includes/js/jquery/jquery.min.js HTTP/2.0" 200 31017 "https://www.ilcorrieredellacitta .com/news" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/113.0.5672.63 Safari/537.36"
Most of you probably know that Google is in Mountain View and have heard of it thousands of times, yet few have any knowledge of where Mountain View is geographically.
Mountain View is a city located in the Silicon Valley region of Santa Clara County, California, United States. Situated on the west coast of the American continent, Mountain View overlooks the Pacific Ocean. Its geographical location is diametrically opposed to Europe: if you travel from east to west, Europe is on the other side of the Atlantic Ocean, thousands of kilometers away. This distance is amplified by the presence of the American continent between the two points.
The distance obviously involves a time to travel and even if we wanted to take the optical fiber as a means of propagation which has a speed close to that of light (but not that of light), the times of the various network devices must necessarily be added, such as routers and switch of the various HOPs as per the following traceroute for example.
Let's now try as an additional test to ping the IP 66.249.64.239 to measure the reverse latency time, i.e. from the web server to the Google bot.
We clearly see a latency of 104ms, and indicatively compared to an efficient European section that pings in a maximum of 20ms, we have an "extra" of about 80ms.
This latency may seem small, but it certainly adds up, and over time, especially if your site has many pages or is frequently visited by crawlers.
A direct consequence of higher crawl times is a visible decrease in Google's crawl requests. This is due to a limitation known as "crawl budget," which is the number of pages Google can and will crawl in a given period of time. If your site takes too long to respond to crawler requests, Google may decide to limit the number of pages it crawls, which could in turn reduce your site's visibility in search results.
For example, let's see a report from one of our former customers who recently migrated his site to a well-known high-performance hosting provider (at least they say so), noting what it entailed in practical terms.
It's obvious and extremely evident that following the supplier change, the orange line for average response time has skyrocketed, going from a healthy 160 milliseconds to a staggering 663 milliseconds, or more than four times slower. In short, if you were to do a back-of-the-envelope calculation, Google bots are taking four times longer to retrieve content from the site than they did before.
In fact, if we look at the total scan requests before and after the migration, we notice how they went from around 30 thousand to just 10 thousand , with a loss of around 2/3, an extremely worrying value especially on editorial-type sites that tend to produce a lot of content every day.
By repeating the analysis of Google's scanning statistics, after about 20 days, we see that the average response time has gone from the optimal 160ms to the current 1550, with peaks of over 300 milliseconds, with a worsening of the scanning speed from 10 to 20 times.
The requests for scanning have obviously worsened further, going from about 30 to just 5000, effectively decreeing the death of the site on search engines, Google News and Discovery.
With regard to the TTFB of search engines, we can find an eloquent testimony through the SpeedVitals tool which reports the following metrics for what concerns European values:
Despite the importance of these metrics, many site owners and technicians aren't fully aware of their impact. Some may not know what Google crawl statistics are, while others may not understand the relationship between TTFB and crawl rate. This lack of awareness can lead to suboptimal decisions regarding site hosting, its structure, and search engine optimization, as well as poor judgment on a hosting provider like us, which, to the layman or self-proclaimed expert, may appear to "sell hosting like everyone else."
For example, a site owner could choose a hosting provider based mainly on price, or on amazing commercial promises and sensational payoffs, without taking into account the fact that beyond marketing and beautiful promises, it is the facts that qualify the goodness of a provider, as well as that slower or more distant servers can increase TTFB and therefore reduce the effectiveness of Google crawls. Similarly, a tech might focus on things like keyword selection or site design, overlooking the fact that complex site structure or inefficient code can increase crawl time and reduce crawl budget, making even the best intentions.
Google crawl statistics and TTFB are two critical factors that can significantly impact your site's visibility in search results. To optimize your site's interaction with Google crawlers, it's essential to understand these metrics and factor them into every decision related to your site. This may include choosing a suitable hosting provider like ours, optimizing your server-side software stack, and constantly monitoring your crawl statistics and TTFB.
CDN services as a solution or palliative to the problem.
At least theoretically, one of the most effective strategies for improving scan time is the implementation of a robust software stack, including a properly configured server-side caching system with as high a HIT ratio as possible.
However, even with the best server-side configuration, performing operations such as accelerating the SSL session handshake, enabling OCSP stapling, TCP BBR, and TCP Fast Open, and meticulously tuning the kernel (as we routinely do at Managed Server Srl), we inevitably face the limitations imposed by physical distance . This is especially true for sites hosted in Europe that need to interact with Google crawlers based in Mountain View, California.
To overcome this challenge, many industry professionals are turning to Content Delivery Network (CDN) services in Platform as a Service (PaaS) mode. These solutions offer the ability to cache copies of site content on nodes closer to Mountain View through the use of AnyCast routing.
A CDN is a network of geographically distributed servers that work together to deliver web content quickly. These servers, known as points of presence (POPs), store copies of website content. When a user or bot requests access to this content, the request is sent to the nearest POP, thus reducing the time required to transmit the data.
Origin, in the context of a CDN, is the server or group of servers from which the original content comes. These origin servers deliver content to the CDN's POPs, who in turn distribute it to end users.
AnyCast routing is a network traffic routing method in which a user's requests are routed to the closest node in terms of response time. This can significantly reduce TTFB since the data doesn't have to travel long distances before reaching the user or bot.
In the context of computer networks, Anycast is a network traffic routing method that allows multiple devices to share the same IP address. This routing method is particularly popular in Content Delivery Network (CDN) and Domain Name Systems (DNS) services, as it reduces latency and improves network resilience.
To understand how Anycast routing works, imagine a network of servers distributed across different geographic locations around the world. Each of these servers shares the same public IP address. When a user sends a request to that IP address, the request is routed to the server closest to that user. Here, "closest" doesn't necessarily refer to physical distance, but rather to distance in terms of network hops or the fastest response time. In practice, this means that the user will connect to the node that can respond most quickly to their request.
The beauty of Anycast routing is that it is completely transparent to the end user. It doesn't matter where the user is or which server answers his request: the IP address is always the same. This makes the system extremely flexible and resilient. Should a server go offline or become overloaded, traffic can easily be redirected to another server without disruption to the user.
However, it is important to point out that “CDN” is an umbrella term that can refer to a wide variety of services, each with its own quirks. Not all CDNs are created equal, and to get the maximum benefits from a CDN, it's essential to understand its specifics and configure it correctly.
In many cases, using common commercial CDNs may not deliver the expected benefits. For example, while a CDN may have POPs near Mountain View, a misconfiguration or suboptimal CDN design can result in these POPs not having a cached copy of the site's content. In this case, Google crawler requests are redirected to the origin, potentially degrading response speed.
This problem can be particularly serious if the origin doesn't have a local caching system capable of serving content to crawlers in a matter of milliseconds. In this case, using a CDN can end up slowing down requests rather than speeding them up, negatively impacting crawl time and, consequently, the site's visibility in Google search results.
The problem can be further exacerbated if the CDN is not properly configured for caching. Some CDNs offer a large number of configuration options, each of which can have a significant impact on performance. If these options are not configured correctly, the CDN may not be able to serve cached content effectively, which can lead to longer response times.
The best way to monitor Google Crawl Stats.
Regardless of the technologies used and the vendors involved, the most effective way to monitor Google crawl stats and average response time is through the use of Google Search Console. The Crawl Stats panel provides detailed and up-to-date information about Google's crawling activity of your website. We recommend that you access this section on a weekly basis to closely monitor your site's performance.
For optimal performance, your average response time should always be less than 200 milliseconds. This value represents the theoretical maximum time Googlebot should take to obtain a response from your server, and a lower value means Googlebot can crawl your pages more efficiently, improving your crawl budget and Google's consideration. You can monitor your average response time through the Crawl Statistics link in Google Search Console.
The 200ms value is not a random value, but rather the maximum value explicitly indicated by Google https://developers.google.com/speed/docs/insights/Server?hl=it , and should be a best practice to be followed regardless as indicated in the suggestions for improving server response.
If you notice that your average response time is higher than 200 milliseconds, or if you're having issues with your crawl statistics, don't hesitate to contact us. Our experts are available to help you find a quick solution and improve your website's performance. Remember, a fast and responsive website is not only beneficial for your users, but also a key factor in good SEO.
