Google has updated its list of official crawlers by adding the name and information for a relatively unknown crawler that publishers have seen from time to time, but for which no documentation existed to date.
Although Google has added official documentation for this crawler, the information provided seems to encourage further clarification.
Special Crawlers Google has several types of crawlers (also known as bots and spiders).
The different forms of crawlers include:
- Common Crawlers These bots are mainly used for indexing different types of content. However, some common crawlers are also used for search testing tools, for internal use by Google's product team, and for AI-related crawling.
- Fetchers Triggered by Users These are bots activated by users. This includes uses such as retrieving feeds or site verification.
- Special Crawlers These are for special cases like checking the quality of mobile ad pages or for push notification messages via Google APIs. These bots do not respect global user guidelines in the robots.txt file which are marked with an asterisk (*).
The new documentation concerns the User Agent Google Safety. The crawler isn't new, but the documentation is.
Google-Safety Crawler The Google-Safety crawler documentation in the Special Crawler category is used by Google processes to detect malware.
Uniquely among Specialty Crawlers, the Google-Safety Crawler completely ignores all directives in the robots.txt file.
Here's what the new documentation for the Google-Safety Crawler says:
The Google-Safety user agent handles crawling specifically for reporting abuse, such as malware discovery for publicly available links on Google properties.
This user agent ignores the rules of the robots.txt file.
