What is Googlebot-News?

Googlebot-News is a specialized web crawler operated by Google that was specifically designed to discover and index content for Google News. First introduced in December 2009, this crawler enabled Google to provide publishers with more granular control over how their content appeared in Google News versus regular search results. While it began as a distinct crawler with its own user agent, Google made a significant infrastructure change in August 2011, consolidating its crawling operations. Since then, Google has used its primary Googlebot crawler for news content, but continues to honor Googlebot-News directives in robots.txt files.

Googlebot-News historically identified itself with user agent strings like Mozilla/5.0 (compatible; Googlebot-News/2.1; +http://www.google.com/bot.html) or the simpler Googlebot-News/2.1. Today, the actual crawling is performed by the standard Googlebot user agent, but Google's systems still recognize and respect the Googlebot-News directives for content filtering purposes. This approach allows Google to maintain backward compatibility with existing publisher configurations while streamlining its crawling infrastructure.

Unlike general web crawlers that index all types of content, Googlebot-News focuses specifically on news-worthy content that meets Google's News content policies.

Why is Googlebot-News crawling my site?

If you're seeing Googlebot-News in your logs (or more likely, the standard Googlebot crawling for news purposes), it typically means Google is evaluating your content for potential inclusion in Google News. This crawler is particularly interested in recent news articles, press releases, investigative reporting, and other journalistic content.

The crawler visits sites that have been approved for Google News or those that Google's algorithms have identified as potential news sources. The frequency of visits depends on how often you publish new content and your site's overall importance in the news ecosystem. Sites that publish breaking news frequently may see multiple visits daily, while those with occasional news content might experience less frequent crawling.

Google News crawling is generally authorized and beneficial, as inclusion in Google News can significantly increase visibility and traffic to your content.

What is the purpose of Googlebot-News?

Googlebot-News serves to build and maintain the content index for Google News, a specialized news aggregation service that presents timely, relevant news content to users. Unlike general web search, Google News focuses exclusively on journalistic content, organizing stories by topics and presenting diverse viewpoints on current events.

The data collected by this crawler helps Google identify breaking news, trending topics, and authoritative sources across various news categories. This benefits users by providing access to current news from diverse sources, while also potentially driving significant traffic to publishers whose content is featured in Google News.

For website owners, being included in Google News can substantially increase visibility, especially for time-sensitive content. This can translate to higher traffic volumes and broader audience reach compared to standard search results alone.

How do I block Googlebot-News?

If you wish to prevent your content from appearing in Google News while still allowing it to be indexed for regular search results, you can use robots.txt directives specifically targeting Googlebot-News. Despite the infrastructure changes, Google continues to honor these directives.

To block all content from Google News, add the following to your robots.txt file:

User-agent: Googlebot-News
Disallow: /

This will prevent your content from appearing in Google News while still allowing it to be indexed for regular search results. For more granular control, you can block specific directories or pages:

User-agent: Googlebot-News
Disallow: /premium/
Disallow: /subscriber-only/

Alternatively, for page-specific exclusion, you can use a meta tag in the HTML of individual pages:

<meta name="Googlebot-News" content="noindex">

Keep in mind that blocking Google News may reduce your content's visibility and potential traffic, especially for news-oriented websites. However, it might be appropriate for subscription-based publications that want to control content distribution or for websites that publish content not suitable for news aggregation. Before implementing blocks, consider whether the potential loss in visibility aligns with your overall content strategy and business goals.

Googlebot-News