YandexSitelinks bot

What is YandexSitelinks?

YandexSitelinksBot is a specialized web crawler operated by Yandex, Russia's leading search engine. This bot is designed specifically to validate and maintain the integrity of sitelinks displayed in Yandex search results. Sitelinks are the secondary links that appear beneath a website's primary search result, allowing users to navigate directly to key pages without first going through the homepage. The bot identifies itself in server logs with the user agent string YandexSitelinksBot. Unlike Yandex's general crawling bot, YandexSitelinksBot operates independently and focuses exclusively on monitoring URLs that have been designated as potential sitelinks. A distinctive characteristic of this bot is that it may require explicit disallow rules targeting its specific user agent in robots.txt files, as it might not always follow the broader directives meant for all bots.

Why is YandexSitelinks crawling my site?

YandexSitelinksBot visits websites to verify the availability and relevance of pages that Yandex has identified as potential sitelinks for your domain. It periodically checks these pages to ensure they remain accessible and continue to serve the purpose indicated by their link text. The bot typically focuses on pages that appear to be important navigation points within your site structure, such as about pages, contact information, product categories, or service listings. The frequency of visits depends on your site's prominence in Yandex search results and how often your content changes. Crawling increases when Yandex detects structural changes to your website or when it's reevaluating which pages should appear as sitelinks. This crawling is authorized as part of Yandex's normal search engine operations, similar to how Google or Bing crawl sites to improve search results.

What is the purpose of YandexSitelinks?

The primary purpose of YandexSitelinksBot is to enhance the quality of Yandex search results by ensuring that sitelinks remain functional and relevant. When the bot crawls your pages, it's checking several criteria: that pages return a proper 200 OK status code, that content aligns with the sitelink's labeled purpose, and that the URL structure remains stable. This verification process helps Yandex maintain user satisfaction by preventing broken or irrelevant links from appearing in search results. For website owners, having sitelinks displayed in Yandex search results can significantly improve click-through rates and user experience by allowing visitors to navigate directly to important sections of your site. The bot's work ultimately benefits both search users and website owners by creating more efficient pathways to relevant content. Website owners can monitor sitelink performance through Yandex Webmaster Tools, which provides reports on indexed sitelinks and any crawl errors.

How do I block YandexSitelinks?

YandexSitelinksBot respects robots.txt directives, but requires specific instructions targeting its user agent. To block this bot from crawling your entire site, add the following to your robots.txt file:

User-agent: YandexSitelinksBot
Disallow: /

To block access to specific directories or pages while allowing the bot to crawl other areas, you can use more targeted directives:

User-agent: YandexSitelinksBot
Disallow: /private-folder/
Disallow: /members-only/
Allow: /

It's important to note that blocking YandexSitelinksBot will prevent your site from displaying sitelinks in Yandex search results, which could reduce your visibility and click-through rates from users of this search engine. Before implementing blocks, consider whether the crawling is causing actual problems for your server. If you're experiencing excessive requests that strain your resources, a better approach might be implementing rate limiting for this bot rather than blocking it entirely. This allows you to maintain sitelinks while controlling server load. You can also use Yandex Webmaster Tools to manage how your site appears in search results and to communicate crawling preferences directly to Yandex.