What is linkdexbot?

Linkdexbot is a specialized web crawler operated by Analytics SEO Ltd (now known as Authoritas), a company that provides enterprise SEO platform services. This bot was first deployed around 2009 and has gone through several versions since then. As a technical SEO crawler, linkdexbot systematically browses websites to collect data for SEO analysis and auditing purposes. Unlike general search engine crawlers, linkdexbot operates under explicit user direction, only visiting websites specified by Authoritas clients.

The bot identifies itself in server logs through several user-agent strings, depending on its version. Common variants include: Mozilla/5.0 (compatible; linkdexbot/2.0; +https://www.linkdex.com/en-us/about/bots/), Mozilla/5.0 (compatible; linkdexbot/2.1; +https://www.linkdex.com/en-us/about/bots/), and a mobile-specific version Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; linkdexbot-mobile/2.1; +https://www.linkdex.com/en-us/about/bots/). Each user-agent string includes a reference URL where website owners can find more information about the bot's purpose.

Why is linkdexbot crawling my site?

If you're seeing linkdexbot in your logs, it typically means that an Authoritas client—likely an SEO professional or marketing agency—is analyzing your website. The bot is examining your site structure, content, and technical elements to provide SEO insights to its users.

Linkdexbot visits websites to collect specific data points that help with SEO analysis, including backlink profiles, technical issues, and on-page elements. The frequency of visits depends on the client's settings and the size of your website. Generally, it's not as frequent a visitor as major search engine bots.

The crawling is typically authorized in the sense that it's initiated by legitimate SEO professionals using a commercial tool, but it is not directly authorized by you as the website owner unless you're working with someone using the Authoritas platform.

What is the purpose of linkdexbot?

Linkdexbot serves the Authoritas SEO platform by collecting data that helps users understand and improve their website's search engine optimization. Its primary functions include:

Backlink Analysis: The bot maps link graphs to identify authoritative domains and referral sources, helping SEO professionals understand their site's link profile.
Technical Audits: Linkdexbot detects issues like orphaned pages, non-canonical URLs, and broken links that might negatively impact search engine rankings.

For website owners, the indirect value comes from SEO professionals using these insights to make recommendations that could improve your site's performance in search results. If you're working with an SEO consultant or agency that uses Authoritas, the data collected helps them provide you with more accurate advice.

How do I block linkdexbot?

If you prefer not to have linkdexbot crawl your site, you can use your robots.txt file to block it. Linkdexbot respects standard robots.txt directives. To block this bot, add the following to your robots.txt file:

User-agent: linkdexbot
Disallow: /

This instructs the crawler to avoid all pages on your site. If you want to block only certain sections, you can replace the "/" with specific directories or files.

Before implementing a complete block, consider whether you're working with any SEO professionals who might be using Authoritas. Blocking the bot could hinder their ability to provide you with comprehensive SEO analysis and recommendations. If server load is your concern, a better approach might be to use rate limiting through your server configuration rather than completely blocking the bot.

If you need more granular control, you can also use your web server's configuration to limit the bot's crawl rate or restrict access based on its user-agent string. However, for most website owners, the robots.txt method provides sufficient control while being the simplest to implement.

linkdexbot