Pingdom.com bot

What is Pingdom.com bot?

Pingdom.com bot is a monitoring crawler operated by SolarWinds through their Pingdom service. Pingdom is a website monitoring service that helps site owners track uptime, performance, and page speed. The bot functions as both a site monitor and a speed tester, regularly checking websites to ensure they're operational and performing well. First deployed around 2009, the bot has evolved through several versions over the years.

This bot identifies itself in server logs through several user agent strings, including Pingdom.com_bot_version_1.4_(http://www.pingdom.com/), Mozilla/5.0 (compatible; pingbot/2.0; +http://www.pingdom.com/), and Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http://www.pingdom.com/). The bot operates from a range of IP addresses, many hosted on Amazon AWS and other data centers worldwide.

Distinctively, Pingdom's bot makes regular, scheduled visits to websites based on the monitoring frequency set up by Pingdom customers. It loads pages completely to verify functionality and measure performance metrics.

Why is Pingdom.com bot crawling my site?

If you're seeing Pingdom.com bot in your logs, it's most likely because someone has set up monitoring for your website using Pingdom's services. This could be:

1. You or someone on your team who configured Pingdom monitoring
2. A client or partner who wants to track your site's performance
3. A third party who has legitimate interest in your site's availability

The bot typically crawls entire pages to verify they load correctly and to measure loading times. Visit frequency depends on the monitoring plan configured by the Pingdom user—ranging from checks every minute to less frequent intervals like hourly or daily.

Pingdom's crawling is generally considered authorized as it's a service-based crawler that only visits sites that Pingdom customers have specifically asked it to monitor. The bot doesn't randomly crawl the web looking for content; it only visits URLs explicitly configured in Pingdom monitoring setups.

What is the purpose of Pingdom.com bot?

Pingdom.com bot serves several specific functions in support of Pingdom's website monitoring service:

1. Uptime monitoring: The bot regularly checks if websites are accessible, alerting owners when outages occur
2. Performance testing: It measures page load times and other performance metrics
3. Transaction monitoring: It can follow predefined paths through websites to ensure key user flows remain functional
4. Content verification: It can check for specific content on pages to ensure they're loading correctly

The data collected helps website owners maintain reliable services by quickly identifying and resolving issues. Pingdom provides value through real-time alerts, detailed performance analytics, and historical uptime data. For website owners, this monitoring helps minimize downtime, optimize performance, and ensure a better user experience.

The bot is beneficial to monitored websites as it helps identify problems quickly, though the additional traffic from frequent checks might slightly impact analytics if not filtered.

How do I block Pingdom.com bot?

While Pingdom.com bot serves a useful purpose, you may want to block it if you're seeing unwanted monitoring activity. It's worth noting that the Pingdom bot doesn't always respect robots.txt directives, particularly when performing its core monitoring functions, as this would defeat its purpose of checking if sites are truly accessible to users.

If you want to attempt blocking via robots.txt, you can add:

User-agent: Pingdom.com_bot
Disallow: /

User-agent: pingbot
Disallow: /

User-agent: PingdomPageSpeed
Disallow: /

Since the bot may not honor these directives for its primary monitoring functions, you might need to implement IP blocking at the server level. Pingdom operates from numerous IP addresses across different data centers, making comprehensive IP blocking challenging and potentially requiring regular updates.

A more effective approach might be to contact the Pingdom user who set up the monitoring. If you're experiencing heavy traffic from Pingdom, check your own organization first to see if someone has configured excessive monitoring. You can also contact Pingdom support directly to address concerns about excessive crawling.

Keep in mind that blocking Pingdom may prevent legitimate monitoring if you or your partners use the service, potentially leaving you unaware of downtime or performance issues on your site.