What is NetcraftSurveyAgent?

NetcraftSurveyAgent is a specialized web crawler operated by Netcraft, a company known for internet security and infrastructure analysis services. This bot systematically visits websites to collect information about web server technologies, SSL certificate configurations, and hosting infrastructure details. It operates as a non-AI crawler that follows predefined patterns to gather data for Netcraft's market intelligence reports and security analyses.

When visiting your site, NetcraftSurveyAgent identifies itself through its user agent string: Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com). Some variations may exist, such as Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0/cc-prepass-https; info@netcraft.com). The bot is designed to be non-invasive, typically spacing its requests at least 15 seconds apart to avoid overloading servers.

Unlike many modern crawlers, NetcraftSurveyAgent doesn't execute JavaScript, process CSS beyond basic rendering, or interact with dynamic content. This design aligns with its primary function of collecting server-side data rather than rendering full pages.

Why is NetcraftSurveyAgent crawling my site?

NetcraftSurveyAgent is likely crawling your site to gather technical information about your web infrastructure as part of Netcraft's broader internet surveys. The bot isn't particularly interested in your content itself, but rather in the technical details that your server reveals.

The crawler is most active in scanning websites across various sectors to build comprehensive datasets about web server usage, security implementations, and hosting patterns. Government and financial sector websites often receive particular attention due to their importance in security auditing.

The frequency of visits depends on your site's profile and changes in your infrastructure. Most sites will see occasional visits as part of regular survey updates, though high-profile or frequently changing sites may experience more regular crawling.

What is the purpose of NetcraftSurveyAgent?

NetcraftSurveyAgent supports Netcraft's commercial services, which include internet security monitoring, anti-phishing protection, and market intelligence reporting. The data it collects contributes to quarterly internet trend reports that track shifts in web server preferences, TLS implementation, and hosting patterns.

Financial institutions, cybersecurity firms, and technology companies use these insights for trend analysis and risk assessment. Software vendors may rely on this data to prioritize feature development and security patches based on real-world deployment statistics.

A secondary function involves trademark monitoring and brand abuse detection. When configured for specific missions, NetcraftSurveyAgent can help identify unauthorized use of corporate logos or phishing sites impersonating legitimate businesses as part of Netcraft's anti-phishing services.

For website owners, this crawling generally poses minimal impact while contributing to broader internet security efforts and technology tracking.

How do I block NetcraftSurveyAgent?

While Netcraft recommends against blocking their crawler due to its non-invasive nature, you can control access if desired. NetcraftSurveyAgent respects standard robots.txt directives, making this the simplest way to manage its behavior.

To completely block the crawler, add these lines to your robots.txt file:

User-agent: NetcraftSurveyAgent
Disallow: /

If you prefer to allow crawling but want to limit its rate, you can use the Crawl-delay directive:

User-agent: NetcraftSurveyAgent
Crawl-delay: 30

This example sets a 30-second delay between requests. You can also block access to specific directories while allowing access to others:

User-agent: NetcraftSurveyAgent
Disallow: /private/
Disallow: /admin/
Allow: /

Keep in mind that blocking this crawler won't significantly impact your site's performance since it's already designed to be lightweight. However, blocking it means your site won't contribute to Netcraft's internet surveys, which some organizations consider valuable for the broader security ecosystem. If you're experiencing issues with the crawler, you can also contact Netcraft directly through the email address in their user agent string.

NetcraftSurveyAgent