What is SiteAuditBot?

SiteAuditBot is a specialized web crawler operated by Semrush, a leading SEO and digital marketing platform. It functions as a technical auditing tool that examines websites to identify issues related to SEO performance, technical health, and compliance with best practices. Unlike autonomous search engine crawlers, SiteAuditBot only visits websites when specifically triggered by Semrush users conducting site audits through the platform.

The bot identifies itself through its user agent string: Mozilla/5.0 (compatible; SiteAuditBot/0.97; +http://www.semrush.com/bot.html), making it easily identifiable in server logs. This transparency allows website administrators to distinguish SiteAuditBot from other crawlers like Googlebot or SemrushBot (Semrush's separate bot for broader data collection).

SiteAuditBot operates as a "developer helper" that crawls websites in a task-specific manner. When a Semrush user initiates an audit, the bot methodically examines all accessible pages, mimicking either mobile or desktop browser behavior depending on the audit settings.

Why is SiteAuditBot crawling my site?

If you're seeing SiteAuditBot in your logs, someone with a Semrush account is actively analyzing your website's technical performance. This could be:

Your own marketing or SEO team conducting regular audits
A consultant or agency you've hired to improve your site
A competitor researching your site's structure and performance
A potential business partner evaluating your technical infrastructure

The bot's visits directly correlate with audit schedules – it doesn't crawl autonomously or randomly. Crawling frequency depends entirely on how often Semrush users initiate audits of your site. Each audit session may generate multiple requests as the bot examines various aspects of your site structure, content, and performance.

What is the purpose of SiteAuditBot?

SiteAuditBot is integral to Semrush's Site Audit tool, which evaluates over 120 technical SEO parameters. Its primary functions include:

Identifying broken links, redirect chains, and 404 errors
Measuring page speed metrics and rendering performance
Assessing mobile-friendliness across devices
Verifying HTTPS compliance and secure connections
Detecting duplicate content, missing meta tags, or thin content
Finding crawlability issues that might affect search engines

The data collected helps website owners identify and resolve technical issues that could be harming their search engine rankings or user experience. Regular audits using this tool can lead to improved search visibility, as fixing issues like broken internal links enhances crawl efficiency for search engines.

How do I block SiteAuditBot?

SiteAuditBot respects robots.txt directives, making it straightforward to control its access to your site. To block it completely, add the following to your robots.txt file:

User-agent: SiteAuditBot
Disallow: /

To allow access (the default if no directive exists):

User-agent: SiteAuditBot
Disallow:

You can also restrict access to specific directories:

User-agent: SiteAuditBot
Disallow: /private/
Disallow: /admin/

If server stability is a concern, you can control crawl rate by adding a crawl-delay directive:

User-agent: SiteAuditBot
Crawl-delay: 10

This forces the bot to wait 10 seconds between requests, though this will only work if the "Respect robots.txt crawl-delay" option is enabled during the audit setup.

Blocking SiteAuditBot may prevent others from obtaining technical insights about your site, but it won't affect your search engine rankings since this isn't a search engine crawler. However, if your own team uses Semrush for audits, blocking the bot will prevent them from getting complete reports.

SiteAuditBot