What is AwarioBot?

AwarioBot is a legitimate web crawler operated by Awario, a social media and web monitoring platform. It functions as a specialized web crawler designed to discover and collect new or updated content across websites, blogs, forums, and social media platforms. The bot helps Awario's service track brand mentions, monitor online conversations, and analyze audience sentiment for their customers.

AwarioBot actually refers to a family of crawlers that includes AwarioSmartBot and AwarioRssBot, each with specific crawling functions. These bots identify themselves in server logs with user-agent strings like Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html), AwarioSmartBot/1.0 (+https://awario.com/bots.html; bots@awario.com), or AwarioRssBot/1.0 (+https://awario.com/bots.html; bots@awario.com).

A distinctive characteristic of AwarioBot is its "gentle" crawling behavior—it's designed to make only one request per three seconds (or less frequently if specified in robots.txt) to minimize impact on website performance. The bot doesn't use consecutive IP blocks, so identification should be done via the user-agent string rather than IP addresses.

Why is AwarioBot crawling my site?

AwarioBot crawls websites to collect publicly available information that might contain mentions of brands, products, or topics that Awario's customers are monitoring. If you've noticed AwarioBot visiting your site, it likely means your content contains keywords or mentions that are relevant to Awario's users.

The bot typically looks for text content that includes specific brand names, product mentions, or industry-related discussions. It doesn't target sensitive or private information—only content that's already publicly accessible.

When AwarioBot visits a page, it's usually because that page has never been analyzed before or needs to be refreshed in Awario's database. You won't typically see recurring requests to the same page unless the content has changed or needs updating. The frequency of visits depends on how often your site publishes new content that matches what Awario users are monitoring.

What is the purpose of AwarioBot?

AwarioBot supports Awario's social listening and brand monitoring service. The data collected helps businesses track their online presence, monitor brand mentions, analyze customer sentiment, and stay informed about relevant conversations happening across the web.

For marketers and brand managers, this information is valuable for understanding audience perception, managing reputation, identifying potential customers, and measuring the impact of marketing campaigns. The bot essentially serves as the data collection mechanism that powers Awario's analytics and reporting features.

Website owners may benefit indirectly from AwarioBot's crawling if their content gets discovered by brands looking for mentions, potentially leading to business relationships or partnerships. However, the primary value is for Awario's customers rather than the websites being crawled.

How do I block AwarioBot?

AwarioBot respects the standard robots.txt protocol, making it straightforward to control its access to your site. You can use robots.txt directives to block the bot completely, restrict it from specific sections of your site, or slow down its crawling rate.

To block AwarioBot from specific parts of your site, add these lines to your robots.txt file:

User-agent: AwarioRssBot
User-agent: AwarioSmartBot
Disallow: /private/

To block AwarioBot from your entire site:

User-agent: AwarioRssBot
User-agent: AwarioSmartBot
Disallow: /

If you want to allow the bot but reduce its crawling frequency, you can use the Crawl-delay directive:

User-agent: AwarioRssBot
User-agent: AwarioSmartBot
Crawl-delay: 10

This example would instruct the bot to wait 10 seconds between requests instead of its default 3 seconds.

If you encounter any issues with the bot or need further assistance, Awario provides a dedicated support email at bots@awario.com where you can report problems. The company states they will quickly implement unique settings for your site if needed to ensure the crawling doesn't affect your site's performance.

AwarioBot