PetalBot

What is PetalBot?

PetalBot is a web crawler operated by Huawei Technologies that serves as the backbone for Petal Search, Huawei's search engine. This automated crawler systematically navigates websites to discover, read, and index content for inclusion in Petal Search's database. PetalBot functions as a conventional search engine crawler, similar to those operated by Google or Bing, following standard web protocols and best practices.

The crawler identifies itself through two primary user-agent strings:

• Desktop version: Mozilla/5.0 (compatible;PetalBot; +https://webmaster.petalsearch.com/site/petalbot)
• Mobile version: Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)

PetalBot's requests originate from IP addresses associated with Huawei International PTE. LTD, and reverse DNS lookups typically resolve to domains following the pattern petalbot-{IP}.petalsearch.com, providing webmasters with a verification method. The crawler respects standard web protocols including robots.txt directives, allowing website administrators to control access according to their preferences.

Why is PetalBot crawling my site?

PetalBot crawls websites to discover and index content for Petal Search's database. If you're seeing PetalBot in your logs, it means the crawler is evaluating your site for inclusion in Huawei's search engine results. The crawler is particularly interested in discovering new content, monitoring changes to existing pages, and maintaining an up-to-date index of the web.

The frequency of PetalBot visits varies based on several factors, including your site's popularity, how often you update content, and your site's overall relevance to search queries. More popular and frequently updated sites typically receive more regular visits. PetalBot may show particular interest in ecommerce content, product listings, and mobile-optimized pages, reflecting Huawei's focus on mobile devices and shopping-related search capabilities.

These crawling activities are standard practice for search engines and are generally considered authorized under the implied consent of publishing content on the open web, though you retain control over access through standard mechanisms.

What is the purpose of PetalBot?

PetalBot exists to power Petal Search, Huawei's search engine alternative that's integrated into Huawei devices and accessible to users worldwide. The crawler collects and indexes web content to help Petal Search provide relevant, timely search results to its users.

For website owners, having content indexed by PetalBot can provide additional visibility and traffic from Huawei device users, particularly in regions where these devices have significant market share. This represents another channel through which potential visitors can discover your content beyond traditional search engines.

The data collected by PetalBot is primarily used to build and maintain Petal Search's index, enabling the search engine to return relevant results when users submit queries. Like other search engines, Petal Search aims to organize web information and make it accessible to users.

How do I block PetalBot?

If you wish to control PetalBot's access to your site, the most straightforward method is using your robots.txt file. PetalBot respects the standard robots exclusion protocol, allowing you to block or limit its access.

To completely block PetalBot from your entire site, add the following to your robots.txt file:

User-agent: PetalBot
Disallow: /

If you only want to block PetalBot from specific sections of your site, you can use more targeted directives:

User-agent: PetalBot
Disallow: /private-directory/
Disallow: /members-only/
Allow: /

Huawei provides official documentation about PetalBot at https://webmaster.petalsearch.com/site/petalbot, where you can find additional information about the crawler and verification methods.

Keep in mind that blocking PetalBot will prevent your content from appearing in Petal Search results, potentially reducing visibility to users of Huawei devices who rely on this search engine. If your site targets international audiences, particularly in regions where Huawei devices are popular, allowing PetalBot to index your content might be beneficial for reaching these users.