360Spider

What is 360Spider?

360Spider is a web crawler operated by Qihoo 360 Technology Co. Ltd., a major Chinese internet security company. It functions as the indexing bot for Haosou (formerly known as 360 Search), one of China's prominent search engines. This crawler systematically browses the web to discover and index content for inclusion in Haosou's search results.

The bot identifies itself in server logs with user agent strings that typically follow patterns like 360Spider, Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider, or more specifically as 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html). These identifiers allow website administrators to recognize when the crawler is accessing their content.

360Spider operates primarily on Windows platforms, using a 32-bit architecture and the Trident rendering engine to process web pages. Like other search engine crawlers, it navigates through links to discover new content, analyzes page structure and content, and sends this information back to Haosou's indexing servers.

Why is 360Spider crawling my site?

If you're noticing 360Spider in your server logs, it's visiting your site to gather information for Haosou's search index. The crawler is particularly interested in discovering and indexing content that might be relevant to users of the Chinese search market.

The frequency of visits depends on various factors, including your site's popularity, how often your content changes, and its relevance to Haosou's user base. Sites with higher traffic or those that update frequently may experience more regular crawling activity.

360Spider typically initiates crawling when it discovers links to your site from other indexed pages, when your site is manually submitted to Haosou, or during routine re-crawling of previously indexed content. This is standard behavior for search engine crawlers and represents authorized crawling activity, provided you haven't explicitly blocked it through robots.txt directives.

What is the purpose of 360Spider?

The primary purpose of 360Spider is to gather web content for Haosou's search engine database. By crawling websites, it helps Haosou maintain an up-to-date index of the web, particularly content relevant to Chinese internet users.

The data collected helps Haosou determine page rankings, content relevance, and search result quality. For website owners, having content indexed by 360Spider can potentially increase visibility to users of Haosou search engine, particularly in the Chinese market where Haosou maintains significant market share.

While the crawler provides the benefit of potential traffic from Haosou users, some website administrators may have concerns about server load from crawling activity or about having their content indexed in markets they don't specifically target.

How do I block 360Spider?

360Spider respects the standard robots.txt protocol, making it relatively straightforward to control its access to your site. To block 360Spider completely, add the following directives to your robots.txt file:

User-agent: 360Spider
Disallow: /

This instructs the crawler not to access any part of your website. If you want to block access to specific sections while allowing the crawler to index other parts, you can use more specific directives:

User-agent: 360Spider
Disallow: /private/
Disallow: /members/
Allow: /

This would prevent 360Spider from accessing your private and members directories while permitting access to other areas of your site.

Keep in mind that blocking 360Spider means your content won't appear in Haosou search results, potentially reducing visibility to users in China who use this search engine. If you're targeting the Chinese market, you might want to allow the crawler while monitoring its impact on your server resources. If you're not targeting this market or are concerned about server load, blocking may be appropriate for your situation.