teoma bot

What is teoma?

Teoma is a web crawler operated by Ask.com (formerly known as Ask Jeeves), designed to index web content for their search engine platform. This specialized crawler originated during the Ask Jeeves era of internet search, with technology initially developed at Rutgers University before Ask Jeeves acquired it in 2001. Teoma functions as a traditional search engine indexing bot, systematically visiting websites to collect and analyze content for inclusion in Ask.com's search results.

The crawler identifies itself in server logs through user agent strings such as Mozilla/2.0 (compatible; Ask Jeeves/Teoma) or the more detailed Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html). These user agent strings include links to Ask.com's webmaster documentation, demonstrating their commitment to transparency in their crawling practices.

Teoma employs a minimalist approach to web interaction—it doesn't process JavaScript, doesn't use cookies, and doesn't render CSS. This streamlined design focuses on efficiently extracting and indexing text content rather than interactive elements or visual presentation. The crawler follows a breadth-first crawling strategy, prioritizing content based on domain authority, freshness, and relevance to common search queries.

Why is teoma crawling my site?

Teoma visits websites primarily to discover, analyze, and index content for inclusion in Ask.com's search results. If you're seeing Teoma in your server logs, it's likely cataloging your site's pages, links, and content to determine how they should rank for relevant search queries.

The crawler typically focuses on indexing text-based content, with particular emphasis on informational pages, FAQs, and academic content. It shows preference for educational domains and document formats like PDFs, reflecting Ask.com's historical focus on question-answering capabilities.

Crawling frequency varies based on your site's perceived authority and content update patterns. High-traffic or frequently updated sites may receive daily visits, while sites with static content might be crawled monthly. Significant content updates can trigger additional crawl passes as Teoma works to keep its index current.

All Teoma crawling is authorized as part of Ask.com's normal search engine operations, similar to how Google or Bing index the web.

What is the purpose of teoma?

Teoma's primary purpose is to build and maintain the content index that powers Ask.com's search engine. By systematically crawling the web, it creates a comprehensive database of websites, pages, and their content, which Ask.com then uses to deliver relevant results to user queries.

The data collected by Teoma supports Ask.com's search algorithms, including their "ExpertRank" system, which evaluates content relevance and authority. While less prominent than Google or Bing, Ask.com continues to serve a niche role in the search ecosystem, with Teoma providing the content foundation for their services.

For website owners, being properly indexed by Teoma potentially increases visibility in Ask.com search results, bringing additional traffic from their user base. While this represents a smaller audience compared to major search engines, it can still contribute to a site's overall discovery and reach.

How do I block teoma?

Teoma respects the standard robots.txt protocol, making it straightforward to control its access to your site. To block Teoma completely, add the following directives to your robots.txt file:

User-agent: teoma
Disallow: /

If you prefer to block Teoma from specific sections while allowing access to others, you can use more targeted directives:

User-agent: teoma
Disallow: /private/
Disallow: /members/
Allow: /public/

You can also use the Crawl-delay parameter to limit Teoma's crawling rate:

User-agent: teoma
Crawl-delay: 10

This setting requests that Teoma wait 10 seconds between requests, reducing server load.

Blocking Teoma will prevent your content from appearing in Ask.com search results, which may reduce your site's overall visibility. However, given Ask.com's relatively small market share compared to major search engines, the impact is likely minimal for most websites. If you're concerned about server resources, using the Crawl-delay directive rather than blocking entirely offers a balanced approach that maintains visibility while managing crawl frequency.