FreshRSS bot

What is FreshRSS?

FreshRSS is a self-hosted, open-source RSS and Atom feed aggregator written in PHP. It's maintained by a community of developers with its code hosted on GitHub. As a feed reader application, FreshRSS allows users to subscribe to and collect content from websites that publish RSS/Atom feeds, presenting this content in a unified, customizable interface.

FreshRSS functions as a feed fetcher that periodically polls subscribed feeds to retrieve updates. When it visits websites, it identifies itself with a user agent string that typically follows the format FreshRSS/[version] ([OS]; https://freshrss.org), such as FreshRSS/1.23.1 (Linux; https://freshrss.org). This minimalist identifier adheres to the software's design philosophy of simplicity and efficiency.

When FreshRSS retrieves content, it operates in a lightweight manner, targeting RSS/Atom XML endpoints at configured intervals. It includes standard HTTP headers such as Accept-Encoding for compression and operates without cookies or JavaScript execution, which aligns with typical feed consumption patterns.

Why is FreshRSS crawling my site?

If you notice FreshRSS in your server logs, it means one or more FreshRSS users have subscribed to your RSS or Atom feeds. The bot is periodically checking your feed URLs to see if new content has been published. This is a normal and expected behavior for RSS readers.

The frequency of visits depends on the refresh intervals configured by individual FreshRSS users or administrators. Some installations might check for updates every hour, while others might check less frequently, such as once a day. FreshRSS will only access the specific feed URLs that users have subscribed to, not your entire website.

These requests are legitimate and authorized as long as you're publicly offering RSS/Atom feeds. The crawling is triggered whenever the configured refresh interval has elapsed or when a user manually refreshes their feeds.

What is the purpose of FreshRSS?

FreshRSS serves as a content aggregation tool that helps users follow updates from multiple websites in one place. Instead of visiting dozens of different websites to check for new content, users can view all updates in a single, streamlined interface.

The primary benefits of FreshRSS include:

1. Content aggregation: Collecting articles and updates from multiple sources
2. Content organization: Categorizing feeds into folders and applying tags
3. Reading experience: Providing a clean, distraction-free reading environment
4. Content filtering: Allowing users to filter content based on keywords or patterns

For website owners, having your content accessed by FreshRSS indicates that users are interested in your material and want to follow your updates. This can lead to more consistent readership and engagement with your content. FreshRSS users are typically more dedicated followers of your site compared to casual visitors.

How do I block FreshRSS?

FreshRSS respects the standard robots.txt protocol, allowing you to control its access to your website. If you wish to restrict FreshRSS from accessing certain parts of your site or block it entirely, you can add specific directives to your robots.txt file.

To block FreshRSS from your entire site, add the following to your robots.txt file:

User-agent: FreshRSS
Disallow: /

To block FreshRSS from accessing specific directories or files:

User-agent: FreshRSS
Disallow: /private-directory/
Disallow: /members-only/
Disallow: /premium-content.xml

However, blocking FreshRSS might not be advisable if you're intentionally providing RSS/Atom feeds for your content. Blocking the bot would prevent users of FreshRSS from receiving updates about your content, potentially reducing your readership and engagement.

If you're experiencing excessive requests from a particular FreshRSS instance that's causing server load issues, you might consider implementing rate limiting based on the user agent string rather than blocking it entirely. This approach allows legitimate use while preventing potential abuse.

Remember that blocking FreshRSS will only affect users of that particular RSS reader, not other feed readers or direct visitors to your website.