Embedly

What is Embedly?

Embedly is a set of front-end tools that help web and app developers create better user experiences by enabling rich content embedding. Operated by Embedly, it functions as a middleware service that fetches metadata from URLs to generate previews, embeds, and structured data for third-party content. Embedly primarily serves as a content fetcher rather than a traditional web crawler—it only accesses URLs explicitly provided by end users through client applications.

When making requests to websites, Embedly identifies itself with the user agent string Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/). This string includes a support link where website owners can find more information about the service.

Unlike search engine crawlers that recursively follow links, Embedly only fetches specific URLs provided through its API or Cards products. It supports over 200 providers including YouTube, Vimeo, and Instagram, using regex-based URL matching to determine compatibility and return standardized metadata.

Why is Embedly crawling my site?

If you see Embedly accessing your site, it's typically because someone has shared a link to your content on a platform that uses Embedly's services. When a user shares your URL on social media, content management systems, or applications that use Embedly, the service fetches your page to generate rich previews with titles, descriptions, thumbnails, and embed codes.

Embedly doesn't proactively crawl websites—it only visits URLs that users have explicitly shared. The frequency of these visits depends entirely on how often your content is shared on platforms using Embedly's services. Each visit is triggered by a specific user action rather than automated crawling schedules.

These requests are generally considered authorized as they're initiated by legitimate user actions and follow standard web protocols. Embedly respects standard HTTP caching headers, which can help reduce unnecessary repeat requests to your server.

What is the purpose of Embedly?

Embedly's core purpose is to enhance user experiences by transforming plain URLs into rich, interactive content previews. The service supports social media platforms, content management systems, and applications that want to provide engaging previews when users share links.

When someone shares a link to your site, Embedly retrieves information about that URL, checks it against malware registries, extracts content, makes additional API calls to providers that support them, parses RSS feeds, and performs validation. It then returns standardized metadata that allows the embedding platform to display rich previews of your content.

For website owners, Embedly can increase engagement with your content by making shared links more visually appealing and interactive. This can drive more traffic to your site and improve user experience when your content is shared across the web.

How do I block Embedly?

While Embedly respects robots.txt directives, blocking it is generally not recommended as doing so would break link previews for users sharing your content, potentially reducing engagement. If you still wish to control Embedly's access, you can add the following to your robots.txt file:

User-agent: Embedly
Disallow: /

This directive will prevent Embedly from accessing your entire site. For more selective blocking, you can specify particular directories:

User-agent: Embedly
Disallow: /private/
Disallow: /members-only/

Keep in mind that blocking Embedly means that when users share links to your content on platforms using Embedly's services, those links will appear as plain text without rich previews, thumbnails, or interactive elements. This could reduce the visibility and appeal of your content when shared.

If you're concerned about bandwidth usage, a better approach might be to implement proper HTTP caching headers, which Embedly respects. This allows you to control how frequently Embedly refreshes its cache of your content without completely blocking the service.