About the Euroly crawler

Euroly uses a crawler to find products and deals in European web shops. The crawler is built to be polite, transparent and compliant with EU rules – including GDPR, ePrivacy and common standards for good crawl behaviour.

How the crawler identifies itself

The crawler always sends a clear User-Agent, so you can see who is visiting your site:

User-Agent: EurolyBot/1.0 (+https://euroly.eu/bot)

How we respect your rules

• robots.txt: We read and follow your robots.txt. Rules for eurolyBot and * are always respected – both Disallow and Allow (longest match wins).
• noindex: Pages with <meta name="robots" content="noindex"> or the header X-Robots-Tag: noindex are not indexed.
• Rate limiting: We visit the same domain at most one request at a time and with at least 2 seconds between requests. If your robots.txt specifies a Crawl-delay, we respect it. Concurrent connections are used across different domains, not against the same one.
• HTTPS only: We crawl only encrypted pages and follow only a few redirects.
• No personal data: We collect only publicly available product information (name, price, link). The image may be fetched and cached briefly in a small format for use in the product link. No cookies, no login, no personal data.
• European domains: The crawler stays within EU/EEA domains (e.g. .dk, .de, .fr).

How to block the crawler

1) robots.txt

Block the whole site for the Euroly crawler:

User-agent: eurolyBot
Disallow: /

Or block only certain paths:

User-agent: eurolyBot
Disallow: /checkout/
Disallow: /account/
Disallow: /search

2) Via meta tag on individual pages

<meta name="robots" content="noindex, nofollow">

3) Via HTTP header

X-Robots-Tag: noindex

4) Via firewall / server configuration

You can also block by User-Agent in your web server, e.g. in Apache:

<IfModule mod_rewrite.c>
  RewriteCond %{HTTP_USER_AGENT} eurolyBot [NC]
  RewriteRule ^ - [F,L]
</IfModule>

if ($http_user_agent ~* "eurolyBot") {
    return 403;
}

Would you rather join?

If you'd like your products shown in Euroly, you can instead add your Google Shopping feed directly – then we fetch prices and deals from it instead of crawling your site.

Questions?

Do you have questions about the crawler, or think it did something wrong? Write to support@euroly.eu and we'll get back to you as soon as possible.