FroocanBot
What our crawler does, in plain language — so you can decide
whether to allow it.
FroocanBot/1.0 (+https://froocan.com/bot; Africa-only search index; contact: hello@froocan.com)
The rules it keeps
- One identity. The user-agent we send is the same
one we check your robots.txt against. Checking robots as one identity and
fetching as another makes the robots decision meaningless.
- robots.txt is obeyed. If it disallows a URL, we
do not fetch it at all.
- One request at a time per site, with a delay of
1.0–2.5 seconds between them, and at
most 400 pages per domain.
- HTML only. Images, video files, PDFs, archives and
binaries are never downloaded.
- No re-hosting. We store a title, a description and
text for the index. Video plays through the platform's own embed, so the creator
keeps their view count.
To block it
User-agent: FroocanBot
Disallow: /
It will stop on the next pass. To be
removed from the index, or to have a site added, get in touch.
