ViraScienceBot, our crawler
ViraScience is a free search engine for scholarly articles, theses and records in Persian and English. ViraScienceBot is our crawler: it reads Iranian scholarly journals so that researchers can find their articles.
What it fetches
- Article metadata: title, authors and affiliations, abstract, keywords, volume, issue, pages and DOI. Where a journal offers OAI-PMH or issue tables of contents, we use those to send fewer requests.
- The PDFs of articles published openly, without a login, so that their text and figures become searchable on ViraScience. Every result links to the article's own page on the journal's site.
- We do not keep authors' email addresses. We do not fetch pages that need a login or a payment, and we never get around a CAPTCHA or any other barrier for bots.
How it behaves
- It reads robots.txt (the rules for ViraScienceBot, or else the rules for
*) and never fetches a disallowed address. It reads each site's robots.txt again once a day. - It follows Crawl-delay; without one, it waits at least 3 seconds between two requests.
- It sends one request at a time to each site.
- On a 429 or 503 answer it waits as Retry-After says, and when errors repeat it spaces its requests further and further apart.
It identifies itself as:
ViraScienceBot/1.0 (+https://www.virascience.com/bot)How to slow it down or stop it
Add rules for ViraScienceBot to your site's robots.txt. To slow it down, for example to one request every 10 seconds:
User-agent: ViraScienceBot
Crawl-delay: 10To close your whole site to our crawler:
User-agent: ViraScienceBot
Disallow: /A change to robots.txt takes effect within a day. If you need it sooner, email us.
Contact
If our crawler causes trouble for your site, or you have a question, email support@virascience.com. If your site keeps bots out but you would like your journal's articles to be searchable on ViraScience, write to us too.