About the ConservaNode fetcher
For website owners: what our fetcher reads, how politely, and how to opt out.
What it fetches
- Federal Register: the public federalregister.gov documents API.
- U.S. Department of Justice: the public justice.gov press-release API.
- GBIF: the public species-matching API, to look up scientific names.
We read only these public APIs and each host's robots.txt, never anything behind a login.
How often and how politely
- At most one run per source per hour.
- We read and obey robots.txt, including Crawl-delay.
- At least 2 seconds between requests.
- We stop on a 403 or 429 response.
- We do not follow redirects.
- Our User-Agent is honest:
ConservaNode/0.2 (+https://conservanode.org/bot.html; [email protected])
What is kept
We keep a copy of each document's text in which personal names have been masked automatically before it is saved, and each record links back to its source. If a name slips through, email us and we remove it.
How to opt out
Add this to your robots.txt and we stop:
User-agent: ConservaNode
Disallow: /Or email [email protected].