BAPCBot
BAPCBot is BAPC's web crawler. It reads a few public pages of a site and records which domains link to which, for BAPC's link data.
What it does
- Obeys robots.txt, and stays away if your robots.txt can't be read because of a server error.
- Reads one page at a time from a site, at least 2 seconds apart, and follows your Crawl-delay.
- Leaves your site alone if it answers 429 or 503.
- Leaves your site alone if your Crawl-delay is over 30 seconds.
- Reads the first 1 MB of a page at most, and never more than a few pages of a site in one visit.
- Keeps only domain names and the address of the page a link was on. It doesn't keep your page text.
Where it comes from
BAPCBot runs on Cloudflare Workers, so its requests come from Cloudflare's network and don't have a fixed address. To recognise it, look for BAPCBot in the User-Agent. Cloudflare also adds a CF-Worker header naming the domain of the account that runs it.
How to keep it off your site
Add this to your robots.txt:
User-agent: BAPCBot Disallow: /
Changes to robots.txt count from our next visit. To ask us to leave your domain out completely and remove it from our data, write to BAPC Help Desk. Sites that ask are skipped from the next batch of the crawl, whatever your robots.txt says.