PressRAGBot

PressRAGBot is the crawler for PressRAG, a service that keeps a searchable copy of a news publisher's archive and updates it when the site publishes, edits or deletes a story.

It fetches pages only from publishers that work with PressRAG.

How to recognise it

It sends this user agent:

PressRAGBot/0.1 (+https://crawler.pressrag.net/)

All its requests come from one IPv4 address: 89.58.44.184. It doesn't use IPv6. If a request with this user agent comes from any other address, it isn't PressRAGBot.

How hard it crawls

At most 1 request a second and 2 connections at a time per host. If your robots.txt sets a longer Crawl-delay, for PressRAGBot or for all crawlers, it waits that long between requests instead.

It fetches robots.txt, sitemaps, RSS feeds, article pages and, on WordPress sites, the REST API, all from the publisher's own hosts. It doesn't follow links to other sites, run JavaScript or pretend to be a browser. If a site answers with a 403, a 429 or a challenge page, it backs off and a person looks into it.

How to block it

Add this to your robots.txt:

User-agent: PressRAGBot
Disallow: /

PressRAGBot checks robots.txt at least once a day, so the change takes effect within 24 hours.

Contact

To ask a question, or to have it slow down or stop: andrew.boborykin@gmail.com.