RiverBot is River's web crawler. It fetches publicly available web pages that we use to train and evaluate our AI models. It always identifies itself, follows robots.txt, and keeps its pace on every site deliberately slow.
Recognising RiverBot
Every request RiverBot makes carries this user agent:
RiverBot/1.0 (+https://river.ai/bot)
Its token in robots.txt is RiverBot. It never imitates a
browser, and it doesn't switch addresses to get around a block.
Blocking or limiting RiverBot
To keep RiverBot off your whole site, add this to your robots.txt:
User-agent: RiverBot
Disallow: /
To keep it out of part of your site, disallow just those paths. To
slow it down, add a Crawl-delay in seconds to the same
group. RiverBot follows the group that names it, or the
* group if none does, and ignores groups written for
other crawlers.
RiverBot reads robots.txt again at least once a day, so a change takes effect within 24 hours. If your robots.txt returns a 4xx error, RiverBot treats the site as unrestricted, as the robots.txt standard says. If it returns a 5xx error or can't be reached, RiverBot fetches nothing from the site until it can read it.
How RiverBot crawls
RiverBot makes one request at a time to each site. It starts at no
more than one request per second and never goes above two, unless
the site's owner has agreed to more with us. It slows down when a
site responds slowly or with server errors. A 429 or 503 response
makes it wait for the time given in Retry-After and
crawl that site more slowly from then on.
It fetches only the pages it has been given and the redirects they lead to, each checked against robots.txt. It doesn't follow links from page to page, run JavaScript, submit forms or log in. If a site answers with a challenge page or keeps refusing it with 403 errors, RiverBot stops crawling that site until someone at River has looked at it.
Contact
If RiverBot is causing a problem on your site, or you'd like it to crawl differently, email [email protected]. Include your domain and, if you can, the time of the requests.