English Amiga Board - Spiders / web crawlers

Today, just now there are 28 BoardReader spiders on EAB :sad

Here's an overview from their website:

Quote:

Overview

BoardReader was developed to address the shortcomings of current search engine technology to accurately find and display information contained on the Web's forums and message boards. Founded in May 2000 by engineers and students from the University of Michigan, Boardreader uses proprietary software that allows users to search multiple message boards simultaneously, allowing users to share information in a truly global sense.

Our Focus

Boardreader is focused on creating the largest repository of searchable information for our users. We also strive to increase traffic and exposure for the individual forum and message board.

By creating an interface that corresponds with multiple boards simultaneously, users can find answers to their questions from others who share similar interests. Our goal is to allow our users to search the "human to human" discussions that exist on the Internet.

Our Technology

Our technology is based on innovative, scaleable software created to quickly and accurately search information contained on message boards. Message boards are part of the Internet known as the 'Invisible Web' and pose many problems to traditional search engine spiders. The dynamic content is usually very deep and hard to search. In addition, many of these sites change their locations, servers, or url's almost daily presenting special searching challenges.

Special retrieval and indexing algorithms as well as unique topic relevance ordering rules are but a few parts of what is needed to allow you to view what we affectionately call the 'human experience'. We sincerely appreciate your comments and hope you enjoy our service.

Here's some more info about BoardReader and blocking them:

... http://www.webhostingtalk.com/showthread.php?t=1176901
... https://community.centminmod.com/thr...in-robots.550/

I love this comment from someone:

Quote:

Blocking them is trivial. The most irritating part about this is the crawl rate -- 20-50 requests per second from multiple IPs? was the bot designed by a complete tool or is this how they intended it?