Server is too busy at the moment

[Crown]

Originally Posted by alexh

Can you put the metadata in to prevent crawlers?

<meta name="ROBOTS" CONTENT="noindex, nofollow">
<meta name="AdsBot-Google" content="noindex" />
<meta name="googlebot" content="noindex" />

if you were to do that, EAB & HOL could no longer be found when doing a search on a search engine.

I don't think this is really a good idea as many people found EAB through a post that was indexed by a search engine.

[Tigerskunk]

Originally Posted by hiho

It could be an attack.

It's probably those ST gangsters...

[emufan]

100 bots - 22x baidu 12x google. if the admins can add the mod_qos, it's for sure worth a try.

[daxb]

Originally Posted by Crown

if you were to do that, EAB & HOL could no longer be found when doing a search on a search engine.

I don't think this is really a good idea as many people found EAB through a post that was indexed by a search engine.

If that only effects google then it is no problem, it's positive. Nobody is forced to use google.

[frank_b]

Originally Posted by Steril707

It's probably those ST gangsters...

A coordinated, distributed and MIDI sequenced denial of service! Put that etra 1 million clock cycles per second to nefarious use

[haps]

Originally Posted by emufan

100 bots - 22x baidu 12x google. if the admins can add the mod_qos, it's for sure worth a try.

A quick look at 'whos online' and one that stands out straight away is 'AhrefsBot Spider'. I block the entire netblock that bot comes from on my webservers because it is a shit of a thing that is quite happy to hammer your webserver with hundreds of concurrent connections. I have had that bot hammer sites so hard they were pretty much unusable until I blocked it. On their site they claim to be a good bot and obey robots.txt, etc, but they are full of shit.

[emufan]

yes, I saw this one too, multiple times.
those google, yandex, baidu bots at least list EAB on their pages,
but on a quick search I cannot see what AhrefsBot gives back in return.