English Amiga Board - View Single Post

nocash · 23 April 2024, 22:22

Google is working again for eab since about 20 hours ago.

I've no idea if it was caused by robots.txt, in the "server is too busy" thread you had mentioned to block a whole range of IPs and (and least temporarily) all Mac users - and now switched to UBBB ultimate bad bot blocker - that might also have caused/fixed the problem (especially if that blocking did apply only to eab.abime, and not to amiga.abime, since the latter one did still work okay with google).

Btw. another (dubious) idea would be to add Crawl-delay in robots.txt to prevent bots from overloading the server, something like this

Code:

User-agent: *
Crawl-delay: 2
Allow: /

That Crawl-delay feature is a somewhat good or well-meant idea, but I am not too convinced that it works in practice. As far as I know only bing and yahoo do support it, and most other bots will probably simply ignore it.
And it's not really clear what the delay is supposed to do at all: Most webpages will confidently tell you that it's a delay in seconds between page accesses. But I've also seen a webpage that claims that the delay is counted in milliseconds. And the "official" specs https://blogs.bing.com/webmaster/Aug...awler,-MSNBot/ are merely saying that the delay must be a positive whole number and that values higher than 10 will severely affect the ability of the bot to effectively crawl your site (whatever that means).
In short, I wouldn't recommend to try to use Crawl-delay (unless you should run into more problems about the server getting too busy in future).

23 April 2024, 22:22	#49
nocash Registered User Join Date: Feb 2016 Location: Homeless Posts: 65	Google is working again for eab since about 20 hours ago. I've no idea if it was caused by robots.txt, in the "server is too busy" thread you had mentioned to block a whole range of IPs and (and least temporarily) all Mac users - and now switched to UBBB ultimate bad bot blocker - that might also have caused/fixed the problem (especially if that blocking did apply only to eab.abime, and not to amiga.abime, since the latter one did still work okay with google). Btw. another (dubious) idea would be to add Crawl-delay in robots.txt to prevent bots from overloading the server, something like this Code: User-agent: * Crawl-delay: 2 Allow: / That Crawl-delay feature is a somewhat good or well-meant idea, but I am not too convinced that it works in practice. As far as I know only bing and yahoo do support it, and most other bots will probably simply ignore it. And it's not really clear what the delay is supposed to do at all: Most webpages will confidently tell you that it's a delay in seconds between page accesses. But I've also seen a webpage that claims that the delay is counted in milliseconds. And the "official" specs https://blogs.bing.com/webmaster/Aug...awler,-MSNBot/ are merely saying that the delay must be a positive whole number and that values higher than 10 will severely affect the ability of the bot to effectively crawl your site (whatever that means). In short, I wouldn't recommend to try to use Crawl-delay (unless you should run into more problems about the server getting too busy in future). Last edited by nocash; 23 April 2024 at 22:30.