Google search results gone

[Photon]

Search engines are overrated. Use a search box (that isn't itself a search engine). Too many vested interests, as opposed to "search the database for these terms".

"Search" engines don't want you to find things that are not products that they can say "ta-da! found it!" and put themselves as a middleman of, and claim to the seller that they let you find their product (from ad words) and so sell ad words to the seller.

It makes no sense (to suits) that a product should be free and useful. If it's free, it should be as useless to the user as they can make it, as long as it's useful to THEM.

Google got really crappy a long time ago. Try DuckDuckGo, it can't be worse. Maybe it's time for WebCrawlers again.

[modrobert]

After testing Google search both logged in and via proxy from other country not logged in, it seems EAB forum is not indexed anymore.

I tried with "amiga site:eab.abime.net" and get two results from attachments, but nothing from the forum posts.

When not restricting search with "site:" I get plenty of results from other Amiga forum sites, but nothing from EAB.

Is this intentional (e.g. blocking google crawl bots, or robots.txt)?

I checked the robots.txt file for eab.abime.net, looks like this:

User-agent: GPTBot
User-agent: CCBot
User-agent: ChatGPT-User
Disallow: /
User-agent: *
Allow: /
Sitemap: https://eab.abime.net/sitemap_index.xml.gz

Perhaps the crawl bots get confused by "Sitemap:"?

[pixie]

I had a simple script that would allow me to search easily over eab, now it doesn't work anymore on google, so now it's the only use I have for edge and bing (that because even if i change the search for bing.com it goes trough google)

[daxb]

IMO there is no reason for using Google search because there are better alternatives. Just some examples: https://www.privacytools.io/private-search

[pixie]

Originally Posted by daxb

IMO there is no reason for using Google search because there are better alternatives. Just some examples: https://www.privacytools.io/private-search

It's a matter of working vs not working at all.. before, if you set site:abime.net it worked like a charm, now it ceased to work. I won't go on if there's better search or not, since I don't know enough...

[Photon]

If there's an indexing issue, you should see it in your Google control panel. Possibly you could try a recrawl.

Probably some of the other search engines get some of their search results from Google, so now we can find out which do. Brave and DuckDuckGo don't, it seems. (Or it could be so, but stored results.)

It certainly seems to be a problem specific to Google, so the WebMaster could check his inbox for some "requires action" email maybe? Or perhaps an HTML validation error that the Google search engine specifically doesn't like. But the most likely reason is always blocking the crawl path to searchable content (i.e. threads) in .htaccess.

Lastly, we could try to play the "I'm a company, I'm more important to commercial search engines!" card. Register a Page on Facebook with EAB as a company and put the web address. Commercial search engines should be all over that.

[Bren McGuire]

mate what do you mean i don't get it like where do i go to do a generic search? if i understand you correctly you talk about going to a specific site and using their search boxes but that's not always a possibility and it makes no sense the idea of the search engine is to look at many sources for the same info to try to provide you the best (or the most) answers

with that said what can everyone recommend to use? i don't really like duckduckgo

Originally Posted by Photon

Search engines are overrated. Use a search box (that isn't itself a search engine).

[Photon]

Originally Posted by Bren McGuire

mate what do you mean i don't get it like where do i go to do a generic search? if i understand you correctly you talk about going to a specific site and using their search boxes but that's not always a possibility and it makes no sense the idea of the search engine is to look at many sources for the same info to try to provide you the best (or the most) answers

with that said what can everyone recommend to use? i don't really like duckduckgo

I mean that some so-called (but commercial) search engines such as those from Microsoft, Google, and Apple can at any time change the priority of results from "articles online containing what I searched for at the top" to "oh, but first: there was a product/service name or something close to it in one of your search words, and we want to position ourselves between the user and seller so we can approach sellers and say that our service is worth something to them, and find ways to sell search result priority and ad slots to them as a product".

Something like "we herd ppl search for x, and we herd u sell x, so here's a proposal".

I also mean that once you know a site that has lots of useful information, like this one, Stackoverflow, maybe even Reddit or Youtube, you can go to that site and use its search box to narrow your results. They might still not be prioritized by actual relevance. No search engine does that today. "One does not simply search the database for the words." meme.

Except a few sites. They still do that. Then you find what you're actually looking for, instead of this that, the other thing, and how about this. :/

Even worse is the "you have to login first, so we know what you like from the data we harvested, so that we can doctor the results and you can never find anything we don't think you don't like and so can never find anything new even if you typed in the exact words for what you want to find".

As for the problem to solve at the moment, crystal ball says EAB webmaster once submitted a directory of EAB to Google, and now Google doesn't support that anymore and the directory must be updated, or the Google account associated with the upload was removed or something along those lines.

I've recently found DuckDuckGo to give me better and more useful results than Google. You also get to click or copy the actual link, instead of a referral link with tracking info that's useless to share. Sometimes the referring takes a few seconds due to poor service by Google.

Give alternatives a try - at least when Google gives you sh#t. Google is far from the "best and only". Maybe 8 years ago, but not now.

[RCK]

Originally Posted by modrobert

After testing Google search both logged in and via proxy from other country not logged in, it seems EAB forum is not indexed anymore.

I tried with "amiga site:eab.abime.net" and get two results from attachments, but nothing from the forum posts.

When not restricting search with "site:" I get plenty of results from other Amiga forum sites, but nothing from EAB.

Is this intentional (e.g. blocking google crawl bots, or robots.txt)?

I checked the robots.txt file for eab.abime.net, looks like this:

User-agent: GPTBot
User-agent: CCBot
User-agent: ChatGPT-User
Disallow: /
User-agent: *
Allow: /
Sitemap: https://eab.abime.net/sitemap_index.xml.gz

Perhaps the crawl bots get confused by "Sitemap:"?

This is not intentional
I tried everything I can to get reindexed (including the generation of the sitemap who is correctly find by googlebot)

They just decided to drop everything from EAB and I don't know why, the only info I have is "explored, but not indexed" for 236k url (threads).

[TCD]

I found this:

Connectivity or Server issues

Server error impacts page indexation in 2 ways:

Page discovered but not indexed
Page explored but not indexed

They both indicate that Google knows that the page exists but doesn't index it. What could be the cause?

This often happens because Google's bot thinks a Crawl will overload your server. So, it backs off and reschedules the crawl.

https://seomatic.ai/blog/programmati...aster-indexing

Hopefully it will resolve itself with the new server hardware.

[malko]

Originally Posted by malko

For daily use, my choice went for startpage.com (think it's older than duckduckgo), which like duckduckgo has an interesting privacy browser plugin as well.

Originally Posted by daxb

IMO there is no reason for using Google search because there are better alternatives. Just some examples: https://www.privacytools.io/private-search

Based on the provided description I now know that startpage is 10 years older than duckduckgo

[modrobert]

Originally Posted by RCK

This is not intentional
I tried everything I can to get reindex (including the generation of the sitemap who is correctly find by googlebot)

They just decided to drop everything from EAB and I don't know why, the only info I have is "explored, but not indexed" for 236k url (threads).

OK, so that sucks. Normally a lot of traffic comes from Google search (when it works), can be like 30% which helps with ad revenue and bringing more users to the site.

Originally Posted by TCD

I found this:

https://seomatic.ai/blog/programmati...aster-indexing

Hopefully it will resolve itself with the new server hardware.

After reading the "TL;DR" part of the URL you linked there, found this:

Exceeding your site's crawl budget (the number of pages Google bot is allowed to crawl daily) by publishing too many pages at once could lead to unindexed pages.

Perhaps I misunderstand, but if this means Google has a max limit of indexed pages from one site, maybe EAB forum is rejected for that reason?

[TCD]

Originally Posted by modrobert

Perhaps I misunderstand, but if this means Google has a max limit of indexed pages from one site, maybe EAB forum is rejected for that reason?

I understand this as Google would only index the max limit of pages and some of them would be left unindexed. It seems like EAB isn't indexed at all. I would say if EAB is still left unindexed after the server upgrade it is a good idea to 'retire' some of the older threads to get under the limit.

[RCK]

Originally Posted by TCD

I found this:

https://seomatic.ai/blog/programmati...aster-indexing

Hopefully it will resolve itself with the new server hardware.

I don't think so, because most the time EAB's response time are pretty good.

[RCK]

Google decided to drop EAB from their index without giving any reason.

- Maybe because the threads doesn't use canonical URLs --> I can try to change the URL scheme with the new server.
- Maybe because the site is not responsive --> We need to move to Xenforo and lost real Amiga support.
- Maybe because the world is now mobile first --> We need to move to Xenforo and lost real Amiga support.
- Maybe because we are just old, and only fresh content is interesting for them --> I can't do anything about this.

[RCK]

If someone work at google and can ask why EAB is no more indexed, I will be happy to know why

[SpeedGeek]

AFAIK, as long as the Google search Bot is not blocked, then it's the Google algorithm which determines the level of indexing.

Could it be that blocking most of the other Bots reduced heavy traffic on EAB, and making the Google algorithm think the site is inactive?

[Dunny]

A while ago, Google announced that they were dropping the caching system they used to employ that you could use when a website went down - it's possible that they just deleted their EAB cache (along with their caches of the rest of the web) and only periodically index new data when they feel like it.

[nocash]

When searching for 'eab amiga', google does find the forum main page.
But it does seem to think that the sole content of the page is a link to that Hall of Light thing, without showing any further content or description.

Interestingly, google also offers something like "people also search for amiga forums", but, again, it is apparently not aware that eab is a forum itself.

I would very much assume that it's caused by a bug in the html code. An html online validator might help, like this https://validator.w3.org/check?uri=h...Inline&group=0
that thing finds about 200 "errors", but most of them are probably unimportant. I've tried 3-4 different browsers, and none of them has problems about displaying the page body (and search engines are probably even more lax about perfect formatting).

There are other webpages "Powered by vBulletin® Version 3.8.11", and google can display correct descriptions for those pages. So the problem must be some eab-specific customization.

I would try to remove the Hall of Light thing, or try to remove the google-specific scripts.

[cloverskull]

As an experiment, would it be possible to remove robots.txt for an hour or so and manually trigger Google to reindex EAB? That would be diagnostic in the sense that we'd understand if robots.txt is the issue.