UNIX.com is getting crushed in google search these days

Thread Tools Search this Thread
The Lounge What is on Your Mind? UNIX.com is getting crushed in google search these days
# 15  
Old 05-11-2019
Well, the many people I have discussed this with all seem to agree that Google is penalizing forums in one way or another, in their algorithm changes.

I serious doubt Google's algo development team wakes up in the morning and says "let penalize forums", but I am confident they are making their algorithms more "AI-like" and we can only see the symptoms of what they are doing.

In the last week, I have directly seen perfectly good forum discussions marked by Google as "soft 404" and not indexed because of a single keyword like "error"in the title of the discussion. When I manually change the title and remove the keyword "error", it passes Google's algorithm in flying colors. I have seen this for other phrases like "not found" as well.

This does make sense if you look at it from a global AI perspective. AI is not intelligence and nor are methods like Bayesian classifiers. It is collecting data globally and I am sure many bad links on the net return responses that have "error" or "not found" in the text. So, then, speaking globally, Google's then classifies links with test with "error" or "not found" in the title or the meta data as "bad".. or in their case "soft 404" and they do not index it.

So, the website a forum about dogs and cats, then that site probably does not have have titles and meta data like "My Dog Has An Error" or "Please Help Me with My Cat Error". So, based on my years of working with such classifiers (we also ran a Bayesian anti-spam classifier here at unix.com for many years), it is easy to see how a classifier could penalize a technology forum dealing with software errors as a matter. These are simply false positives in Google's algorithm.

This same is true of "thin content"

If someone asks a short question about grep and they get a short but accurate reply, and even if that reply is very helpful to everyone, Google's classifier cannot score that. Google will just score on the content "thinness".

I spend most of the week looking at all the links on our site which Google has classified as "soft 404 errors" and in each cause, either "thinness" or a keyword in the title or meta data like "error" or "not found" was the cause. In each case I confirmed it by double checking before and after I made the change.

So, to help with the soft 404s on posts and discussion threads, I added summary test to "similar threads". Now those pass Google's classifier and are currently being validated as "looking good".

As for all the "titles" and meta data with keywords like "error" and "not found".. that is a huge problem and of course we cannot change 12K threads and make the title and metadata senseless to pass Google's classifiers.

Unfortunately, this is how the classifiers work and it is really a very poor design which would classify a technology forum with links with "error" and "not found" in the metadata as "soft 404" but Google does not listen to me. In fact, since I left the US over 10 years ago and live on the seacoast in Thailand, very few people listen to me like they used to. People are mostly jealous! LOL

Anyway, I digress.

The good news is that I have made a lot of changes this week and learned a lot. The bad news is I cannot say with any assurance that the changes I make will have any immediate effect. Five or ten years ago, I could see changes have an effect very quickly; but as many have pointed out to me recently, the network is orders of magnitude larger now than it was back then and it is growing larger at neck-breaking speeds.
# 16  
Old 05-11-2019
Also, one of the best ways for any site to have a better search engine ranking is to have many quality backlinks.

I have looked at the web pages of many of the members who have many posts at unix.com and you might be surprised to know that very few of our members who have web sites have a link to unix.com on their web pages.

Every credible link from relevant pages (technology related sites, personal pages, social media posts, posts in other tech sites) which points back to any unix.com page (homepage, your profile page, or a discussion thread or post) helps boost our search engine rankings.
# 17  
Old 05-17-2019

Google Search Console shows downward trend has reversed and traffic is on the increase again.

Whew! That was a lot of work! But it seems to be working.

I will do a video on this in about a week or so and show the downward trend, the various issues, changes I made to reverse the downward trend, and results.
These 4 Users Gave Thanks to Neo For This Post:
# 18  
Old 05-22-2019
Well, it been around two weeks and in many geographic areas, for example the USA, a Google search for keyword "unix' has our site back on the first page.

Google Search Console shows the traffic from Google has increased to near mid March levels (erasing around six weeks of decline) and and is rising steadily.

This seems to indicate that we are "out of the woods" and my long hours of reviewing our search optimization in Google Search Console and making a few changes here and there, has paid off.


In a week or so, if the upwards trend continues, I will do a YT video on this. If not, no video on this topic, LOL.
These 5 Users Gave Thanks to Neo For This Post:
# 19  
Old 05-27-2019
Well, at least we are number one on Bing for the "unix" keyword, but only back up to number 8 - 10 on Google (currently #8 or #9 for Google US)

# 20  
Old 06-05-2019
Working on all the "soft 404" errors seem to be working.


Currently Active Users: 4168 (3 members and 4145 guests and 20 Spiders)

Edit: Or maybe not really... seeing a lot of abnormal bot traffic from Taiwan.
This User Gave Thanks to Neo For This Post:
# 21  
Old 06-06-2019
I got a new badge! by pressing reply!

Anyway - more seriously - this also means that "file not found", or "xxx not found" should return fewer/no results. Hopefully, eventually, google AI (analytics) will notice that people still "search" for that text.

Sad state indeed - when technology leads over common sense. I wonder what "social" problem they are trying to solve now.
Login or Register to Ask a Question

Previous Thread | Next Thread

6 More Discussions You Might Find Interesting

1. What is on Your Mind?

YouTube: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search

Getting a bit more comfortable making quick YT videos in 4K, here is: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search Console https://youtu.be/I6b9T2qcqFo (0 Replies)
Discussion started by: Neo
0 Replies

2. What is on Your Mind?

Search Results for the UNIX keyword - Google, Bing, DuckDuckGo

Some search results for the keyword "unix" searches: DuckDuckGo #1 https://www.unix.com/members/1-albums215-picture1254.png Bing #2 https://www.unix.com/members/1-albums215-picture1253.png Google #15 (page 2) https://www.unix.com/members/1-albums215-picture1252.png (1 Reply)
Discussion started by: Neo
1 Replies

3. What is on Your Mind?

Google Site Search in Search Drop Down Menu (Again)

Have just added (after missing for some time), the latest version of Google Site Search for our site in the Navbar Search Menu: https://www.unix.com/members/1-albums215-picture791.png Cheers and Enjoy. Here is the URL for that link in case you need it: https://goo.gl/P8p82c (4 Replies)
Discussion started by: Neo
4 Replies

4. What is on Your Mind?

Patching Google Search engine/application in Unix.

Hi Unix Gurus, In my Co. we have intranet site hosted on Unix box. In Explorer there is a text box for searching information on internet. By default it is using Google Custom Search. This search engine is little old one. Now I want to patch this search engine with latest patch. If any one know... (0 Replies)
Discussion started by: sriramis4u
0 Replies

5. Web Development

Google search appliance

Please let me know if Google search appliance supports JSON or XML interface? If yes please provide some references (3 Replies)
Discussion started by: uunniixx
3 Replies

6. Shell Programming and Scripting

A google search shellscript

This little doey allows you do fire up a google search right from your terminal. --------------------------------------------------- #!/bin/sh #(save me into the path as "google") clear && for i in "$@"; do lynx http://www.google.com/search?q="$@"; done ... (3 Replies)
Discussion started by: JoeTheGuy
3 Replies
Login or Register to Ask a Question

Featured Tech Videos