UNIX.com is getting crushed in google search these days

Thread Tools Search this Thread
The Lounge What is on Your Mind? UNIX.com is getting crushed in google search these days
# 1  
Old 05-08-2019
UNIX.com is getting crushed in google search these days

For over a decade, unix.com has been in the top tier for search referrals. The keyword "unix" used to rank #4, and when it was down, it was #9. At times, we were close to #2 on Google for the "unix" keyword. Now, in some geos (in the US for example yesterday), in Google search the "unix" keyword search puts unix.com on page 4.

In other words, unix.com is getting crushed in Google search results based on recent changes to their search algorithm.

This problem started around the end of March, early April.

Our "on page SEO" is great. I have had it reviewed by a number of "SEO people" and they all say it's fine and offer no suggestions for "on page" SEO improvement.

Originally, I thought there was a bug in our Apache2 server causing "soft 404 errors", but after digging deeper into this, I confirmed:

Google Webmaster Tools Shows Problems with Soft 404 Errors

These "soft 404 errors" in Google Search Console are not really "errors". These Google search "soft 404 errors' are generated by Google's internal "AI" (or their algorithms) and they are basically penalizing the forum format where people tend to ask short questions and replies (which are often but not always, short replies).

Yesterday I confirmed this "penalty" by examining dozens of links where Google Search Console reported "soft 404 errors" and in each case I could correct this "error" by adding more text to the posts or replies.

So, basically we are being penalized now for being a "forum" and not a "articles" site.

This seems to correspond to advice from SEO'ers recently (who were not aware of the soft 404 error issue on this site) that we transform the site to a "news article site"... something I have been very hesitant to do (and am not going to do); as I have always wanted unix.com to be a resource to help unix and linux users on a day to day basis when they are in need of help.

Unfortunately, our traffic from Google Search referrals is tanking because of changes in Google's algorithm, and this is bring less and less users to the site with their questions.

This is not the case in Bing, where we rank very high and the "unix" keyword bounces from the #1 to the #3 position daily for this site. But Bing does not not generate much traffic when compared to Google, so unix.com is getting "killed" by Google now due to our decade long forum format.

When I was testing the Google "soft 404 errors" yesterday, I found that if a user had posted a message with a title like:

"Please help me with my file not found error"

Google ranked this as a "soft 404 error" because of the "file not found" in the title, even if there were pages of replies.

In other words, Google's current indexing algorithm is penalizing unix.com and the overall forum format and using some kind of "AI" to even look at the content. If the content had words like "file not found", they classify this as a "soft 404 error". This really took me by surprise. I expected the "thin content" to be problematic, but not "file not found" in a forum user's unix filesystem question.

So, in a nutshell, this is the reason that are visitors have dropped around 50% since the beginning of March. We are getting crushed by Google's algo changes this year.

I do not have any good answer to this. I cannot change how Google does things. Our on-page SEO is fine.

My deep analysis yesterday confirmed that Google is penalizing us ("soft 404 errors", a Google term not a real HTTP error) because we are using a "forum format" and many of the questions and answers are short replies and not long articles with a lot of content. This is especially true of short questions with no answers or questions with deleted answers, etc.
These 2 Users Gave Thanks to Neo For This Post:
# 2  
Old 05-08-2019
Also, note this additional "ding" from a Google bug:


first mentioned around 5 April.

This also roughly corresponds to the decrease in Google referral traffic over the same period.

April 9-25, 2019

Search Console experienced a data outage that ceased on April 26. The outage affected all reports except the Performance report. All missing data from April 9-25 was replaced with the data from April 26, when data began to be available again.

April 5, 2019

Because of internal management of our Search index, some pages of your websites might have been dropped from our index for a period of a few days without any action on your part. The matter has been resolved, no further action from your side is needed.
This User Gave Thanks to Neo For This Post:
# 3  
Old 05-08-2019
So the way the title is formulated has an impact... hmmm
But asking a question, a suggestion or help in plain english doesn't mean to be verbose, you expect that in the body...

This may relaunch the debate on compiling some knowledge we have here, I tried to see what it can represent, and was frightened by the amount of time and the judgement effort it asks when you have many solutions... But not impossible, to not be drowned by the load it would ask a serious team work to share the tasks, and I believe the first thing to do is to create a list of Q , the most frequent ones and important that needs to be answered, then each of us take one Q and search the forum for the answers provided here...
The group effort again is to decide after in what form we deliver the lot
A fundamental question, links to good solutions, then adding variation the the themes? Do we vote for best solution and provide alternatives?
This will only make sense if we choose the correct Q/A which represent the most of beginners in UNIX world challenges, it might if we formulate correctly the title and make an effort in body content help unix.com gain lost popularity, maybe also diminish workload in replying again and again to the same questions...
This also make me wonder if replies like: Searching the forum you would have found this solution to you Q <link to unix.com solution> are being now penalised and so we are doomed to answer systematically which means the above effort for a helpful Q/A effort would be vain...
This User Gave Thanks to vbe For This Post:
# 4  
Old 05-08-2019
I think when a user asks a "short question", there is no problem if they get a "long reply".

One idea I had was to write some PHP code that would add a block with a "Fun UNIX Fact" or "Fun Linux Fact" and append it to any post which:
  • A short question without a reply, and;
  • The original question is very short.

Yesterday, when checking the links in Google Search Console, tried this technique on a few unanswered initial posts, and I added a block of text to those 'soft 404 errors' and then the thread passed.

Or instead of "Fun UNIX Fact", I would write some code to take the "Similar Thread" data for that unanswered thread, and create a forum reply with the post data from a similar thread somehow, or append a block of the summary data to the original question, and kind of embedded "Topics Related to This" block, appended to short questions with no replies after a period of time (like a week).

I need to check to see if the current "soft 404 error" pages have similar threads at the bottom.

If they do, then another possibility would be to rewrite the similar thread block to include summary data for those similar thread, say the first 10 link lines for each "similar thread" in that block. That might work Smilie
These 3 Users Gave Thanks to Neo For This Post:
# 5  
Old 05-08-2019
Also, here is an example of how broken Google's algorithm is now.

This thread:


Before that thread above (about 10 minutes ago) used to have a title:

Error Executing Function

It would not pass Google's algorithm and so it was rejected in the index (not included in the Google search index).

I then changed the title to:

Executing Function

and re-ran the Google "checker tool" then it passed with flying colors.

The means that Google is doing simple algorithmic classification of titles (and maybe body text) for keywords like "error" or "not found" and they are rejecting those pages in their index!

This is very, very bad for unix.com, because people are always asking about errors, files not found, and using all these kind of keywords which Google's "very bad AI" thing means the page has a error or is a page not found, but actually it is a user asking about such things in a forum.

This is a perfect example of AI gone bad.

Failed Google's Index Checking:


Passed Google's Index Checking:


This is what these highly paid $200K USD a year Google AI engineers are doing with all their "AI and classification skills" ? !!! . ROTFL

Amazing how stupid AI actually is, isn't it? It's even more amazing that "big tech" thinks this is "the future of mankind"..... They actually pay teams huge salaries money to write these ' very simple, "child-like" classification algorithms.. ?

Google, you should be ashamed of yourself!
# 6  
Old 05-08-2019
Check this out!

Here I query the forum DB and quickly check the title in each thread for the word "error" ... this alone is around 12,000 pages Google will probably reject in their index because it has the world "error" in the title:

mysql> select count(*) from thread where  title LIKE '%error%';
| count(*) |
|    12031 |
1 row in set (0.10 sec)

These will probably get rejected too:

mysql> select count(*) from thread where  title LIKE '%not found%';
| count(*) |
|      350 |
1 row in set (0.09 sec)

LOL, I guess we will need use a thesaurus for "error" and to write SQL REGEX queries and change "error" to "misunderstanding" or "transgression" or "screw-up" to get these pages to index now !

Misunderstanding Executing Function

Or maybe:

Screw-up Executing Function

We truly live in a dystopian world now... thanks to big tech.
These 3 Users Gave Thanks to Neo For This Post:
# 7  
Old 05-08-2019
Hi Neo
In any case, a links to similar threads is a very common practice. Even there are options invisible to authorized users.
Login or Register to Ask a Question

Previous Thread | Next Thread

6 More Discussions You Might Find Interesting

1. What is on Your Mind?

YouTube: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search

Getting a bit more comfortable making quick YT videos in 4K, here is: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search Console https://youtu.be/I6b9T2qcqFo (0 Replies)
Discussion started by: Neo
0 Replies

2. What is on Your Mind?

Search Results for the UNIX keyword - Google, Bing, DuckDuckGo

Some search results for the keyword "unix" searches: DuckDuckGo #1 https://www.unix.com/members/1-albums215-picture1254.png Bing #2 https://www.unix.com/members/1-albums215-picture1253.png Google #15 (page 2) https://www.unix.com/members/1-albums215-picture1252.png (1 Reply)
Discussion started by: Neo
1 Replies

3. What is on Your Mind?

Google Site Search in Search Drop Down Menu (Again)

Have just added (after missing for some time), the latest version of Google Site Search for our site in the Navbar Search Menu: https://www.unix.com/members/1-albums215-picture791.png Cheers and Enjoy. Here is the URL for that link in case you need it: https://goo.gl/P8p82c (4 Replies)
Discussion started by: Neo
4 Replies

4. What is on Your Mind?

Patching Google Search engine/application in Unix.

Hi Unix Gurus, In my Co. we have intranet site hosted on Unix box. In Explorer there is a text box for searching information on internet. By default it is using Google Custom Search. This search engine is little old one. Now I want to patch this search engine with latest patch. If any one know... (0 Replies)
Discussion started by: sriramis4u
0 Replies

5. Web Development

Google search appliance

Please let me know if Google search appliance supports JSON or XML interface? If yes please provide some references (3 Replies)
Discussion started by: uunniixx
3 Replies

6. Shell Programming and Scripting

A google search shellscript

This little doey allows you do fire up a google search right from your terminal. --------------------------------------------------- #!/bin/sh #(save me into the path as "google") clear && for i in "$@"; do lynx http://www.google.com/search?q="$@"; done ... (3 Replies)
Discussion started by: JoeTheGuy
3 Replies
Login or Register to Ask a Question

Featured Tech Videos