What is on Your Mind?

05-07-2019

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

Update:

Oddly enough, Google seems to have added "AI" to evaluate link names and titles and classify them "soft 404 errors".

For example, I have found a number of threads with titles and links with phrases like "file not found" when a user is asking a basic question about some file not being found.

Google's AI will also flag this as a "soft 404 error", I assume because of their AI classifier.

Editorial Comment

This shows, yet again, why AI is bad and certainly not "artificially intelligent" ... it is definitely "artificial" and "systematic" and "stupid", so better to call it what is really is:

"Artificially and Systematically Stupid" OR "ASS" for short. .

Or maybe even more accurate for this kind of AI:

Artificially and Systematically Stupid Having Only Low Endowment

Neo

11-10-2019

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

Quote:

Originally Posted by Neo

OK.

I have rechecked three dozen or more links that Google Search Console had listed as "soft 404" errors, using three different HTTP header inspection tools (using Chome browser):

DevTools (Chrome 73)
HTTP Header Spy 2.0.49
HTTP Headers 1.1

None of them show any 404 errors of any kind, hard or soft; and all links are providing the correct 200 responses. Some of the URL rewritten by DBSEO go 302 -> 200, but this is technically correct.

As far as all the tools in my toolbox, we are running as we should.

This problem with 302 -> 200 redirects has been solved.

See Discussion Below:

302 Redirects Issues Effected Google Search Console (GSC) Fixed - DBSEO Goto Rewrite Problems Solved

Neo

12-19-2019

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

I noticed today some "Google 404 errors" for man pages with "thin content", so I am going to create "similar man pages" for "man pages" using a mysql full text search algorithm, for all man pages less than 3000 chars.

Created the DB entry and PHP code already and am running a cron in the background to create "similarman for man pages".

Neo

12-20-2019

Administrator

19,118, 3,359

Join Date: Sep 2000

Last Activity: 15 July 2022, 8:51 AM EDT

Location: Asia Pacific, Cyberspace, in the Dark Dystopia

Posts: 19,118

Thanks Given: 2,351

Thanked 3,359 Times in 1,878 Posts

Still running this cron to create similar man pages from man pages, and just tested some of the results and the similar man pages are looking good.

FWIW, here is the PHP code I quickly put together for this caper:

Code:

<?php
include_once '/var/www/global.php';
global $vbulletin, $neo_global, $quickload;
$t1 = time();
if ($quickload > 5) {
    $makeLimit = 1;
} elseif ($quickload > 4) {
    $makeLimit = 10;
} elseif ($quickload > 3) {
    $makeLimit = 20;
} elseif ($quickload > 2) {
    $makeLimit = 25;
} else {
    $makeLimit = 30;
}

$strlen = 2500;

$neo_conn = getManDBConn();
$sql = 'select manid, text from neo_man_page_entry where similarman = "blank" and strlen < ' . $strlen . ' order by strlen ASC LIMIT ' . $makeLimit;
$maninfo = mysqli_query($neo_conn, $sql);
$t1 = time();
mysqli_query($neo_conn, "SET sort_buffer_size = 2048000");
$counter = 0;
while ($man = mysqli_fetch_assoc($maninfo)) {
    $out = '';
    $searchstr = $man['text'];
    $searchstr = html_entity_decode($searchstr);
    $searchstr = stripslashes($searchstr);
    $searchstr = strip_tags($searchstr);
    $searchstr = str_replace("'", " ", $searchstr);
    //echo $searchstr ."<p>";
    $sql2 = "select manid, MATCH(text) AGAINST ('" . $searchstr . "' IN NATURAL LANGUAGE MODE) as score,strlen FROM neo_man_page_entry where strlen > 2000 AND strlen < 2000000  ORDER BY score DESC LIMIT 15";
    $resultMan = mysqli_query($neo_conn, $sql2);

    $counter++;
    while ($manpage_raw = mysqli_fetch_assoc($resultMan)) {
        //echo $manpage_raw['manid']."<P>";
        $out .= $manpage_raw['manid'] . ",";
    }
    $out = substr($out, 0, -1);
    //echo $out."<p>";
    $insert = "UPDATE neo_man_page_entry set similarman = '" . $out . "' where manid =" . $man['manid'];
    mysqli_query($neo_conn, $insert);
    //echo $insert . "<p>";

}

$sql2 = 'select count(1) as count from neo_man_page_entry where similarman = "blank" and strlen < ' . $strlen;
$countinfo = mysqli_query($neo_conn, $sql2);
while ($count_raw = mysqli_fetch_assoc($countinfo)) {
    $rem = $count_raw['count'] / ($makeLimit * 60);
    $remaining = number_format($count_raw['count'] / ($makeLimit * 60), 1);
    if ($rem > 0) {
        break;
    }

}
$t2 = time();
$td = $t2 - $t1;
error_log(time() . " Time: " . $td . " Inserts: " . $counter . " Floor: " . $strlen . " Limit: " . $makeLimit . " ToDo: " . $count_raw['count'] . " RemainingTime: " . $remaining . " Hours QLoad: " . $quickload . "\n", 3, '/var/log/apache2/debug/neo_sim_man_man_pages_timing.log');
closeManDBConn($neo_conn, $maninfo);
closeManDBConn($neo_conn, $resultMan);

Code:

ubuntu@ tail -f neo_sim_man_man_pages_timing.log
1576821537 Time: 53 Inserts: 30 Floor: 2500 Limit: 30 ToDo: 56594 RemainingTime: 31.4 Hours QLoad: 1.56
1576821597 Time: 53 Inserts: 30 Floor: 2500 Limit: 30 ToDo: 56564 RemainingTime: 31.4 Hours QLoad: 1.59
1576821656 Time: 54 Inserts: 30 Floor: 2500 Limit: 30 ToDo: 56534 RemainingTime: 31.4 Hours QLoad: 1.59
1576821716 Time: 54 Inserts: 30 Floor: 2500 Limit: 30 ToDo: 56504 RemainingTime: 31.4 Hours QLoad: 1.79
1576821776 Time: 54 Inserts: 30 Floor: 2500 Limit: 30 ToDo: 56474 RemainingTime: 31.4 Hours QLoad: 1.77
1576821826 Time: 44 Inserts: 25 Floor: 2500 Limit: 25 ToDo: 56449 RemainingTime: 37.6 Hours QLoad: 2.07
1576821895 Time: 53 Inserts: 30 Floor: 2500 Limit: 30 ToDo: 56419 RemainingTime: 31.3 Hours QLoad: 1.83
1576821956 Time: 51 Inserts: 30 Floor: 2500 Limit: 30 ToDo: 56389 RemainingTime: 31.3 Hours QLoad: 1.84
1576822014 Time: 52 Inserts: 30 Floor: 2500 Limit: 30 ToDo: 56359 RemainingTime: 31.3 Hours QLoad: 1.53
1576822078 Time: 52 Inserts: 30 Floor: 2500 Limit: 30 ToDo: 56329 RemainingTime: 31.3 Hours QLoad: 1.12

As a side note,

I was a bit surprised to see how good the results are so far. When I checked about 20 similar man page entries, they were "spot on" and will be helpful for readers / future voyagers to the site. Of course man page with "nothing very similar" get more mixed results.

In a few days, I will add the code to the man pages that checks the strlen of the man page requested and if under 2000 (or maybe 1500), it will include another man page underneath in a section called something like "Check Out this Similar Man Page".. or something like that. This should mitigate the "soft 404" errors keeping certain man pages from being indexed by Google. (Status Update: This Todo is DONE)

This User Gave Thanks to Neo For This Post:

Neo