Thanks Scrutinizer,

I'm feeling confident that the root cause is related to MySQL (maybe 80% confidence level, off the top of my head).

The reasons are as follows:

Perhaps, more importantly is that historically, when the server has had a performance issue, the root cause has always been related to mysql performance.

So combining "what we know now" with "what we know from the past", the logical direction to investigate in detail is MySQL.

So, back to MySQL "basics" again, I have reenabled the MySQL slow query log and set the time "back up to 10 seconds" and will see if this traps the antagonist of this caper:

Code:

mysql> SET GLOBAL slow_query_log = 'ON';
mysql> SET GLOBAL long_query_time = 10;

Back to basics, as they say.

Since I am using MQTT for most of my instrumentation these days, I may write a parser to parse the output of:

Code:

mysqldumpslow /var/log/mysql/neo-mysql-slow.log

... and publish any interesting queries out to my MQTT instrumentation pub/sub bus I use now for consolidating these types of analytical tasks.

For example, in here is an example from logrotate, which is just one example of how I am now experimenting with MQTT to log server events to the DB :

Code:

ubuntu# cat certbot
/var/log/letsencrypt/*.log {
    rotate 12
    weekly
    compress
    missingok
    postrotate
    /usr/bin/mosquitto_pub -t server/logrotate -m "letsencrypt.log" -q 1 -u user -P password
    endscript
}

And in the many PHP application cron and server-batch scripts as a part of the LAMP implementation:

Code:

$C_Start = '/usr/bin/mosquitto_pub -t forum/cron/cleanup2 -m "start" -q 1 -u user -P password;
$output = shell_exec($C_Start);

 // code

$C_End = '/usr/bin/mosquitto_pub -t forum/cron/cleanup2 -m "end" -q 1 -u user -P password;
$output = shell_exec($C_END);

These days, I do nearly all my server instrumentation using MQTT because I use two MQTT analytical tools on two mobile phones, and one MQTT analysis tool on the desktop, and also use Node-RED on the server-side for analysis and visualization, and of course raw database searches in my instrumentation DB table (and also phpmyadmin, for quick looks).

As it stands now, I am a huge fan of MQTT (these days) to use in a variety of monitoring and instrumentation applications.

If MySQL slow query analysis does not yield any fruit, will need to come up with a new analysis plan / instrumentation. Let's see what happens after a day or two with the MySQL slow query analysis.