UNIX for Advanced & Expert Users

I'm not sure if you understood it right. Prometheus is getting the Performance data via http, that's correct, but it does not use the installed http-server which is serving normal http requests. That obviously would be senseless for the reason you mentioned.

That data provisioning is done by exporters which are tiny lightweight webservers running standalone on the target systems. Like the apache exporter(see here). And well - "lightweight" might be a bit irritating label for a binary with 12 MB size, but as it is go it is statically compiled that's a bit different from a normal dynamically linked executable.

The apache exporter fetches data from apache status module and listens on port 9117 by default.

Update:

I have confirmed 100% the source of the these spikes were very aggressive, rogue, unidentified bots originating on Chinese networks. After blocking the resident networks of these bots, all spikes have stopped, completely.

This is a "huge success story", going from unknown, uncorrelated performance hits / spikes due to nearly random spikes in performance to cause identification and total resolution. As you can see from the graph over the last 24 hours, there have been zero spikes.



I will keep the same MQTT and Node-RED instrumentation in place (which I am very pleased with) and will also keep all "spike trapping" instrumentation and DB logging in place, so if other spikes appear, which I am fairly confident more of these "pesky" bots will appear sooner or later, I will trap them, identify the source and block their resident networks.

Success!

MQTT and Node-RED did not "solve the problem". MQTT and Node-RED provided a very powerful and flexible way for me to quickly instrument custom sensors and logging, which helped me identify the problem.

I highly, recommend MQTT and Node-RED. These tools are free. Thank you very much MQTT and Node-RED developers!

Thanks for clarifying.

Honestly, I have great working tools and instrumentation already which I am very (extremely) happy with; and so I am not looking for other tools (in the project).

If you (or anyone) wants to start a new thread on Prometheus and / or create a tutorial and show "how you use it in solving a real-world problem", I encourage you to do so, for sure.

Thanks!

Update:

There have been no spike after two fulls, busy weekdays (Monday and Tuesday).

Mission Accomplished!