Using Node-RED and MQTT to Monitor Server and Application Stats


 
Thread Tools Search this Thread
Special Forums UNIX and Linux Applications Infrastructure Monitoring Using Node-RED and MQTT to Monitor Server and Application Stats
# 1  
Old 02-04-2020
Using Node-RED and MQTT to Monitor Server and Application Stats

After setting up MQTT and testing some ESP8266 and ESP32 modules, where I noted that testing in Programming ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages, I was so impressed with MQTT that I installed MQTT on three different computers, instantly and permanently.

MQTT is so easy to set up and start publishing alerts and small messages in pub/sub style that I searched around for a good dashboard app for the desktop and then I found Node-RED. Wow. Node-RED is very impressive free software. Thank you MQTT and Node-RED!

Quote:
Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways.

It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.]



Node-RED provides a browser-based flow editor that makes it easy to wire together flows using the wide range of nodes in the palette. Flows can be then deployed to the runtime in a single-click.

JavaScript functions can be created within the editor using a rich text editor.
But of course, Node-RED is for folks comfortable with Node.js, NPM, Javascript; so if you are comfortable with at least those three technologies, getting Node-RED up and running quickly should be a piece of cake, otherwise YMMV.

Quote:
The light-weight runtime is built on Node.js, taking full advantage of its event-driven, non-blocking model. This makes it ideal to run at the edge of the network on low-cost hardware such as the Raspberry Pi as well as in the cloud.

With over 225,000 modules in Node's package repository, it is easy to extend the

A built-in library allows you to save useful functions, templates or flows for re-use.
Taking a break from my Arduino testing and proof-of-concept projects, I installed the following (with three MQTT servers / brokers already up and running). These node manager packages are easily found, well maintained and easy to install (easy to Google). Just make sure you install each component in your main Node-Red directory. Also, if you get confused on where to edit the few config files (based on your installation - mine was on macOS), make sure you look at your terminal logs when you start Node-RED. It's all there, the paths to settings for the js files, etc.

So, I installed these three packages:

After about a hour of getting used to the layout and core features, reading some online docs when I got stuck (the usual way, dive right in and Google when you hit a roadblock), I got the following dashboard working, mostly flawlessly. Keep in mind that was my first attempt, so that's really impressive software! I may do a video on this combo, MQTT and Node-RED.

Using Node-RED and MQTT to Monitor Server and Application Stats-screen-shot-2020-02-04-94249-pmjpg


Here is what my first Node-RED dashboard project looked like (so far):

Using Node-RED and MQTT to Monitor Server and Application Stats-screen-shot-2020-02-04-100459-pmjpg


It's funny. The part I like the best at the moment is the female Australian voice announcing the total number of online users every minute (hope my wife does not get jealous!).

Anyway, this It's just great! In fact, between MQTT and Node-RED, this is a truly life changing event, haha. MQTT opens up every device to talk to every other device. I had no idea there was such great, free pub/sub software available which can be set up on a server, up and running, in five minutes. This is truly remarkable.

You can see from the time series graph how bots can really load down the site:

Using Node-RED and MQTT to Monitor Server and Application Stats-screen-shot-2020-02-04-100836-pmjpg


But I have some code which sends the bots to the "403 bit bucket" when the load goes crazy; and I adjusted them down and a few minutes later:

Using Node-RED and MQTT to Monitor Server and Application Stats-screen-shot-2020-02-04-101511-pmjpg


In fact, as I was writing this post.... when the nice Australian lady announced "UNIX LOAD AVERAGE IS 181", and combined with the Node-RED time series graph, I adjusted a few parameters and all went back to normal.

There is one more MQTT package from NASA , called the NASA Open MCT. Have you heard of it? Building an IoT dashboard with NASA Open MC . I will try this package and see how it works before getting back into Arduino modules and programming small devices. Then I'll use MQTT on the ESP devices (and maybe on my NB-IoT shield as well) to send sensor data from various modules to my MQTT broker and display in some nice dashboard widgets.

All in a days work!

Cheers!
# 2  
Old 02-05-2020
Expanded the dashboard a bit, and found the time series data helpful in tracking down some nagging performance issues.

The time series charts in these screen shots illustrate two brief performance hits. Since these hits happen at the top of the hours, I looked carefully a the application scheduled tasks and found a number of "clean up" tasks which were not an issue with the DB was small, but with a large DB these "clean up" hourly scheduled tasks can slow the site down.

So, I moved around five clean-up tasks to Saturday and Sunday, since moving these batch jobs to the weekend will not effect the application very much, but does improve performance quite a bit.

Using Node-RED and MQTT to Monitor Server and Application Stats-screen-shot-2020-02-05-54437-pmjpg


Using Node-RED and MQTT to Monitor Server and Application Stats-screen-shot-2020-02-05-54333-pmjpg
# 3  
Old 02-05-2020
Well, those changes did not help much.... looks like here is some "underground" process running every three hours causing the spikes:

Using Node-RED and MQTT to Monitor Server and Application Stats-screen-shot-2020-02-05-72457-pmjpg


I checked all the forum software and all the cron files on the server, but so far, no joy in finding the process causing the intermittent performance hit.
Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. Programming

Node-RED: Writing MQTT Messages to MySQL DB with UNIX timestamp

First, I want to thank Neo (LOL) for this post from 2018, Node.js and mysql - ER_ACCESS_DENIED_ERROR I could not get the Node-RED mysql module to work and searched Google until all my links were purple! I kept getting ER_ACCESS_DENIED_ERROR with the right credentials. Nothing on the web was... (0 Replies)
Discussion started by: Neo
0 Replies

2. Programming

Publish and Subscribe to AES-256 Encrypted MQTT Messages to Node-RED from PHP Scripts

Various Node-Red crypto modules do not work with PHP, so to send an encrypted message from a PHP script (in this case from a Ubuntu server) to Node-RED we need our own code. After a few hours of searching, testing various libs, more testing and debugging, I got this PHP to Node-RED code... (0 Replies)
Discussion started by: Neo
0 Replies

3. Programming

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages

Here we go.... Preface: ..... so in a galaxy far, far, far away from commercial, data sharing corporations..... For this project, I used the ESP-WROOM-32 as an MQTT (publish / subscribe) client which receives Linux server "load averages" as messages published as MQTT pub/sub messages.... (6 Replies)
Discussion started by: Neo
6 Replies

4. Red Hat

Red Hat application server ssl keystore problem

A client is accessing our JBoss server. In the past, we set up a keystore and everything worked fine. That certificat expired and we've installed the new one. Now the client is getting the following error - HTTP/1.1 500 Internal Server Error Date: Mon, 14 Apr 2014 13:25:44 GMT Server:... (1 Reply)
Discussion started by: kkinney
1 Replies

5. Homework & Coursework Questions

Accessing one UNIX node from another node of the same server

Hi Experts, I am in need of running a script from one node say node 1 via node 2. My scheduling tool dont have access to node2 , so i need to invoke the list file from node1 but the script needs to run from node2. because the server to which i am hitting, is having access only for the node... (5 Replies)
Discussion started by: arun1377
5 Replies

6. AIX

powerha application monitor restart counter?

I have configured a custom application monitor with restart count = 3. say the application has restarted 2 times, if the application fail 2 more times within restart interval, it will failover. Is there any counter to tell me the recent restart count? Thanks!! (1 Reply)
Discussion started by: skeyeung
1 Replies

7. AIX

How to add node to the cluster with stoping the application?

In the production hacmp 5.3 is running with three nodes but i want to add new node to cluster with out stopping the application ie with same resource group if any bady know pls help me. (1 Reply)
Discussion started by: manjunath.m
1 Replies
Login or Register to Ask a Question