Login or Register to Ask a Question and Join Our Community


Programming

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Programming ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages
# 1  
Old 02-02-2020
ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages
Here we go....

Preface:

Quote:
Rant: After a "very interesting" experience back-and-forth with Blynk yesterday over their "pull money out of your pocket for every tiny widget" business model when I was publishing a quick public service app to help folks in China regarding the Wuhan coronavirus, in also combination with the Blynk "terms of service" which permits Blynk to share all of a user's network data, every message, all user mined behavioral analytical "raw material" and share that data with any and all third parties; I decided to build my own messaging network to test these ESP32.......
..... so in a galaxy far, far, far away from commercial, data sharing corporations.....

For this project, I used the ESP-WROOM-32 as an MQTT (publish / subscribe) client which receives Linux server "load averages" as messages published as MQTT pub/sub messages. Here is the Arduino IDE code I wrote for the MQTT client running on the ESP32:

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-s20200201_012jpg



ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-img_9116jpg


Code:
/*****************************************************************
 *  ESP32 (ESP-WROOM-32) MQTT Client Test  v0.1
 *  The ESP32 subscribes to MQTT messages from an MQTT server
 *  published by a Linux server. In this example, server load 
 *  average messages.  The MQTT server requires username and password
 *  for basic authentication security.
 * 
 *  Neo  www.unix.com  v0.1  2 Feb 2020
 *  You are free to use this code anyway imaginable
 *  for your ESP32 MQTT projects.
 *****************************************************************/

#include <WiFi.h>
#include <PubSubClient.h>
#define BAUD_RATE 115200
#define DEBUG false
const char *ssid = "YOUR_WIFI_SSID";
const char *password = "YOUR_WIFI_PASSWORD";
const char *mqttServer = "YOUR_MQTT_SERVER";
const int mqttPort = 1883;                      // YOUR_MQTT_PORT
const char *mqttUser = "YOUR_MQTT_SERVER_USER"; // Basic user and password authentication
const char *mqttPassword = YOUR_MQTT_SERVER_USER_PASSWORD "; // Basic user and password authentication
    const int lengthTopic = 16; //required for the if-elseif message decoding based on topic
WiFiClient espClient;
PubSubClient client(espClient);

// The code below is self documenting, I think

void callback(char *topic, byte *payload, unsigned int length)
{
    float loadvar, loadvar0, loadvar5, loadvar15;
    if (DEBUG)
    {
        Serial.print("Message arrived in topic: ");
        Serial.println(topic);
        Serial.print("Message: ");
    }
    String w;
    String receivedTopic = String(topic);
    if (DEBUG)
    {
        int len = receivedTopic.length();
        Serial.print("Length :");
        Serial.println(String(len));
    }
    for (int i = 0; i < length; i++)
    {
        w = w + String((char)payload[i]);
    }

    if (receivedTopic.indexOf("loads/l1") > 0 and receivedTopic.length() < lengthTopic)
    {
        loadvar0 = w.toFloat();
        Serial.printf("LOAD  1 ... %s \n", w);
    }
    else if (receivedTopic.indexOf("loads/l5") > 0 and receivedTopic.length() < lengthTopic)
    {
        loadvar0 = w.toFloat();
        Serial.printf("LOAD  5 ... %s \n", w);
    }
    else if (receivedTopic.indexOf("loads/l15") > 0)
    {
        loadvar0 = w.toFloat();
        Serial.printf("LOAD 15 ... %s \n", w);
    }
    else
    {
        loadvar = w.toFloat();
        Serial.printf("Message: %s \n", w);
    }

    if (DEBUG)
    {
        Serial.println("-----------------------");
    }
}

void setup()
{

    Serial.begin(BAUD_RATE);
    WiFi.begin(ssid, password);

    while (WiFi.status() != WL_CONNECTED)
    {
        delay(500);
        Serial.printf("Connecting to WiFi: %s \n", ssid);
    }

    Serial.printf("Connected to the WiFi network: %s \n", ssid);
    client.setServer(mqttServer, mqttPort);
    client.setCallback(callback);

    while (!client.connected())
    {
        Serial.printf("Connecting to MQTT Server: %s:%d\n", mqttServer, mqttPort);

        if (client.connect("ESP32Client", mqttUser, mqttPassword))
        {
            Serial.println("Connected!");
        }
        else
        {
            Serial.print("failed with state ");
            Serial.print(client.state());
            delay(2000);
        }
    }

    client.subscribe("server/loads");
    client.subscribe("server/loads/l1");
    client.subscribe("server/loads/l5");
    client.subscribe("server/loads/l15");
}

void loop()
{
    client.loop();
}

Crontab entry on Linux:

Code:
* * * * * /usr/bin/nice -n 19 php /usr/local/bin/loads_mqtt.php > /dev/null 2>&1

Here is the crontab PHP file which publishes the server load average messages:

Code:
<?php
/********************************************************
 * Linux PHP Script to Publish Server Load Averges v0.1
 * Simple Script using MQTT to publish messages used
 * in ESP32 test.
 *
 * Neo www.unix.com 2 Feb 2020
 * You are free to use this as you find useful.
 ********************************************************/

$load = shell_exec('cat /proc/loadavg');
$pieces = explode(" ", $load);
define('SEND_TEXT', false);

if (SEND_TEXT) {
    $l1 = "LOAD 1: " . $pieces[0];
    $l5 = "LOAD 5: " . $pieces[1];
    $l15 = "LOAD 15: " . $pieces[2];
} else {
    $l1 = $pieces[0];
    $l5 = $pieces[1];
    $l15 = $pieces[2];
}

$c1 = '/usr/bin/mosquitto_pub -t server/loads/l1 -m "' . $l1 . '" -q 1 -u YOUR_USER -P YOUR_USER_PASSWORD';
$c2 = '/usr/bin/mosquitto_pub -t server/loads/l5 -m "' . $l5 . '" -q 1 -u YOUR_USER -P YOUR_USER_PASSWORD';
$c3 = '/usr/bin/mosquitto_pub -t server/loads/l15 -m "' . $l15 . '" -q 1 -u YOUR_USER -P YOUR_USER_PASSWORD';
echo $c1 . "\n";
$output = shell_exec($c1);
sleep(1);
$output = shell_exec($c2);
sleep(1);
$output = shell_exec($c3);

Here is the sample output in the Arduino IDE serial monitor:

Code:
16:08:05.726 -> Connecting to WiFi: YOUR_WIFI_SSID
16:08:06.226 -> Connecting to WiFi: YOUR_WIFI_SSID
16:08:06.739 -> Connecting to WiFi: YOUR_WIFI_SSID
16:08:07.244 -> Connecting to WiFi: YOUR_WIFI_SSID
16:08:07.724 -> Connecting to WiFi: YOUR_WIFI_SSID
16:08:08.226 -> Connecting to WiFi: YOUR_WIFI_SSID
16:08:08.226 -> Connected to the WiFi network: TIM_WIFI 
16:08:08.263 -> Connecting to MQTT Server: www.your_mqtt_server.com:1883
16:08:09.826 -> connected
16:09:02.238 -> LOAD  1 ... 1.31 
16:09:03.162 -> LOAD  5 ... 3.74 
16:09:04.189 -> LOAD 15 ... 3.78 
16:10:02.252 -> LOAD  1 ... 0.96 
16:10:03.262 -> LOAD  5 ... 3.20 
16:10:04.211 -> LOAD 15 ... 3.59 
16:11:04.808 -> LOAD  1 ... 0.99 
16:11:04.808 -> LOAD  5 ... 2.80 
16:11:04.808 -> LOAD 15 ... 3.42 
16:12:09.545 -> LOAD  1 ... 1.01 
16:12:09.545 -> LOAD  5 ... 2.47 
16:12:09.545 -> LOAD 15 ... 3.27 
16:13:12.122 -> LOAD  1 ... 0.89 
16:13:12.122 -> LOAD  5 ... 2.18 
16:13:12.122 -> LOAD 15 ... 3.12 
16:14:02.504 -> LOAD  1 ... 0.90 
16:14:02.689 -> LOAD  5 ... 1.95 
16:14:04.953 -> LOAD 15 ... 2.98 
16:15:08.289 -> LOAD  1 ... 0.71 
16:15:08.289 -> LOAD  5 ... 1.71 
16:15:08.289 -> LOAD 15 ... 2.83 
16:16:05.209 -> LOAD  1 ... 0.48 
16:16:05.209 -> LOAD  5 ... 1.47 
16:16:05.209 -> LOAD 15 ... 2.68 
16:17:11.766 -> LOAD  1 ... 0.43 
16:17:11.766 -> LOAD  5 ... 1.27 
16:17:11.766 -> LOAD 15 ... 2.53 
16:18:01.881 -> LOAD  1 ... 0.41 
16:18:02.928 -> LOAD  5 ... 1.11 
16:18:07.767 -> LOAD 15 ... 2.39 
16:19:13.309 -> LOAD  1 ... 0.41 
16:19:13.309 -> LOAD  5 ... 0.98 
16:19:13.309 -> LOAD 15 ... 2.27 
16:20:05.301 -> LOAD  1 ... 0.69 
16:20:05.301 -> LOAD  5 ... 0.98 
16:20:05.301 -> LOAD 15 ... 2.18 
16:21:05.305 -> LOAD  1 ... 0.78 
16:21:05.305 -> LOAD  5 ... 0.95 
16:21:05.305 -> LOAD 15 ... 2.10 
16:22:02.641 -> LOAD  1 ... 0.69 
16:22:02.853 -> LOAD  5 ... 0.91 
16:22:03.877 -> LOAD 15 ... 2.01 
16:23:07.344 -> LOAD  1 ... 0.90 
16:23:07.344 -> LOAD  5 ... 0.94 
16:23:07.344 -> LOAD 15 ... 1.95 
16:24:01.310 -> LOAD  1 ... 0.70 
16:24:02.331 -> LOAD  5 ... 0.87 
16:24:03.306 -> LOAD 15 ... 1.87 
16:25:01.924 -> LOAD  1 ... 0.66 
16:25:02.765 -> LOAD  5 ... 0.81 
16:25:03.785 -> LOAD 15 ... 1.78 
16:26:09.173 -> LOAD  1 ... 0.69 
16:26:09.173 -> LOAD  5 ... 0.80 
16:26:09.173 -> LOAD 15 ... 1.72 
16:27:01.644 -> LOAD  1 ... 0.71 
16:27:04.816 -> LOAD  5 ... 0.79 
16:27:04.816 -> LOAD 15 ... 1.65 
16:28:04.723 -> LOAD  1 ... 0.90 
16:28:04.723 -> LOAD  5 ... 0.82 
16:28:04.723 -> LOAD 15 ... 1.60 
16:29:08.297 -> LOAD  1 ... 0.89 
16:29:08.297 -> LOAD  5 ... 0.82 
16:29:08.297 -> LOAD 15 ... 1.55 
16:30:02.891 -> LOAD  1 ... 0.61 
16:30:02.891 -> LOAD  5 ... 0.75 
16:30:03.875 -> LOAD 15 ... 1.48

So far, no joy in finding a good IOS app for the iPhone to display this data in a nice dashboard.

I did find an OK "MQTT Terminal App" for the iPhone and I did confirm I can subscribe to these messages with MQTT on the iPhone. Unfortunately, all the "pretty dashboard apps for MQTT" I tried in the app store has some problems and none worked well "out of the box". However, the simple MQTT terminal apps work fine (but are not "pretty")

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-mqtt_terminal_appjpeg


The MQTT terminal app I tested, which managed to work, does not add line breaks at the end of a message, and when I tried to send line breaks over MQTT, the clients stopped receiving the messages, which is expected since these pub/sub client/servers are not really meant for formatted data. Maybe I'm missing something small?

Also refer to:

Install MQTT Broker on Ubuntu 18.04 & 14.04

to install the MQTT broker on Ubuntu...

I'm still searching for a "pretty dashboard" app for IOS which will display the MQTT data, but so far, nothing has worked well.

Will update this thread if I find an MQTT dashboard app for IOS which "works" OK Smilie
# 2  
Old 02-02-2020
Here is an IOS app which is a kind of "debugger" for MQTT messages; and works "OK, more or less" to prove things are working; but it is very far from a "pretty" dashboard app:

MQTTool for IOS

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-img_9118jpg


Still on the hunt for a "pretty" dashboard app. I could not get IOT-ON-OFF to work (will not receive any messages), even thought it did look promising!

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-screen-shot-2020-02-02-60836-pmjpg
# 3  
Old 02-02-2020
Well, it's not really "pretty" yet (as a dashboard) but MQTTool for IOS works, "not bad"... and it's totally free.

I changed the PHP script around a bit to publish all the "Load Averages" (in one message) and all the "Wuhan Coronavirus Stats" (in one message) and subscribed to both of them. In the "Load Average" app, there is the one, five and fifteen minute server load averages (very handy if managing a server, obviously) and in the "Wuhan" app, there are the C (confirmed cases), the S (suspected cases), the D (deaths) and R (recovered cases). You can easily start to image the myriad way to use MQTT for you own private "things" network from this short tutorial, I think.

BTW, I am still searching for a good "dashboard" app for MQTT (for either IOS or macOS), but no joy yet.

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-img_70bca7805208-1jpeg


When you think about it. MQTT is "the only way to go" if you are developing these kinds of private "IoT" app, every devices can be both a publisher and a subscribe, and if you run your own private MQTT server, you do not have to worry about a third parties mining your data.
This User Gave Thanks to Neo For This Post:
vbe
# 4  
Old 02-02-2020
Update:

After a lot of "messing around" with every combination of settings under the sun, I finally got the IOT-ON-OFF IOS Dashboard app working. Success. It's not a "pretty" as Blynk, but in many ways it is much better. For example we can easily switch between dashboards (see image below) and we don't have to pay for "widgets" foreach new app; and with a one time, very small purchase, we can get UNLIMITED widgets and dashboards. Here are some samples from my two previous mini-apps, (1) Wuhan Coronavirus Status and (2) Server Load Averages, all done with MQTT.

It's really quite remarkable if you think about it. I've built these two apps from scratch in two days, and that included wasting a lot of time evaluating and bumping heads with Blynk yesterday (and the headaches that caused).

Later I will summarize my notes on the "tricks" to get the IOT-ON-OFF app working so you don't have to waste your time fiddling around with all the settings. This iPhone dashboard app is not really "super polished" but it's not bad at all.



WUHAN CORONAVIRUS STATS:
[
ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-img_724c56545d50-1jpeg


SERVER LOAD AVERAGES:
ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-img_9122jpg


SWITCHING BETWEEN DASHBOARDS
ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-img_d6b1cb7865ca-1jpeg


SERVER LOAD AVERAGES:
ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-img_9125jpg
This User Gave Thanks to Neo For This Post:
vbe
# 5  
Old 02-02-2020
By the way, I just purchased IoT-OnOff from the Apple app sure, the premium version with unlimited widgets for $2.99 USD. Compare that to Blynk who needed about $1.99 to add only one additional widget!

For everyone who wants a decent MQTT dashboard app with unlimited widgets for about $3 USD, here is the info (however, it is not user friendly to set up, LOL):

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-img_9127jpg
# 6  
Old 02-02-2020
Update:

Current screen shots of this test project, using only MQTT and the IOS IoT-OnOff app.

Server Dashboard App (Will add more to this later):

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-img_0462jpg


Wuhan Corona Virus Stats Tracking App:

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-img_0463jpg
# 7  
Old 02-03-2020
MQTT server data on IoT OnOff app (only using MQTT and iPhone and not using any external board for this);

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages-img_0477jpg
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Nearly Random, Uncorrelated Server Load Average Spikes

I have been wrangling with a small problem on a Ubuntu server which runs a LAMP application. Linux ubuntu 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux This server runs fine, basically: ubuntu:/var/www# uptime 20:17:13 up 105 days,... (45 Replies)
Discussion started by: Neo
45 Replies

2. Programming

A Slightly Better NTP Client for the ESP32 (ESPWROOM32) on macOS Catalina

Currently have two ESP8266 modules testing some Blynk apps, whereI'm not so happy with the Blynk business model for developers, but that's another story. So, with two of my ESP8266s currently "busy", I decided to work on the ESP32, and in particular the ESPWROOM32. I installed the... (0 Replies)
Discussion started by: Neo
0 Replies

3. UNIX for Dummies Questions & Answers

Help with load average?

how load average is calculated and what exactly is it difference between cpu% and load average (9 Replies)
Discussion started by: robo
9 Replies

4. UNIX for Dummies Questions & Answers

linux tomcat load average issue

Hello I am running a liferay application in tomcat on one of my hosted linux machine having 4GB of ram and 1 CPU. I get the tomcat response really slow and with much digging I found that the cpu might be loaded and can be tracked with top command. The following is the output of top command:... (0 Replies)
Discussion started by: dhavaln
0 Replies

5. UNIX for Dummies Questions & Answers

Please Help me in my load average

Hello AlL,.. I want from experts to help me as my load average is increased and i dont know where is the problem !! this is my top result : root@a4s # top top - 11:30:38 up 40 min, 1 user, load average: 3.06, 2.49, 4.66 Mem: 8168788k total, 2889596k used, 5279192k free, 47792k... (3 Replies)
Discussion started by: black-code
3 Replies

6. Linux

how to setup a virtual IP to control 2 server load for linux

Hi anyone know how to setup a setup a virtual IP to control 2 server load for linux? i only have 2 server, i don want to buy another just for the load balance... is there a way to do it? Sumemr (0 Replies)
Discussion started by: summerpeh
0 Replies

7. UNIX for Dummies Questions & Answers

Load Average

Hello all, I have a question about load averages. I've read the man pages for the uptime and w command for two or three different flavors of Unix (Red Hat, Tru64, Solaris). All of them agree that in the output of the 2 aforementioned commands, you are given the load average for the box, but... (3 Replies)
Discussion started by: Heathe_Kyle
3 Replies

8. UNIX for Advanced & Expert Users

load average

we have an unix system which has load average normally about 20. but while i am running a particular unix batch which performs heavy operations on filesystem and database average load reduces to 15. how can we explain this situation? while running that batch idle cpu time is about %60-65... (0 Replies)
Discussion started by: gfhgfnhhn
0 Replies

9. Cybersecurity

linux prog for server space, load etc

Hi all what are the ways by which we can know and generate a report of the space remaining, memory(ram) used and the load on the server over a period of time. (1 Reply)
Discussion started by: arlan
1 Replies

10. UNIX for Advanced & Expert Users

Server load (Unix/Linux, Redhat, CPanel)

Hello, I'm facing a big problem with my hosting server (Dual Xeon 2.4GHz), I'm having a load in the CPU usage and the memory (maybe it's related) ALSO mySQL: Server Load 5.34 (2 cpus) (to 22 sometime) Memory Used 68.4 % (to 70% sometime) When I go to 'CPU/Memory/MySQL Usage' I found: ... (3 Replies)
Discussion started by: Kh@lid
3 Replies
Login or Register to Ask a Question