Sponsored Content
Top Forums UNIX for Advanced & Expert Users Nearly Random, Uncorrelated Server Load Average Spikes Post 303044138 by Neo on Saturday 15th of February 2020 02:28:08 AM
Old 02-15-2020
Next:

I have some old "cyberspace situational awareness" PHP code I used for a visualization project a few years ago, which captures and stores details information on web session activity; this code has proven handy identifying rouge bots in the past.

So, I have modified that code to capture and store detailed session information, including the number of hits per IP address, the user agent string, country code, etc. when the 1 minute load average is above 20 and less than 50.

Code:
$theload = getLoadAvg();
if (floatval($theload) > 20.0 && floatval($theload) < 50.0) 
{
  /// the old CSA code to parse web session activity and store the results in the DB
}

So, let's see what happens the next time we get a spike... this should be interesting.


Code:
mysql> describe neo_csa_session_manager;
+--------------+------------------+------+-----+---------+----------------+
| Field        | Type             | Null | Key | Default | Extra          |
+--------------+------------------+------+-----+---------+----------------+
| id           | int(11) unsigned | NO   | PRI | NULL    | auto_increment |
| user_id      | int(11)          | NO   | MUL | 0       |                |
| session_id   | varchar(255)     | NO   |     | NULL    |                |
| url          | text             | NO   |     | NULL    |                |
| ip_address   | varchar(45)      | NO   | MUL | NULL    |                |
| user_agent   | varchar(255)     | NO   |     | NULL    |                |
| bot_flag     | tinyint(1)       | NO   |     | 0       |                |
| robot_txt    | mediumint(6)     | NO   |     | 0       |                |
| sitemap      | mediumint(6)     | NO   |     | 0       |                |
| riskscore    | int(11)          | NO   |     | 0       |                |
| country_iso2 | varchar(2)       | NO   |     | UN      |                |
| country      | varchar(50)      | NO   |     | UNKNOWN |                |
| hitcount     | int(10) unsigned | NO   |     | 1       |                |
| firstseen    | bigint(11)       | NO   |     | NULL    |                |
| unixtime     | bigint(11)       | YES  |     | NULL    |                |
| longitude    | float            | NO   |     | 0       |                |
| latitude     | float            | NO   |     | 0       |                |
+--------------+------------------+------+-----+---------+----------------+
17 rows in set (0.00 sec)

This User Gave Thanks to Neo For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

load average

we have an unix system which has load average normally about 20. but while i am running a particular unix batch which performs heavy operations on filesystem and database average load reduces to 15. how can we explain this situation? while running that batch idle cpu time is about %60-65... (0 Replies)
Discussion started by: gfhgfnhhn
0 Replies

2. UNIX for Dummies Questions & Answers

Load Average

Hello all, I have a question about load averages. I've read the man pages for the uptime and w command for two or three different flavors of Unix (Red Hat, Tru64, Solaris). All of them agree that in the output of the 2 aforementioned commands, you are given the load average for the box, but... (3 Replies)
Discussion started by: Heathe_Kyle
3 Replies

3. UNIX for Dummies Questions & Answers

top - Load average

Hello, Here is the output of top command. My understanding here is, the load average 0.03 in last 1 min, 0.02 is in last 5 min, 0.00 is in last 15 min. By seeing this load average, When can we say that, the system load averge is too high? When can we say that, load average is medium/low??... (8 Replies)
Discussion started by: govindts
8 Replies

4. Solaris

load average query.

Hi, i have installed solaris 10 on t-5120 sparc enterprise. I am little surprised to see load average of 2 or around on this OS. when checked with ps command following process is using highest CPU. looks like it is running for long time and does not want to stop, but I do not know... (5 Replies)
Discussion started by: upengan78
5 Replies

5. UNIX for Dummies Questions & Answers

Please Help me in my load average

Hello AlL,.. I want from experts to help me as my load average is increased and i dont know where is the problem !! this is my top result : root@a4s # top top - 11:30:38 up 40 min, 1 user, load average: 3.06, 2.49, 4.66 Mem: 8168788k total, 2889596k used, 5279192k free, 47792k... (3 Replies)
Discussion started by: black-code
3 Replies

6. UNIX for Advanced & Expert Users

Load average in UNIX

Hi , I am using 48 CPU sunOS server at my work. The application has facility to check the current load average before starting a new process to control the load. Right now it is configured as 48. So it does mean that each CPU can take maximum one proces and no processe is waiting. ... (2 Replies)
Discussion started by: kumaran_5555
2 Replies

7. Solaris

Load Average and Lwps

NPROC USERNAME SWAP RSS MEMORY TIME CPU 320 oracle 23G 22G 69% 582:55:11 85% 47 root 148M 101M 0.3% 99:29:40 0.3% 53 rafmsdb 38M 60M 0.2% 0:46:17 0.1% 1 smmsp 1296K 5440K 0.0% 0:00:08 0.0% 7 daemon ... (2 Replies)
Discussion started by: snjksh
2 Replies

8. UNIX for Dummies Questions & Answers

Load average spikes once an hour

Hi, I am getting a high load average, around 7, once an hour. It last for about 4 minutes and makes things fairly unusable for this time. How do I find out what is using this. Looking at top the only thing running at the time is md5sum. I have looked at the crontab and there is nothing... (10 Replies)
Discussion started by: sm9ai
10 Replies

9. UNIX for Dummies Questions & Answers

Help with load average?

how load average is calculated and what exactly is it difference between cpu% and load average (9 Replies)
Discussion started by: robo
9 Replies

10. Programming

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages

Here we go.... Preface: ..... so in a galaxy far, far, far away from commercial, data sharing corporations..... For this project, I used the ESP-WROOM-32 as an MQTT (publish / subscribe) client which receives Linux server "load averages" as messages published as MQTT pub/sub messages.... (6 Replies)
Discussion started by: Neo
6 Replies
GAMMU-SMSD-MYSQL(7)						       Gammu						       GAMMU-SMSD-MYSQL(7)

NAME
gammu-smsd-mysql - gammu-smsd(1) backend using MySQL database server as a message storage DESCRIPTION
MYSQL backend stores all data in a MySQL database server, which parameters are defined by configuration (see gammu-smsdrc for description of configuration options). For tables description see gammu-smsd-tables. This backend is based on gammu-smsd-sql. CONFIGURATION
Before running gammu-smsd you need to create necessary tables in the database, which is described bellow. The configuration file then can look like: [smsd] service = sql driver = native_mysql host = localhost See also gammu-smsdrc PRIVILEGES
The user accessing the database does not need much privileges, the following privleges should be enough: GRANT USAGE ON *.* TO 'smsd'@'localhost' IDENTIFIED BY 'password'; GRANT SELECT, INSERT, UPDATE, DELETE ON `smsd`.* TO 'smsd'@'localhost'; Note For creating the SQL tables you need more privileges, especially for creating triggers, which are used for some functionality. CREATING TABLES
SQL script for creating tables in MySQL database: -- phpMyAdmin SQL Dump -- version 2.8.0.3 -- http://www.phpmyadmin.net -- -- Host: localhost -- Generation Time: Jun 10, 2006 at 11:08 PM -- Server version: 5.0.18 -- PHP Version: 5.1.3 -- -- Database: `smsd` -- -- -------------------------------------------------------- -- -- Table structure for table `daemons` -- CREATE TABLE `daemons` ( `Start` text NOT NULL, `Info` text NOT NULL ) ENGINE=MyISAM DEFAULT CHARSET=utf8; -- -- Dumping data for table `daemons` -- -- -------------------------------------------------------- -- -- Table structure for table `gammu` -- CREATE TABLE `gammu` ( `Version` integer NOT NULL default '0' ) ENGINE=MyISAM DEFAULT CHARSET=utf8; -- -- Dumping data for table `gammu` -- INSERT INTO `gammu` (`Version`) VALUES(13); -- -------------------------------------------------------- -- -- Table structure for table `inbox` -- CREATE TABLE `inbox` ( `UpdatedInDB` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, `ReceivingDateTime` timestamp NOT NULL default '0000-00-00 00:00:00', `Text` text NOT NULL, `SenderNumber` varchar(20) NOT NULL default '', `Coding` enum('Default_No_Compression','Unicode_No_Compression','8bit','Default_Compression','Unicode_Compression') NOT NULL default 'Default_No_Compression', `UDH` text NOT NULL, `SMSCNumber` varchar(20) NOT NULL default '', `Class` integer NOT NULL default '-1', `TextDecoded` text NOT NULL default '', `ID` integer unsigned NOT NULL auto_increment, `RecipientID` text NOT NULL, `Processed` enum('false','true') NOT NULL default 'false', PRIMARY KEY `ID` (`ID`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ; -- -- Dumping data for table `inbox` -- -- -------------------------------------------------------- -- -- Table structure for table `outbox` -- CREATE TABLE `outbox` ( `UpdatedInDB` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, `InsertIntoDB` timestamp NOT NULL default '0000-00-00 00:00:00', `SendingDateTime` timestamp NOT NULL default '0000-00-00 00:00:00', `SendBefore` time NOT NULL DEFAULT '23:59:59', `SendAfter` time NOT NULL DEFAULT '00:00:00', `Text` text, `DestinationNumber` varchar(20) NOT NULL default '', `Coding` enum('Default_No_Compression','Unicode_No_Compression','8bit','Default_Compression','Unicode_Compression') NOT NULL default 'Default_No_Compression', `UDH` text, `Class` integer default '-1', `TextDecoded` text NOT NULL default '', `ID` integer unsigned NOT NULL auto_increment, `MultiPart` enum('false','true') default 'false', `RelativeValidity` integer default '-1', `SenderID` varchar(255), `SendingTimeOut` timestamp NULL default '0000-00-00 00:00:00', `DeliveryReport` enum('default','yes','no') default 'default', `CreatorID` text NOT NULL, PRIMARY KEY `ID` (`ID`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8; CREATE INDEX outbox_date ON outbox(SendingDateTime, SendingTimeOut); CREATE INDEX outbox_sender ON outbox(SenderID); -- -- Dumping data for table `outbox` -- -- -------------------------------------------------------- -- -- Table structure for table `outbox_multipart` -- CREATE TABLE `outbox_multipart` ( `Text` text, `Coding` enum('Default_No_Compression','Unicode_No_Compression','8bit','Default_Compression','Unicode_Compression') NOT NULL default 'Default_No_Compression', `UDH` text, `Class` integer default '-1', `TextDecoded` text default NULL, `ID` integer unsigned NOT NULL default '0', `SequencePosition` integer NOT NULL default '1', PRIMARY KEY (`ID`, `SequencePosition`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8; -- -- Dumping data for table `outbox_multipart` -- -- -------------------------------------------------------- -- -- Table structure for table `pbk` -- CREATE TABLE `pbk` ( `ID` integer NOT NULL auto_increment, `GroupID` integer NOT NULL default '-1', `Name` text NOT NULL, `Number` text NOT NULL, PRIMARY KEY (`ID`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8; -- -- Dumping data for table `pbk` -- -- -------------------------------------------------------- -- -- Table structure for table `pbk_groups` -- CREATE TABLE `pbk_groups` ( `Name` text NOT NULL, `ID` integer NOT NULL auto_increment, PRIMARY KEY `ID` (`ID`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ; -- -- Dumping data for table `pbk_groups` -- -- -------------------------------------------------------- -- -- Table structure for table `phones` -- CREATE TABLE `phones` ( `ID` text NOT NULL, `UpdatedInDB` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, `InsertIntoDB` timestamp NOT NULL default '0000-00-00 00:00:00', `TimeOut` timestamp NOT NULL default '0000-00-00 00:00:00', `Send` enum('yes','no') NOT NULL default 'no', `Receive` enum('yes','no') NOT NULL default 'no', `IMEI` varchar(35) NOT NULL, `Client` text NOT NULL, `Battery` integer NOT NULL DEFAULT -1, `Signal` integer NOT NULL DEFAULT -1, `Sent` int NOT NULL DEFAULT 0, `Received` int NOT NULL DEFAULT 0, PRIMARY KEY (`IMEI`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8; -- -- Dumping data for table `phones` -- -- -------------------------------------------------------- -- -- Table structure for table `sentitems` -- CREATE TABLE `sentitems` ( `UpdatedInDB` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, `InsertIntoDB` timestamp NOT NULL default '0000-00-00 00:00:00', `SendingDateTime` timestamp NOT NULL default '0000-00-00 00:00:00', `DeliveryDateTime` timestamp NULL, `Text` text NOT NULL, `DestinationNumber` varchar(20) NOT NULL default '', `Coding` enum('Default_No_Compression','Unicode_No_Compression','8bit','Default_Compression','Unicode_Compression') NOT NULL default 'Default_No_Compression', `UDH` text NOT NULL, `SMSCNumber` varchar(20) NOT NULL default '', `Class` integer NOT NULL default '-1', `TextDecoded` text NOT NULL default '', `ID` integer unsigned NOT NULL default '0', `SenderID` varchar(255) NOT NULL, `SequencePosition` integer NOT NULL default '1', `Status` enum('SendingOK','SendingOKNoReport','SendingError','DeliveryOK','DeliveryFailed','DeliveryPending','DeliveryUnknown','Error') NOT NULL default 'SendingOK', `StatusError` integer NOT NULL default '-1', `TPMR` integer NOT NULL default '-1', `RelativeValidity` integer NOT NULL default '-1', `CreatorID` text NOT NULL, PRIMARY KEY (`ID`, `SequencePosition`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8; CREATE INDEX sentitems_date ON sentitems(DeliveryDateTime); CREATE INDEX sentitems_tpmr ON sentitems(TPMR); CREATE INDEX sentitems_dest ON sentitems(DestinationNumber); CREATE INDEX sentitems_sender ON sentitems(SenderID); -- -- Dumping data for table `sentitems` -- -- -- Triggers for setting default timestamps -- DELIMITER // CREATE TRIGGER inbox_timestamp BEFORE INSERT ON inbox FOR EACH ROW BEGIN IF NEW.ReceivingDateTime = '0000-00-00 00:00:00' THEN SET NEW.ReceivingDateTime = CURRENT_TIMESTAMP(); END IF; END;// CREATE TRIGGER outbox_timestamp BEFORE INSERT ON outbox FOR EACH ROW BEGIN IF NEW.InsertIntoDB = '0000-00-00 00:00:00' THEN SET NEW.InsertIntoDB = CURRENT_TIMESTAMP(); END IF; IF NEW.SendingDateTime = '0000-00-00 00:00:00' THEN SET NEW.SendingDateTime = CURRENT_TIMESTAMP(); END IF; IF NEW.SendingTimeOut = '0000-00-00 00:00:00' THEN SET NEW.SendingTimeOut = CURRENT_TIMESTAMP(); END IF; END;// CREATE TRIGGER phones_timestamp BEFORE INSERT ON phones FOR EACH ROW BEGIN IF NEW.InsertIntoDB = '0000-00-00 00:00:00' THEN SET NEW.InsertIntoDB = CURRENT_TIMESTAMP(); END IF; IF NEW.TimeOut = '0000-00-00 00:00:00' THEN SET NEW.TimeOut = CURRENT_TIMESTAMP(); END IF; END;// CREATE TRIGGER sentitems_timestamp BEFORE INSERT ON sentitems FOR EACH ROW BEGIN IF NEW.InsertIntoDB = '0000-00-00 00:00:00' THEN SET NEW.InsertIntoDB = CURRENT_TIMESTAMP(); END IF; IF NEW.SendingDateTime = '0000-00-00 00:00:00' THEN SET NEW.SendingDateTime = CURRENT_TIMESTAMP(); END IF; END;// DELIMITER ; Note You can find the script in docs/sql/mysql.sql as well. UPGRADING TABLES
The easiest way to upgrade database structure is to backup old one and start with creating new one based on example above. For upgrading existing database, you can use changes described in smsd-tables-history and then manually update Version field in gammu ta- ble. AUTHOR
Michal iha <michal@cihar.com> COPYRIGHT
2009-2012, Michal iha <michal@cihar.com> 1.31.90 February 24, 2012 GAMMU-SMSD-MYSQL(7)
All times are GMT -4. The time now is 05:09 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy