Sponsored Content
Top Forums UNIX for Advanced & Expert Users Nearly Random, Uncorrelated Server Load Average Spikes Post 303044138 by Neo on Saturday 15th of February 2020 02:28:08 AM
Old 02-15-2020
Next:

I have some old "cyberspace situational awareness" PHP code I used for a visualization project a few years ago, which captures and stores details information on web session activity; this code has proven handy identifying rouge bots in the past.

So, I have modified that code to capture and store detailed session information, including the number of hits per IP address, the user agent string, country code, etc. when the 1 minute load average is above 20 and less than 50.

Code:
$theload = getLoadAvg();
if (floatval($theload) > 20.0 && floatval($theload) < 50.0) 
{
  /// the old CSA code to parse web session activity and store the results in the DB
}

So, let's see what happens the next time we get a spike... this should be interesting.


Code:
mysql> describe neo_csa_session_manager;
+--------------+------------------+------+-----+---------+----------------+
| Field        | Type             | Null | Key | Default | Extra          |
+--------------+------------------+------+-----+---------+----------------+
| id           | int(11) unsigned | NO   | PRI | NULL    | auto_increment |
| user_id      | int(11)          | NO   | MUL | 0       |                |
| session_id   | varchar(255)     | NO   |     | NULL    |                |
| url          | text             | NO   |     | NULL    |                |
| ip_address   | varchar(45)      | NO   | MUL | NULL    |                |
| user_agent   | varchar(255)     | NO   |     | NULL    |                |
| bot_flag     | tinyint(1)       | NO   |     | 0       |                |
| robot_txt    | mediumint(6)     | NO   |     | 0       |                |
| sitemap      | mediumint(6)     | NO   |     | 0       |                |
| riskscore    | int(11)          | NO   |     | 0       |                |
| country_iso2 | varchar(2)       | NO   |     | UN      |                |
| country      | varchar(50)      | NO   |     | UNKNOWN |                |
| hitcount     | int(10) unsigned | NO   |     | 1       |                |
| firstseen    | bigint(11)       | NO   |     | NULL    |                |
| unixtime     | bigint(11)       | YES  |     | NULL    |                |
| longitude    | float            | NO   |     | 0       |                |
| latitude     | float            | NO   |     | 0       |                |
+--------------+------------------+------+-----+---------+----------------+
17 rows in set (0.00 sec)

This User Gave Thanks to Neo For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

load average

we have an unix system which has load average normally about 20. but while i am running a particular unix batch which performs heavy operations on filesystem and database average load reduces to 15. how can we explain this situation? while running that batch idle cpu time is about %60-65... (0 Replies)
Discussion started by: gfhgfnhhn
0 Replies

2. UNIX for Dummies Questions & Answers

Load Average

Hello all, I have a question about load averages. I've read the man pages for the uptime and w command for two or three different flavors of Unix (Red Hat, Tru64, Solaris). All of them agree that in the output of the 2 aforementioned commands, you are given the load average for the box, but... (3 Replies)
Discussion started by: Heathe_Kyle
3 Replies

3. UNIX for Dummies Questions & Answers

top - Load average

Hello, Here is the output of top command. My understanding here is, the load average 0.03 in last 1 min, 0.02 is in last 5 min, 0.00 is in last 15 min. By seeing this load average, When can we say that, the system load averge is too high? When can we say that, load average is medium/low??... (8 Replies)
Discussion started by: govindts
8 Replies

4. Solaris

load average query.

Hi, i have installed solaris 10 on t-5120 sparc enterprise. I am little surprised to see load average of 2 or around on this OS. when checked with ps command following process is using highest CPU. looks like it is running for long time and does not want to stop, but I do not know... (5 Replies)
Discussion started by: upengan78
5 Replies

5. UNIX for Dummies Questions & Answers

Please Help me in my load average

Hello AlL,.. I want from experts to help me as my load average is increased and i dont know where is the problem !! this is my top result : root@a4s # top top - 11:30:38 up 40 min, 1 user, load average: 3.06, 2.49, 4.66 Mem: 8168788k total, 2889596k used, 5279192k free, 47792k... (3 Replies)
Discussion started by: black-code
3 Replies

6. UNIX for Advanced & Expert Users

Load average in UNIX

Hi , I am using 48 CPU sunOS server at my work. The application has facility to check the current load average before starting a new process to control the load. Right now it is configured as 48. So it does mean that each CPU can take maximum one proces and no processe is waiting. ... (2 Replies)
Discussion started by: kumaran_5555
2 Replies

7. Solaris

Load Average and Lwps

NPROC USERNAME SWAP RSS MEMORY TIME CPU 320 oracle 23G 22G 69% 582:55:11 85% 47 root 148M 101M 0.3% 99:29:40 0.3% 53 rafmsdb 38M 60M 0.2% 0:46:17 0.1% 1 smmsp 1296K 5440K 0.0% 0:00:08 0.0% 7 daemon ... (2 Replies)
Discussion started by: snjksh
2 Replies

8. UNIX for Dummies Questions & Answers

Load average spikes once an hour

Hi, I am getting a high load average, around 7, once an hour. It last for about 4 minutes and makes things fairly unusable for this time. How do I find out what is using this. Looking at top the only thing running at the time is md5sum. I have looked at the crontab and there is nothing... (10 Replies)
Discussion started by: sm9ai
10 Replies

9. UNIX for Dummies Questions & Answers

Help with load average?

how load average is calculated and what exactly is it difference between cpu% and load average (9 Replies)
Discussion started by: robo
9 Replies

10. Programming

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages

Here we go.... Preface: ..... so in a galaxy far, far, far away from commercial, data sharing corporations..... For this project, I used the ESP-WROOM-32 as an MQTT (publish / subscribe) client which receives Linux server "load averages" as messages published as MQTT pub/sub messages.... (6 Replies)
Discussion started by: Neo
6 Replies
BeanCounter(3pm)					User Contributed Perl Documentation					  BeanCounter(3pm)

NAME
Finance::BeanCounter - Module for stock portfolio performance functions. SYNOPSIS
use Finance::BeanCounter; DESCRIPTION
Finance::BeanCounter provides functions to download, store and analyse stock market data. Downloads are available of current (or rather: 15 or 20 minute-delayed) price and company data as well as of historical price data. Both forms can be stored in an SQL database (for which we currently default to PostgreSQL though MySQL is supported as well; furthermore any database reachable by means of an ODBC connection should work). Analysis currently consists of performance and risk analysis. Performance reports comprise a profit-and-loss (or 'p/l' in the lingo) report which can be run over arbitrary time intervals such as "--prevdate 'friday six months ago' --date 'yesterday'" -- in essence, whatever the wonderful Date::Manip module understands -- as well as dayendreport which defaults to changes in the last trading day. A risk report show parametric and non-parametric value-at-risk (VaR) estimates. Most available functionality is also provided in the reference implementation beancounter, a convenient command-line script. The API might change and evolve over time. The low version number really means to say that the code is not in its final form yet, but it has been in use for well over four years. More documentation is in the Perl source code. DATABASE LAYOUT
The easiest way to see the table design is to look at the content of the setup_beancounter script. It creates the five tables stockinfo, stockprices, fxprices, portfolio and indices. Note also that is supports the creation of database for both PostgreSQL and MySQL. THE STOCKINFO TABLE The stockinfo table contains general (non-price) information and is index by symbol: symbol varchar(12) not null, name varchar(64) not null, exchange varchar(16) not null, capitalisation float4, low_52weeks float4, high_52weeks float4, earnings float4, dividend float4, p_e_ratio float4, avg_volume int4 This table is updated by overwriting the previous content. THE STOCKPRICES TABLE The stockprices table contains (daily) price and volume information. It is indexed by both date and symbol: symbol varchar(12) not null, date date, previous_close float4, day_open float4, day_low float4, day_high float4, day_close float4, day_change float4, bid float4, ask float4, volume int4 During updates, information is appended to this table. THE FXPRICES TABLE The fxprices table contains (daily) foreign exchange rates. It can be used to calculate home market values of foreign stocks: currency varchar(12) not null, date date, previous_close float4, day_open float4, day_low float4, day_high float4, day_close float4, day_change float4 Similar to the stockprices table, it is index on date and symbol. THE STOCKPORTFOLIO TABLE The portfolio table contains contains the holdings information: symbol varchar(16) not null, shares float4, currency varchar(12), type varchar(16), owner varchar(16), cost float(4), date date It is indexed on symbol,owner,date. THE INDICES TABLE The indices table links a stock symbol with one or several market indices: symbol varchar(12) not null, stockindex varchar(12) not null BUGS
Finance::BeanCounter and beancounter are so fresh that there are only missing features :) On a more serious note, this code (or its earlier predecessors) have been in use since the fall of 1998. Known bugs or limitations are documented in TODO file in the source package. SEE ALSO
beancounter.1, smtm.1, Finance::YahooQuote.3pm, LWP.3pm, Date::Manip.3pm COPYRIGHT
Finance::BeanCounter.pm (c) 2000 -- 2006 by Dirk Eddelbuettel <edd@debian.org> Updates to this program might appear at http://eddelbuettel.com/dirk/code/beancounter.html. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. There is NO warranty whatsoever. The information that you obtain with this program may be copyrighted by Yahoo! Inc., and is governed by their usage license. See http://www.yahoo.com/docs/info/gen_disclaimer.html for more information. ACKNOWLEDGEMENTS
The Finance::YahooQuote module by Dj Padzensky (on the web at http://www.padz.net/~djpadz/YahooQuote/) served as the backbone for data retrieval, and a guideline for the extension to the non-North American quotes which was already very useful for the real-time ticker http://eddelbuettel.com/dirk/code/smtm.html. perl v5.10.1 2010-06-13 BeanCounter(3pm)
All times are GMT -4. The time now is 07:24 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy