Sponsored Content
Top Forums UNIX for Advanced & Expert Users Nearly Random, Uncorrelated Server Load Average Spikes Post 303044156 by Neo on Saturday 15th of February 2020 10:14:42 PM
Old 02-15-2020
So, let's try this:

Code:
iptables -A INPUT -s 117.144.138.130/24 -j DROP  #  rogue chinese bot
iptables -A INPUT -s 116.232.49.231/24 -j DROP  #  rogue chinese bot

Empty the "trap" again and block two Chinese subnetworks with rouge, unidentified bot activity.

Honestly, this is starting to "annoy me a lot" in the possibility that these performance hits, and all the time I am spending to find the cause of these hits / spikes, wasting valuable "time in life" is related to rouge, unidentified bots from Chinese networks.

If this continues, I am going to start blocking Chinese networks at the /16 and /8 levels (entire networks).

First, let's see if this is indeed the main source of these spikes. As we all know from situational awareness theory and the famous OODA loop by John Boyd.
  1. OBSERVE
  2. ORIENT
  3. DECIDE
  4. ACT

Already, we have enough information to ACT. But lets continue to OBSERVE Smilie

The loop goes on ... and on ....

Please note that we cannot trust apache2 modules and other third-party software to automatically block IPs, because this can results in blocking the "good bots" which are important for search engine optimization and site traffic.

That means, if this is confirmed that these kinds of bots continue to be the cause of problems, then I will need to DECIDE how to deal with this situation moving forward. I think point in time, I am going to continue to "trap and trace" before making a decision. However, it does seem, at this point, that rouge, unidentified bots from Chinese networks are causing performance issues and need to be "dealt with".

If anyone else has experienced similar issues and has an interesting potential solution to this problem, please reply and share your ideas.

Thanks!

PS: I may consider automating this, as follows:
  1. Capture network session activity when one minute load average exceeds a threshold (as I am doing now).
  2. Filter results captured in the DB based on "hitcount" and "country".
  3. If the "hitcount" exceeds a certain threshold and "country" is in an array of "known to have rouge bots countries".
  4. THEN BLOCK the ip_address/24
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

load average

we have an unix system which has load average normally about 20. but while i am running a particular unix batch which performs heavy operations on filesystem and database average load reduces to 15. how can we explain this situation? while running that batch idle cpu time is about %60-65... (0 Replies)
Discussion started by: gfhgfnhhn
0 Replies

2. UNIX for Dummies Questions & Answers

Load Average

Hello all, I have a question about load averages. I've read the man pages for the uptime and w command for two or three different flavors of Unix (Red Hat, Tru64, Solaris). All of them agree that in the output of the 2 aforementioned commands, you are given the load average for the box, but... (3 Replies)
Discussion started by: Heathe_Kyle
3 Replies

3. UNIX for Dummies Questions & Answers

top - Load average

Hello, Here is the output of top command. My understanding here is, the load average 0.03 in last 1 min, 0.02 is in last 5 min, 0.00 is in last 15 min. By seeing this load average, When can we say that, the system load averge is too high? When can we say that, load average is medium/low??... (8 Replies)
Discussion started by: govindts
8 Replies

4. Solaris

load average query.

Hi, i have installed solaris 10 on t-5120 sparc enterprise. I am little surprised to see load average of 2 or around on this OS. when checked with ps command following process is using highest CPU. looks like it is running for long time and does not want to stop, but I do not know... (5 Replies)
Discussion started by: upengan78
5 Replies

5. UNIX for Dummies Questions & Answers

Please Help me in my load average

Hello AlL,.. I want from experts to help me as my load average is increased and i dont know where is the problem !! this is my top result : root@a4s # top top - 11:30:38 up 40 min, 1 user, load average: 3.06, 2.49, 4.66 Mem: 8168788k total, 2889596k used, 5279192k free, 47792k... (3 Replies)
Discussion started by: black-code
3 Replies

6. UNIX for Advanced & Expert Users

Load average in UNIX

Hi , I am using 48 CPU sunOS server at my work. The application has facility to check the current load average before starting a new process to control the load. Right now it is configured as 48. So it does mean that each CPU can take maximum one proces and no processe is waiting. ... (2 Replies)
Discussion started by: kumaran_5555
2 Replies

7. Solaris

Load Average and Lwps

NPROC USERNAME SWAP RSS MEMORY TIME CPU 320 oracle 23G 22G 69% 582:55:11 85% 47 root 148M 101M 0.3% 99:29:40 0.3% 53 rafmsdb 38M 60M 0.2% 0:46:17 0.1% 1 smmsp 1296K 5440K 0.0% 0:00:08 0.0% 7 daemon ... (2 Replies)
Discussion started by: snjksh
2 Replies

8. UNIX for Dummies Questions & Answers

Load average spikes once an hour

Hi, I am getting a high load average, around 7, once an hour. It last for about 4 minutes and makes things fairly unusable for this time. How do I find out what is using this. Looking at top the only thing running at the time is md5sum. I have looked at the crontab and there is nothing... (10 Replies)
Discussion started by: sm9ai
10 Replies

9. UNIX for Dummies Questions & Answers

Help with load average?

how load average is calculated and what exactly is it difference between cpu% and load average (9 Replies)
Discussion started by: robo
9 Replies

10. Programming

ESP32 (ESP-WROOM-32) as an MQTT Client Subscribed to Linux Server Load Average Messages

Here we go.... Preface: ..... so in a galaxy far, far, far away from commercial, data sharing corporations..... For this project, I used the ESP-WROOM-32 as an MQTT (publish / subscribe) client which receives Linux server "load averages" as messages published as MQTT pub/sub messages.... (6 Replies)
Discussion started by: Neo
6 Replies
JABBER-QUERYBOT(1p)					User Contributed Perl Documentation				       JABBER-QUERYBOT(1p)

NAME
jabber-querybot - a modular perl jabber bot DESCRIPTION
jabber-querybot connects a jabber account and wait for messages. If a message comes in, it forward it to your self programmend modul. The return string of your module, jabber-querybot send it back to the jabber sender. It is designed to be re-usable and to make it easy to write small Jabber bots that do one thing and do it well. A simple concept with a lot of examples and experiences are implemented. 1. Create a jabber account on a jabber-server around 2. Create a bot application: cd examples cp Querymodule.pm /etc/jabber-querybot/Mybot.pm cd /etc/jabber-querybot ln -s Mybot.pm Querymodule.pm Modify login parameters to your jabber-bot-account vim Mybot.pm our $hostname = "swissjabber.ch"; our $user = ""; our $password = ""; our $ident = "Testbot"; our $bot_admin = "@swissjabber.ch"; our $port = "5222"; our $timeout = "5"; our $service_name = "$user@$hostname"; our $bot_description = "Bot help title Bot description"; For each jabber message, jabber-querybot will execute sub run_query, that you can write here your application. You can control how your jabber response will be: o error = error message stanza o presence = error as presence stanza o ignore = ignore message OPTIONS
jabber-querybot has a lot of variables which you can easy modify for what you need: querystatus $querystatus = [ 0 | 1 ] o 0 = Bot will not proceed any incoming jabber messages. o 1 = Bot will proceed incoming messages. penalty_status If the bot has too much workload, it goes to penalty status and wait some time until his status change back to normal. $timer_reconnect_default = 21600 Every 21600 seconds (6 hours) the bot will shutdown automatically, wait 10 seconds and starting up again. $timer_auto_query = 0 If you set in your module this variable to 60, the bot will every 60 seconds call the function run_auto_query() which you may use for several things. System load If your systems load is >=6, this bot will shutdown the jabber connection and check every 10 seconds systems load. If load <=2, bot will start over. EXAMPLES
/usr/share/doc/jabber-querybot/examples/Testbot.pm FILES
/etc/jabber-querybot/Querymodule.pm /usr/bin/jabber-querybot RESOURCES
http://github.com/micressor/jabber-querybot/ METHODS
InMessage() An incoming jabber message to the bot will hook this function. o Read parameters of incoming stanza o Decode utf8 string o increment timer overload and do not process message if bot is overloaded. o Ignore message if it is from myself o Be sure, that it is not a message from another transport o Check any systemcommands for the bot o If the bot has sleeping status, change it to work o We process only normal text or chat type jabber messages o And now we give the real text string which was incoming to the bot via run_query(). o If run_query() say us 'ignore` we do a log entry and do not answer via jabber to the user. o If run_query() says 'error` we send a jabber error stanza wiht the status message from run_query() back to the user. o If there was no error, we update the statistic vars and send the answer from run_query() back to the jabber user. connect_server() This function connects to the jabber server with the given credentials from Querymodule.pm. Set the call back functions. This functions will be executed if a message of the types <message/> <iq/> or <presence/> are incoming. Stop() Shutdown jabber connection and exit main program disconnect_server() Only disconnect from the jabber-server. reconnect_server() Reconnect and create a log entry. InIQ() We do not proceed any iq (information query), this is only for statisic. InPresence() o Increment timer overload o Do not process message if we are overloaded penalty_handler(). We have a problem in Net::Jabber. An incoming message with a ` in resource blocks the bot. We will hotfix that for the moment. A subscription type `subscribe` is incoming. Send `subscribed` tho the user and say hello ;) o Send presence to user o Remove subscription if a user remove this bot from his roster sendPresence() Send presence information to user systemcommands() o If user type '!help` send a help instruction to the user o Send statistic information to the user if he types '!status` o Is it a bot command? o Is the bot command from the bot admin? If not, send "not allowed" o '!shutdown` will shutdown your bot via jabber invoke. o '!query off` turn off queries o '!query on` turn on queries get_barejid() Remove resource id from a jabber id. calcualte_stats() Calculate message statistics jabber_set_presence() Set new presence if we have another presence status or we have set the force flag (for transport presence). jabber_send_message() This function send all jabber messages which are outgoing from the jabber-querybot. set_wakeup_mode() Set jabber presence via jabber_set_presence() penalty_handler() This function checks if the bot is overloaded with incoming mesages and reject if it is. Two times that's ok so 2:1 because while in main. Bot admin got everytime an answer. jabber_add_footer() Add footer to the processed message. calcualte_elapsed_time() Calculate elapsed worktime for a query sendError() sendError($message, $from, $to, 404, "text"); system_load_check() o Calculate system load o If load is too high shutdown bot. o If load is ok, starting up bot connect_bot() Connect bot and initialize all timers. check_before_start() Check configuration variables in Querybotmodule.pm and give answer if anything does not match or is missing. COPYRIGHT AND LICENSE
Copyright (C) 2009-2012 Marco Balmer <marco@balmer.name> The Debian packaging is licensed under the GPL, see `/usr/share/common-licenses/GPL-3'. perl v5.14.2 2012-02-03 JABBER-QUERYBOT(1p)
All times are GMT -4. The time now is 06:37 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy