Visit Our UNIX and Linux User Community

Search Results

Search: Posts Made By: Neo
9,288
Posted By Neo
Update: There have been no spike after two...
Update:

There have been no spike after two fulls, busy weekdays (Monday and Tuesday).

Mission Accomplished!

8172
9,288
Posted By Neo
Thanks for clarifying. Honestly, I have...
Thanks for clarifying.

Honestly, I have great working tools and instrumentation already which I am very (extremely) happy with; and so I am not looking for other tools (in the project).

If you...
9,288
Posted By Neo
Update: I have confirmed 100% the source of...
Update:

I have confirmed 100% the source of the these spikes were very aggressive, rogue, unidentified bots originating on Chinese networks. After blocking the resident networks of these bots,...
9,288
Posted By Neo
Update: Experienced (and trapped) another...
Update:

Experienced (and trapped) another spike from another Chinese IP address (which is at the top of the "hitcount" list during the spikes):

8168

116.232.48.112

with the same user...
9,288
Posted By Neo
So, let's try this: iptables -A INPUT -s...
So, let's try this:

iptables -A INPUT -s 117.144.138.130/24 -j DROP # rogue chinese bot
iptables -A INPUT -s 116.232.49.231/24 -j DROP # rogue chinese bot

Empty the "trap" again and block...
9,288
Posted By Neo
Just got another spike exactly three hours after...
Just got another spike exactly three hours after the last one, not correlated to any cron / batch process:

8162

8163

Chinese IPs:


117.144.138.130
116.232.49.231
116.232.48.112

...
9,288
Posted By Neo
Update: 8161 There were two spikes...
Update:

8161

There were two spikes three hours apart; both were captured by my HTTP session logging program, which logs session detaisl aggregated by IP address. In this case, the code starts...
9,288
Posted By Neo
Update: Just noticed, after digging around...
Update:

Just noticed, after digging around in the DB logs from my MQTT instrumentation, that the last spike correlated with a jump in data transferred out of the network interface:

8156
...
9,288
Posted By Neo
Next: I have some old "cyberspace...
Next:

I have some old "cyberspace situational awareness" PHP code I used for a visualization project a few years ago, which captures and stores details information on web session activity; this...
9,288
Posted By Neo
Update: Did not help at all. Slowed...
Update:



Did not help at all. Slowed the site down a bit and did not stop any spikes.

ModPagespeed on
9,288
Posted By Neo
Prometheus uses HTTP on the same server where...
Prometheus uses HTTP on the same server where HTTP is the main application under observation. This is from the Prometheus docs:


a multi-dimensional data model with time series data identified...
9,288
Posted By Neo
Not yet. Last night did not confirm the...
Not yet.

Last night did not confirm the "rogue bots are the cause" .... hypothesis (see above post). Two more spikes, no correlation to increase bot number or network I/O. But I'm still...
9,288
Posted By Neo
Update: Two more spikes over the course of...
Update:

Two more spikes over the course of the past 12 hours, none of which show any correlation to an increase the number of bots; however, that does not say anything about the velocity of bots...
9,288
Posted By Neo
Exactly Victor, It's not a big deal because...
Exactly Victor,

It's not a big deal because the spikes are just for a minute 4 to 6 times a day; but the problem is when (potentially) all the "bad things" align all at once (bots, DB, system...
9,288
Posted By Neo
Looks like it was "bot related" TImeline...
Looks like it was "bot related"

TImeline from my MQTT instrumentation logged in the DB:


1581684184 Bot activity starts to peak

1581684491 . Apache process and CPU% begin to spike

...
9,288
Posted By Neo
Update: After adding more instrumentation,...
Update:

After adding more instrumentation, including Apache2 processes, Apache2 CPU and a questionable MySQL CPU graph, the first spike of the last half day occurred and there is correlation...
9,288
Posted By Neo
Thanks Scrutinizer, I'm feeling confident...
Thanks Scrutinizer,

I'm feeling confident that the root cause is related to MySQL (maybe 80% confidence level, off the top of my head).

The reasons are as follows:


Instrumentation...
9,288
Posted By Neo
Thanks for the suggestion. I was sitting at...
Thanks for the suggestion.

I was sitting at my desk with another spike occurred and there were no unusual or phantom processes popping up.

MySQL remained at the top of the CPU utilization with,...
9,288
Posted By Neo
Or, I may use atop (they are all very similar...
Or, I may use atop (they are all very similar linux command line tools for this kind of thing.... )

Thanks for the suggestions and ideas.

It's great to have some outside input; as it is hard to...
9,288
Posted By Neo
Agreed... I think I will try iostat or iotop...
Agreed...

I think I will try iostat or iotop during anticipated spike periods (if I can predict one, LOL)

Or I will write some code to instrument this when the spike happens and try to trap the...
9,288
Posted By Neo
Hi Dennis, No. I have mentioned this...
Hi Dennis,



No. I have mentioned this a number of times already, including the first post :) . There are no network I/O spikes.

Regarding disk I/O, I have not yet set up any...
9,288
Posted By Neo
Just spiked again.... nearly exactly Thursday,...
Just spiked again.... nearly exactly Thursday, February 13, 2020 10:02 AM UTC to Thursday, February 13, 2020 10:03 AM UTC, (5PM my time) just a one minute spike hit. Instrumentation shows no cron...
9,288
Posted By Neo
Yes, most of the DB tables (99 percent) are...
Yes, most of the DB tables (99 percent) are MYISAM tables, especially the larger ones.

I don't have SAN.... The SCSI disks are directly attached in the box.
9,288
Posted By Neo
FYI, there has been no spike in the past 8-9...
FYI, there has been no spike in the past 8-9 hours (my time):

8148
9,288
Posted By Neo
Based on prior experience, a SYN Flood attack msg...
Based on prior experience, a SYN Flood attack msg in dmesg is a fraction of the traffic the site has (it's noise), so I don't think that is an issue (it's noise, I think... not "signal" .... ).
...
Showing results 1 to 25 of 31

 

Featured Tech Videos

All times are GMT -4. The time now is 01:12 PM.
Unix & Linux Forums Content Copyright 1993-2020. All Rights Reserved.
Privacy Policy