UNIX for Advanced & Expert Users

So, let's try this:

Code:

iptables -A INPUT -s 117.144.138.130/24 -j DROP  #  rogue chinese bot
iptables -A INPUT -s 116.232.49.231/24 -j DROP  #  rogue chinese bot

Empty the "trap" again and block two Chinese subnetworks with rouge, unidentified bot activity.

Honestly, this is starting to "annoy me a lot" in the possibility that these performance hits, and all the time I am spending to find the cause of these hits / spikes, wasting valuable "time in life" is related to rouge, unidentified bots from Chinese networks.

If this continues, I am going to start blocking Chinese networks at the /16 and /8 levels (entire networks).

First, let's see if this is indeed the main source of these spikes. As we all know from situational awareness theory and the famous OODA loop by John Boyd.

Already, we have enough information to ACT. But lets continue to OBSERVE

The loop goes on ... and on ....

Please note that we cannot trust apache2 modules and other third-party software to automatically block IPs, because this can results in blocking the "good bots" which are important for search engine optimization and site traffic.

That means, if this is confirmed that these kinds of bots continue to be the cause of problems, then I will need to DECIDE how to deal with this situation moving forward. I think point in time, I am going to continue to "trap and trace" before making a decision. However, it does seem, at this point, that rouge, unidentified bots from Chinese networks are causing performance issues and need to be "dealt with".

If anyone else has experienced similar issues and has an interesting potential solution to this problem, please reply and share your ideas.

Thanks!

PS: I may consider automating this, as follows:

Update:

Experienced (and trapped) another spike from another Chinese IP address (which is at the top of the "hitcount" list during the spikes):

Code:

116.232.48.112

with the same user agent string as before:

Code:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36

Yesterday, as reader who follow this caper my call recall, I blocked two Chinese subnetworks /24

Code:

iptables -A INPUT -s 117.144.138.130/24 -j DROP  #  rogue chinese bot
iptables -A INPUT -s 116.232.49.231/24 -j DROP  #  rogue chinese bot

Now, we see rouge, unidentified bot activity from 116.232.48.112, more than likely in the same data center.

So, I will change the block to:

Code:

iptables -A INPUT -s 117.144.138.130/24 -j DROP  #  rogue chinese bot
iptables -A INPUT -s 116.232.49.231/22 -j DROP  #  rogue chinese bot

... let's what what they do next...... I am interested to learn if "they" are manually shifting servers or this is an automatic response to the block.

Update:

I have confirmed 100% the source of the these spikes were very aggressive, rogue, unidentified bots originating on Chinese networks. After blocking the resident networks of these bots, all spikes have stopped, completely.

This is a "huge success story", going from unknown, uncorrelated performance hits / spikes due to nearly random spikes in performance to cause identification and total resolution. As you can see from the graph over the last 24 hours, there have been zero spikes.



I will keep the same MQTT and Node-RED instrumentation in place (which I am very pleased with) and will also keep all "spike trapping" instrumentation and DB logging in place, so if other spikes appear, which I am fairly confident more of these "pesky" bots will appear sooner or later, I will trap them, identify the source and block their resident networks.

Success!

MQTT and Node-RED did not "solve the problem". MQTT and Node-RED provided a very powerful and flexible way for me to quickly instrument custom sensors and logging, which helped me identify the problem.

I highly, recommend MQTT and Node-RED. These tools are free. Thank you very much MQTT and Node-RED developers!

Update:

There have been no spike after two fulls, busy weekdays (Monday and Tuesday).

Mission Accomplished!