Sponsored Content
Top Forums Web Development Discrepancies Between Our Mobile Detection Code and Google Analytics Regarding Chinese User Agents Post 303040224 by Neo on Friday 25th of October 2019 06:09:01 AM
Old 10-25-2019
Here is a link to a similar discussion about the same Chinese bots, identified as "mobile devices" by our system which have been hammering the site of late:

Blocking aggressive Chinese crawlers/scrapers/bots - John Large - Technology, Hardware,Web Development, Digital Privacy & Ethics

So far, I am just sending many of these UAs to 403, but may come up with some better idea after more logging and analysis.

I am not keen on "mass blocking" but these bots are really extreme and the IP addresses are changing so much that IP address blocking will not work; so I have to block based on UA.
 

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

SED Search and Replace Question for Google Analytics

Well, I'm losing my regex ability in sed! Please help. I need to search for this text in multiple html files in a directory: </body> and add the following lines in front of the text above: <script src="http://www.google-analytics.com/urchin.js" type="text/javascript"> </script> ... (11 Replies)
Discussion started by: Neo
11 Replies

2. Google Chrome OS

Google Chrome Mobile?

I know that Google Chrome came out with the Android under a Unix based system, but did it come out yet for Windows Mobile? I have Windows Mobile 6 on my device. (0 Replies)
Discussion started by: Anna Hussie
0 Replies

3. Shell Programming and Scripting

User Agents Identifier

Hi, Can anyone help me write a script tp determine the web browser from a user agent. A user agent is stored in a file and consists of hundreds of lines e.g. 37050 Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko 29404 Mozilla/5.0 (Windows NT 6.3; WOW64)... (4 Replies)
Discussion started by: cyberfrog
4 Replies

4. Web Development

Quick Fix for Google Search Console "Page is not mobile friendly"

Over the past 10 plus years, we have countless posts where the user did not use CODE tags or they used ICODE tags incorrectly. This has has the results of this site penalized by Google for having pages which are "not mobile friendly". So, working quietly in the background, in the thankless... (0 Replies)
Discussion started by: Neo
0 Replies

5. What is on Your Mind?

Google Search Console - Mobile Usability - No Errors or Issues - New Milestone

For the first time in the history of the site Google Search Console (GSC) has unix.com showing "no mobile viewability errors". This is no small achievement considering the hundreds of thousand of lines of legacy code we run at a site which has been around much longer than Facebook or LinkedIn: ... (0 Replies)
Discussion started by: Neo
0 Replies
io_nonblock(3)						     Library Functions Manual						    io_nonblock(3)

NAME
io_nonblock - switch to non-blocking I/O SYNTAX
#include <io.h> void io_nonblock(int64 fd); DESCRIPTION
io_nonblock puts UNIX descriptor fd into ``non-blocking mode.'' Calling io_nonblock(fd) before io_fd(fd) makes io_tryread and io_trywrite faster and more efficient. Actually, current UNIX kernels do not support non-blocking descriptors; they support non-blocking open files. Furthermore, many programs will break if they encounter non-blocking mode. This means that you must not use io_nonblock for a descriptor inherited from another pro- gram. io_nonblock has no return value; it always succeeds. If d is not the number of a UNIX descriptor, io_nonblock has no effect. If io_fd is given a descriptor in blocking mode, io_tryread and io_trywrite go through the following contortions to avoid blocking: 1 Stop if poll says that the descriptor is not ready. Otherwise there's a good chance, but not a guarantee: even if poll says the descrip- tor is ready, the descriptor might not be ready a moment later. (Furthermore, poll can fail on some systems.) 2 Catch SIGALRM. SIGALRM must not be blocked, and must not be used elsewhere in the program. 3 Set an interval timer so that any blocking call will be interrupted by SIGALRM within 10 milliseconds. (Current UNIX kernels do not allow any shorter interval.) Of course, this may still mean a 10-millisecond delay. If io_fd is given a descriptor in non-blocking mode (or a descriptor for a regular disk file), io_tryread and io_trywrite avoid these con- tortions. SEE ALSO
io_wait(3), io_canwrite(3) io_nonblock(3)
All times are GMT -4. The time now is 03:45 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy