Sponsored Content
The Lounge What is on Your Mind? Baiduspider and Forum Performance Issues Post 302992507 by Neo on Monday 27th of February 2017 01:34:31 AM
Old 02-27-2017
Baiduspider and Forum Performance Issues

For years we blocked Baiduspider due to the fact their bots do not obey the robots.txt directive and can really hurt site performance when they unleash 100 bots on the site each pulling pages many times per second.

Last year, I unblocked Baiduspider's IP addresses, and now the problem is back.

I'm not a fan of blocking bots (or people) but I have noticed the site occasionally slows down dramatically because of Baiduspider.

So, unfortunately, I may need to block Baiduspider again.

My apologies to all the good people in China.
These 5 Users Gave Thanks to Neo For This Post:
 

9 More Discussions You Might Find Interesting

1. Post Here to Contact Site Administrators and Moderators

Dos issues forum.

Didn't we used to a a forum for unix/microsoft interoperability issues? Was it dropped intentionally? Or did it get nailed by the same bug that got the shell forum a little while ago? (0 Replies)
Discussion started by: Perderabo
0 Replies

2. Shell Programming and Scripting

shell script performance issues --Urgent

I need help in awk please help immediatly. This below function is taking lot of time Please help me to fine tune it so that it runs faster. The file count is around 3million records # Process Body processbody() { #set -x while read line do ... (18 Replies)
Discussion started by: icefish
18 Replies

3. Solaris

raidctl performance issues

using the internal 2 drives mirror was created using raidctl on 100's of our servers . sometime when one drive fails we dont face any issue & we replace the drive with out any problem . but sometimes when one drive fails , system becomes unresponsive and doesnot allow us to login , the only way to... (1 Reply)
Discussion started by: skamal4u
1 Replies

4. UNIX for Dummies Questions & Answers

Awk Performance Issues

Hi All, I'm facing an issue in my awk script. The script is processing a large text file having the details of a number of persons, each person's details being written from 100 to 250 tags as given below: 100 START| 101klklk| ... 245 opr| 246 55| 250 END| 100 START| ... 245 pp| 246... (4 Replies)
Discussion started by: pgp_acc1
4 Replies

5. Programming

performance issues of calling a function in if condition

Hi, I have written a program in C and have to test the return value of the functions. So the normal way of doin this wud b int rc rc=myfunction(input); if(rc=TRUE){ } else{ } But instead of doing this I have called the function in the if() condition. Does this have any... (2 Replies)
Discussion started by: sidmania
2 Replies

6. AIX

Performance issues for LPAR with GPFS 3.4

Hi, We have GPFS 3.4 Installed on two AIX 6.1 Nodes. We have 3 GPFS Mount points: /abc01 4TB (Comprises of 14 x 300GB disks from XIV SAN) /abc02 4TB (Comprises of 14 x 300GB disks from XIV SAN) /abc03 1TB ((Comprises of Multiple 300GB disks from XIV SAN) Now these 40... (1 Reply)
Discussion started by: aixromeo
1 Replies

7. Solaris

zfs send receive performance issues

I 'm trying to clone a zfs file system pool/u01 to a new file system called newpool/u01 using following commands zfs list zfs snapshot pool/u01@new zfs send pool/u01@new | zfs -F receive newpool/u01 Its a 100G file system snapshot and copied to same server on different pool and... (9 Replies)
Discussion started by: fugitive
9 Replies

8. Solaris

Getcwd performance issues

Hello everyone, recently we have been experiencing performance issues with chmod. We managed to narrow it down to getcwd. The following folder exists: /Folder1/subfol1/subfol2/subfol3 cd /Folder1/subfol1/subfol2/subfol3 truss -D pwd 2>&1 | grep getcwd 0.0001... (4 Replies)
Discussion started by: KotekBury
4 Replies

9. AIX

AIX 6.1 Memory Performance issues

Good Day Everyone, Just wonder anyone has encounter AIX 6.1 Memory Performance issues ? What I have in my current scenario is we have 3 datastage servers (Segregate server and EE jobs - for those who know Datastage achitect) and 2 db servers(running HA to load balance 4 nodes partitions for... (3 Replies)
Discussion started by: ckwan
3 Replies
JSONBOT(1)							    jsb manual								JSONBOT(1)

NAME
JSONBOT - The JSON everywhere bot! SYNOPSIS
JSONBOT is a remote event-driven framework for building bots that talk JSON to each other over XMPP. This manual page documents briefly the jsb distribution.. DESCRIPTION
This distribution provides bots built on this framework for console, IRC, XMPP, Convore and WWW on the shell. USAGE
JSONBOT uses command line options to configure a bot. See the --help option to get more help on a command options or see the corresponding man page. the jsb pakage contains the following programs: * jsb - console version of jsb * jsb-backup - create backup of JSONBOT datadir * jsb-convore - convore version of jsb * jsb-init - create data directory and config examples, default ~/.jsb * jsb-irc - IRC version of jsb * jsb-fleet - mix IRC and XMPP bots * jsb-sed - sed a whole directory * jsb-stop - stop a running bot * jsb-tornado - a shell web server based on tornado * jsb-udp - send udp packets to the bot that will relay the data * jsb-xmpp - XMPP version of jsb note: JSONBOT is in BETA stage right now and still subject to change of protocols and API. see http://jsonbot.googlecode.com. see https://jsonbot.org for documentation on the bot. SEE ALSO
The program is documented fully on http://jsonbot.org or see http://jsonbot.googlecode.com SEE ALSO
jsb(1), jsb-backup(1), jsb-init(1), jsb-irc(1), jsb-xmpp(1), jsb-fleet(1), jsb-tornado(1), jsb-stop(1), jsb-udp(1) AUTHOR
This manual page was written by Bart Thate <bthate@gmail.com>, for the Debian GNU/Linux system (but may be used by others). Debian GNU/Linux 22 Nov 2011 JSONBOT(1)
All times are GMT -4. The time now is 12:47 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy