02-27-2017
Baiduspider and Forum Performance Issues
For years we blocked Baiduspider due to the fact their bots do not obey the robots.txt directive and can really hurt site performance when they unleash 100 bots on the site each pulling pages many times per second.
Last year, I unblocked Baiduspider's IP addresses, and now the problem is back.
I'm not a fan of blocking bots (or people) but I have noticed the site occasionally slows down dramatically because of Baiduspider.
So, unfortunately, I may need to block Baiduspider again.
My apologies to all the good people in China.
These 5 Users Gave Thanks to Neo For This Post:
9 More Discussions You Might Find Interesting
1. Post Here to Contact Site Administrators and Moderators
Didn't we used to a a forum for unix/microsoft interoperability issues? Was it dropped intentionally? Or did it get nailed by the same bug that got the shell forum a little while ago? (0 Replies)
Discussion started by: Perderabo
0 Replies
2. Shell Programming and Scripting
I need help in awk please help immediatly.
This below function is taking lot of time
Please help me to fine tune it so that it runs faster.
The file count is around 3million records
# Process Body
processbody() {
#set -x
while read line
do
... (18 Replies)
Discussion started by: icefish
18 Replies
3. Solaris
using the internal 2 drives mirror was created using raidctl on 100's of our servers . sometime when one drive fails we dont face any issue & we replace the drive with out any problem . but sometimes when one drive fails , system becomes unresponsive and doesnot allow us to login , the only way to... (1 Reply)
Discussion started by: skamal4u
1 Replies
4. UNIX for Dummies Questions & Answers
Hi All,
I'm facing an issue in my awk script.
The script is processing a large text file having the details of a number of persons, each person's details being written from 100 to 250 tags as given below:
100 START|
101klklk|
...
245 opr|
246 55|
250 END|
100 START|
...
245 pp|
246... (4 Replies)
Discussion started by: pgp_acc1
4 Replies
5. Programming
Hi,
I have written a program in C and have to test the return value of the functions. So the normal way of doin this wud b
int rc
rc=myfunction(input);
if(rc=TRUE){
}
else{
}
But instead of doing this I have called the function in the if() condition. Does this have any... (2 Replies)
Discussion started by: sidmania
2 Replies
6. AIX
Hi,
We have GPFS 3.4 Installed on two AIX 6.1 Nodes. We have 3 GPFS Mount points:
/abc01 4TB (Comprises of 14 x 300GB disks from XIV SAN)
/abc02 4TB (Comprises of 14 x 300GB disks from XIV SAN)
/abc03 1TB ((Comprises of Multiple 300GB disks from XIV SAN)
Now these 40... (1 Reply)
Discussion started by: aixromeo
1 Replies
7. Solaris
I 'm trying to clone a zfs file system pool/u01 to a new file system called newpool/u01 using following commands
zfs list
zfs snapshot pool/u01@new
zfs send pool/u01@new | zfs -F receive newpool/u01
Its a 100G file system snapshot and copied to same server on different pool and... (9 Replies)
Discussion started by: fugitive
9 Replies
8. Solaris
Hello everyone,
recently we have been experiencing performance issues with chmod. We managed to narrow it down to getcwd.
The following folder exists:
/Folder1/subfol1/subfol2/subfol3
cd /Folder1/subfol1/subfol2/subfol3
truss -D pwd 2>&1 | grep getcwd
0.0001... (4 Replies)
Discussion started by: KotekBury
4 Replies
9. AIX
Good Day Everyone,
Just wonder anyone has encounter AIX 6.1 Memory Performance issues ? What I have in my current scenario is we have 3 datastage servers (Segregate server and EE jobs - for those who know Datastage achitect) and 2 db servers(running HA to load balance 4 nodes partitions for... (3 Replies)
Discussion started by: ckwan
3 Replies
LEARN ABOUT DEBIAN
jsonbot
JSONBOT(1) jsb manual JSONBOT(1)
NAME
JSONBOT - The JSON everywhere bot!
SYNOPSIS
JSONBOT is a remote event-driven framework for building bots that talk JSON to each other over XMPP. This manual page documents briefly the
jsb distribution..
DESCRIPTION
This distribution provides bots built on this framework for console, IRC, XMPP, Convore and WWW on the shell.
USAGE
JSONBOT uses command line options to configure a bot. See the --help option to get more help on a command options or see the corresponding
man page.
the jsb pakage contains the following programs:
* jsb - console version of jsb
* jsb-backup - create backup of JSONBOT datadir
* jsb-convore - convore version of jsb
* jsb-init - create data directory and config examples, default ~/.jsb
* jsb-irc - IRC version of jsb
* jsb-fleet - mix IRC and XMPP bots
* jsb-sed - sed a whole directory
* jsb-stop - stop a running bot
* jsb-tornado - a shell web server based on tornado
* jsb-udp - send udp packets to the bot that will relay the data
* jsb-xmpp - XMPP version of jsb
note: JSONBOT is in BETA stage right now and still subject to change of protocols and API.
see http://jsonbot.googlecode.com. see https://jsonbot.org for documentation on the bot.
SEE ALSO
The program is documented fully on http://jsonbot.org or see http://jsonbot.googlecode.com
SEE ALSO
jsb(1), jsb-backup(1), jsb-init(1), jsb-irc(1), jsb-xmpp(1), jsb-fleet(1), jsb-tornado(1), jsb-stop(1), jsb-udp(1)
AUTHOR
This manual page was written by Bart Thate <bthate@gmail.com>, for the Debian GNU/Linux system (but may be used by others).
Debian GNU/Linux 22 Nov 2011 JSONBOT(1)