Login or Register to Ask a Question and Join Our Community


What is on Your Mind?

Baiduspider and Forum Performance Issues

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
The Lounge What is on Your Mind? Baiduspider and Forum Performance Issues
# 1  
Old 02-27-2017
Baiduspider and Forum Performance Issues
For years we blocked Baiduspider due to the fact their bots do not obey the robots.txt directive and can really hurt site performance when they unleash 100 bots on the site each pulling pages many times per second.

Last year, I unblocked Baiduspider's IP addresses, and now the problem is back.

I'm not a fan of blocking bots (or people) but I have noticed the site occasionally slows down dramatically because of Baiduspider.

So, unfortunately, I may need to block Baiduspider again.

My apologies to all the good people in China.
These 5 Users Gave Thanks to Neo For This Post:
Corona688 Don Cragun hicksd8 rbatte1 RudiC
# 2  
Old 03-09-2017
Sadly, have blocked a number of Baidu spider networks due to repeated problems.
These 3 Users Gave Thanks to Neo For This Post:
Don Cragun jim mcnamara vgersh99
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. AIX

AIX 6.1 Memory Performance issues

Good Day Everyone, Just wonder anyone has encounter AIX 6.1 Memory Performance issues ? What I have in my current scenario is we have 3 datastage servers (Segregate server and EE jobs - for those who know Datastage achitect) and 2 db servers(running HA to load balance 4 nodes partitions for... (3 Replies)
Discussion started by: ckwan
3 Replies

2. Solaris

Getcwd performance issues

Hello everyone, recently we have been experiencing performance issues with chmod. We managed to narrow it down to getcwd. The following folder exists: /Folder1/subfol1/subfol2/subfol3 cd /Folder1/subfol1/subfol2/subfol3 truss -D pwd 2>&1 | grep getcwd 0.0001... (4 Replies)
Discussion started by: KotekBury
4 Replies

3. Solaris

zfs send receive performance issues

I 'm trying to clone a zfs file system pool/u01 to a new file system called newpool/u01 using following commands zfs list zfs snapshot pool/u01@new zfs send pool/u01@new | zfs -F receive newpool/u01 Its a 100G file system snapshot and copied to same server on different pool and... (9 Replies)
Discussion started by: fugitive
9 Replies

4. AIX

Performance issues for LPAR with GPFS 3.4

Hi, We have GPFS 3.4 Installed on two AIX 6.1 Nodes. We have 3 GPFS Mount points: /abc01 4TB (Comprises of 14 x 300GB disks from XIV SAN) /abc02 4TB (Comprises of 14 x 300GB disks from XIV SAN) /abc03 1TB ((Comprises of Multiple 300GB disks from XIV SAN) Now these 40... (1 Reply)
Discussion started by: aixromeo
1 Replies

5. Programming

performance issues of calling a function in if condition

Hi, I have written a program in C and have to test the return value of the functions. So the normal way of doin this wud b int rc rc=myfunction(input); if(rc=TRUE){ } else{ } But instead of doing this I have called the function in the if() condition. Does this have any... (2 Replies)
Discussion started by: sidmania
2 Replies

6. UNIX for Dummies Questions & Answers

Awk Performance Issues

Hi All, I'm facing an issue in my awk script. The script is processing a large text file having the details of a number of persons, each person's details being written from 100 to 250 tags as given below: 100 START| 101klklk| ... 245 opr| 246 55| 250 END| 100 START| ... 245 pp| 246... (4 Replies)
Discussion started by: pgp_acc1
4 Replies

7. Solaris

raidctl performance issues

using the internal 2 drives mirror was created using raidctl on 100's of our servers . sometime when one drive fails we dont face any issue & we replace the drive with out any problem . but sometimes when one drive fails , system becomes unresponsive and doesnot allow us to login , the only way to... (1 Reply)
Discussion started by: skamal4u
1 Replies

8. Shell Programming and Scripting

shell script performance issues --Urgent

I need help in awk please help immediatly. This below function is taking lot of time Please help me to fine tune it so that it runs faster. The file count is around 3million records # Process Body processbody() { #set -x while read line do ... (18 Replies)
Discussion started by: icefish
18 Replies

9. Post Here to Contact Site Administrators and Moderators

Dos issues forum.

Didn't we used to a a forum for unix/microsoft interoperability issues? Was it dropped intentionally? Or did it get nailed by the same bug that got the shell forum a little while ago? (0 Replies)
Discussion started by: Perderabo
0 Replies
Login or Register to Ask a Question