Unix/Linux Go Back    

Post Here to Contact Site Administrators and Moderators Forum members, please contact us here (not via email or PM).
Bug reports and minor feature requests should be posted here.

UNIX.com response times

Post Here to Contact Site Administrators and Moderators


Tags
bingbot, robots.txt, solved

Reply    
 
Thread Tools Search this Thread Display Modes
    #8  
Old Unix and Linux 4 Weeks Ago
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 16 October 2017, 6:42 PM EDT
Location: Saskatchewan
Posts: 22,462
Thanks: 1,139
Thanked 4,259 Times in 3,933 Posts
It's usually blazing fast for me, honestly. I've never seen anything but minute or two intervals of slow performance, then back to fast the majority of the time.
    #9  
Old Unix and Linux 4 Weeks Ago
Neo's Unix or Linux Image
Neo Neo is online now Forum Staff  
Administrator
 
Join Date: Sep 2000
Last Activity: 17 October 2017, 10:50 AM EDT
Location: Asia pacific region
Posts: 13,716
Thanks: 888
Thanked 1,203 Times in 564 Posts
I'm in Asia and it's blazing fast for me now.

However, yesterday I noticed a period of very slow response and I traced the problem to BingBot hammering the site.

Most of the problems I have noticed of lately have been related to bots indexing the site in an abusive way.

Normally I block them, but I hesitate to blog the BingBot so I just set the crawl-delay to 1


Code:
User-agent: msnbot 
Crawl-delay: 1

User-agent: bingbot
Crawl-delay: 1

Let's see if this helps slow down the bingbots!

PS: This is a constant problem that we have to deal with..... most bots do not follow the robots.txt directive, especially those from China, Russia, etc.
The Following User Says Thank You to Neo For This Useful Post:
MadeInGermany (4 Weeks Ago)
    #10  
Old Unix and Linux 4 Weeks Ago
MadeInGermany MadeInGermany is offline Forum Staff  
Moderator
 
Join Date: May 2012
Last Activity: 17 October 2017, 3:44 AM EDT
Location: Simplicity
Posts: 3,777
Thanks: 311
Thanked 1,265 Times in 1,144 Posts
I also noticed +30 second response times since a few days.
But now it's good again, 1-2 seconds Linux
    #11  
Old Unix and Linux 4 Weeks Ago
Neo's Unix or Linux Image
Neo Neo is online now Forum Staff  
Administrator
 
Join Date: Sep 2000
Last Activity: 17 October 2017, 10:50 AM EDT
Location: Asia pacific region
Posts: 13,716
Thanks: 888
Thanked 1,203 Times in 564 Posts
Yeah, it's a full time job just managing malicious and poorly configured bots.

I'll try to look at some other adjustments to see if I can mitigate this in other ways. Some DB parameter tweaking might also help mitigate this.
The Following User Says Thank You to Neo For This Useful Post:
bakunin (4 Weeks Ago)
    #12  
Old Unix and Linux 4 Weeks Ago
bakunin bakunin is offline Forum Staff  
Bughunter Extraordinaire
 
Join Date: May 2005
Last Activity: 17 October 2017, 9:43 AM EDT
Location: In the leftmost byte of /dev/kmem
Posts: 5,606
Thanks: 106
Thanked 1,592 Times in 1,173 Posts
Quote:
Originally Posted by Neo View Post
so I just set the crawl-delay to 1


Code:
User-agent: msnbot 
Crawl-delay: 1

User-agent: bingbot
Crawl-delay: 1

This seems to have done the trick, unix.com is back to normal again now.

bakunin
    #13  
Old Unix and Linux 4 Weeks Ago
Neo's Unix or Linux Image
Neo Neo is online now Forum Staff  
Administrator
 
Join Date: Sep 2000
Last Activity: 17 October 2017, 10:50 AM EDT
Location: Asia pacific region
Posts: 13,716
Thanks: 888
Thanked 1,203 Times in 564 Posts
Thanks!

It's constant admin work trying to beat back all the misconfigured and malicious indexing bots. Seems they pop up and try to suck all the posts and threads from the site with no benefit to us or the site, only draining resources and causing the site to slow down.

Most bots, unfortunately, do not follow the robots.txt directives and must be blocked.
The Following 3 Users Say Thank You to Neo For This Useful Post:
RudiC (4 Weeks Ago), vbe (4 Weeks Ago), vgersh99 (4 Weeks Ago)
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Response Times rsheikh01 Red Hat 0 02-26-2015 08:28 PM
feasibility of opening a website link from unix and get a response in the form of xml or html vivek d r Shell Programming and Scripting 2 06-14-2012 02:34 AM
Problem with implementing the times() function in C (struct tms times return zero/negative values) g_p Programming 1 10-31-2011 04:04 PM



All times are GMT -4. The time now is 10:59 AM.