Today (Saturday) We will make some minor tuning adjustments to MySQL.

You may experience 2 up to 10 seconds "glitch time" when we restart MySQL. We expect to make these adjustments around 1AM Eastern Daylight Saving Time (EDT) US.


UNIX.com response times


Login or Register to Reply

 
Thread Tools Search this Thread
# 8  
It's usually blazing fast for me, honestly. I've never seen anything but minute or two intervals of slow performance, then back to fast the majority of the time.
# 9  
I'm in Asia and it's blazing fast for me now.

However, yesterday I noticed a period of very slow response and I traced the problem to BingBot hammering the site.

Most of the problems I have noticed of lately have been related to bots indexing the site in an abusive way.

Normally I block them, but I hesitate to blog the BingBot so I just set the crawl-delay to 1

Code:
User-agent: msnbot 
Crawl-delay: 1

User-agent: bingbot
Crawl-delay: 1

Let's see if this helps slow down the bingbots!

PS: This is a constant problem that we have to deal with..... most bots do not follow the robots.txt directive, especially those from China, Russia, etc.
This User Gave Thanks to Neo For This Post:
# 11  
Yeah, it's a full time job just managing malicious and poorly configured bots.

I'll try to look at some other adjustments to see if I can mitigate this in other ways. Some DB parameter tweaking might also help mitigate this.
This User Gave Thanks to Neo For This Post:
# 12  
Quote:
Originally Posted by Neo
so I just set the crawl-delay to 1

Code:
User-agent: msnbot 
Crawl-delay: 1

User-agent: bingbot
Crawl-delay: 1

This seems to have done the trick, unix.com is back to normal again now.

bakunin
# 13  
Thanks!

It's constant admin work trying to beat back all the misconfigured and malicious indexing bots. Seems they pop up and try to suck all the posts and threads from the site with no benefit to us or the site, only draining resources and causing the site to slow down.

Most bots, unfortunately, do not follow the robots.txt directives and must be blocked.
These 3 Users Gave Thanks to Neo For This Post:
# 14  
I couldn't help but notice that over recent days response has been really fast (accessing from the UK).

Last edited by hicksd8; 10-19-2017 at 05:09 AM..
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Changing Times at UNIX.COM
Neo
Over the past year, I have written so much code at UNIX.COM, I've gained 4 KGs just sitting at my desk and not exercising! However, it seems that "no good deed goes unpunished" and not only have I sacrificed my health (gaining weight, not exercising as much), but there is also my family who is...... What is on Your Mind?
4
What is on Your Mind?
Response Times
rsheikh01
Hello all. Let me qualify my question by saying that I am struggling with how to ask the question I am semi green but have no issue reading up if pointed in the right direction. Please be gentle! A RHEL server 6.2. Hosts a statistical application that has some web apps and batch programming...... Red Hat
0
Red Hat
feasibility of opening a website link from unix and get a response in the form of xml or html
vivek d r
i just wanted to know whether is it possible to open a website link and get a response in the form of xml or html format... the website is of local network... for example something like this wget http://blahblah.samplesite.com/blachblahcblach/User/jsp/ShowPerson.jsp?empid=123456 ...... Shell Programming and Scripting
2
Shell Programming and Scripting

Featured Tech Videos