Visit The New, Modern Unix Linux Community


UNIX.com response times


 
Thread Tools Search this Thread
Contact Us Post Here to Contact Site Administrators and Moderators UNIX.com response times
# 8  
It's usually blazing fast for me, honestly. I've never seen anything but minute or two intervals of slow performance, then back to fast the majority of the time.
# 9  
I'm in Asia and it's blazing fast for me now.

However, yesterday I noticed a period of very slow response and I traced the problem to BingBot hammering the site.

Most of the problems I have noticed of lately have been related to bots indexing the site in an abusive way.

Normally I block them, but I hesitate to blog the BingBot so I just set the crawl-delay to 1

Code:
User-agent: msnbot 
Crawl-delay: 1

User-agent: bingbot
Crawl-delay: 1

Let's see if this helps slow down the bingbots!

PS: This is a constant problem that we have to deal with..... most bots do not follow the robots.txt directive, especially those from China, Russia, etc.
This User Gave Thanks to Neo For This Post:
# 10  
I also noticed +30 second response times since a few days.
But now it's good again, 1-2 seconds Smilie
# 11  
Yeah, it's a full time job just managing malicious and poorly configured bots.

I'll try to look at some other adjustments to see if I can mitigate this in other ways. Some DB parameter tweaking might also help mitigate this.
This User Gave Thanks to Neo For This Post:
# 12  
Quote:
Originally Posted by Neo
so I just set the crawl-delay to 1

Code:
User-agent: msnbot 
Crawl-delay: 1

User-agent: bingbot
Crawl-delay: 1

This seems to have done the trick, unix.com is back to normal again now.

bakunin
# 13  
Thanks!

It's constant admin work trying to beat back all the misconfigured and malicious indexing bots. Seems they pop up and try to suck all the posts and threads from the site with no benefit to us or the site, only draining resources and causing the site to slow down.

Most bots, unfortunately, do not follow the robots.txt directives and must be blocked.
These 3 Users Gave Thanks to Neo For This Post:
# 14  
I couldn't help but notice that over recent days response has been really fast (accessing from the UK).

Last edited by hicksd8; 10-19-2017 at 05:09 AM..

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #31
Difficulty: Easy
Alan Turing was the founder of IBM.
True or False?

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Choosing VPN server based on server response times

Hello all, I am using the VPN provider Private Internet Access. I am using the Raspberry Pi 4 with 4GB of RAM, performance on this upgraded board is great. Anyways I am connecting to its service using systemd's openvpn-client @ US_New_York_City.service I wonder if I can create a... (5 Replies)
Discussion started by: haloslayer255
5 Replies

2. What is on Your Mind?

Changing Times at UNIX.COM

Over the past year, I have written so much code at UNIX.COM, I've gained 4 KGs just sitting at my desk and not exercising! However, it seems that "no good deed goes unpunished" and not only have I sacrificed my health (gaining weight, not exercising as much), but there is also my family who is... (4 Replies)
Discussion started by: Neo
4 Replies

3. Red Hat

Response Times

Hello all. Let me qualify my question by saying that I am struggling with how to ask the question I am semi green but have no issue reading up if pointed in the right direction. Please be gentle! A RHEL server 6.2. Hosts a statistical application that has some web apps and batch programming... (0 Replies)
Discussion started by: rsheikh01
0 Replies

4. Shell Programming and Scripting

feasibility of opening a website link from unix and get a response in the form of xml or html

i just wanted to know whether is it possible to open a website link and get a response in the form of xml or html format... the website is of local network... for example something like this wget http://blahblah.samplesite.com/blachblahcblach/User/jsp/ShowPerson.jsp?empid=123456 ... (2 Replies)
Discussion started by: vivek d r
2 Replies

5. Programming

Problem with implementing the times() function in C (struct tms times return zero/negative values)

Hello, i'm trying to implement the times() function and i'm programming in C. I'm using the "struct tms" structure which consists of the fields: The tms_utime structure member is the CPU time charged for the execution of user instructions of the calling process. The tms_stime structure... (1 Reply)
Discussion started by: g_p
1 Replies

Featured Tech Videos