Login or Register to Ask a Question and Join Our Community


Web Development

robots.txt usage

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Web Development robots.txt usage
# 1  
Old 06-25-2009
robots.txt usage
Dear all,

I want to use robots.txt to control the "spider". can i specify a IP address to ALLOW the website can be accessed by the "spider"??
thank you.

Rick
# 2  
Old 06-25-2009
Almost everything there is to know about robots.txt can be found here: The Web Robots Pages
Hope this helps
# 3  
Old 06-25-2009
Short answer - no.
# 4  
Old 06-26-2009
Quote:
Originally Posted by rickhlwong
Dear all,

I want to use robots.txt to control the "spider". can i specify a IP address to ALLOW the website can be accessed by the "spider"??
thank you.

Rick
We block unwanted spiders by IP address using ipchains. The robots.txt file uses the User Agent field in HTTP.

Many aggressive spiders do not follow robots.txt and will have to be blocked using something like ipchains or "insert your favorite" firewall tool
# 5  
Old 06-28-2009
thank you~~

---------- Post updated at 09:13 AM ---------- Previous update was at 09:07 AM ----------

thank you~~
Login or Register to Ask a Question

Previous Thread | Next Thread

5 More Discussions You Might Find Interesting

1. IP Networking

TXT Records: Usage

Ok..the last DNS question. I've been on a DNS kick lately. So when poking around, I keep bumping sites that have txt records all with cryptic, but yet similar text in them.. Something like: cnn.com. 3305 IN TXT "882269757-4422010" cnn.com. 3305 IN TXT "ms=ms97284866"... (1 Reply)
Discussion started by: Lost in Cyberia
1 Replies

2. Solaris

Netbackup robots not working

Hi All, I am facing a issue with robtest not working on netbackup 7.1 on solaris 10. I can see the robots and drives are deteted by O.S but not sure why robtest is not working. Below are few ouputs of few commands. $PWD>cfgadm -al -o show_FCP_dev Ap_Id Type ... (0 Replies)
Discussion started by: sahil_shine
0 Replies

3. AIX

How to monitor the IBM AIX server for I/O usage,memory usage,CPU usage,network..?

How to monitor the IBM AIX server for I/O usage, memory usage, CPU usage, network usage, storage usage? (3 Replies)
Discussion started by: laknar
3 Replies

4. HP-UX

how can I find cpu usage memory usage swap usage and logical volume usage

how can I find cpu usage memory usage swap usage and I want to know CPU usage above X% and contiue Y times and memory usage above X % and contiue Y times my final destination is monitor process logical volume usage above X % and number of Logical voluage above can I not to... (3 Replies)
Discussion started by: alert0919
3 Replies

5. UNIX for Dummies Questions & Answers

echo "ABC" > file1.txt file2.txt file3.txt

Hi Guru's, I need to create 3 files with the contents "ABC" using single command. Iam using: echo "ABC" > file1.txt file2.txt file3.txt the above command is not working. pls help me... With Regards / Ganapati (4 Replies)
Discussion started by: ganapati
4 Replies
Login or Register to Ask a Question