The UNIX and Linux Forums  

Go Back   The UNIX and Linux Forums > Special Forums > Web Programming, Web 2.0 and Mashups
Reload this Page robots.txt usage
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read



Thread: robots.txt usage
View Single Post in the UNIX and Linux Forums - Click on the Thread or Permalink to View Entire Thread -->
  #1 (permalink)  
Old 06-25-2009
rickhlwong rickhlwong is offline
Registered User
  
 

Join Date: Jul 2008
Posts: 8
robots.txt usage
Dear all,

I want to use robots.txt to control the "spider". can i specify a IP address to ALLOW the website can be accessed by the "spider"??
thank you.

Rick