The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Special Forums > Web Programming, Web 2.0 and Mashups
Reload this Page robots.txt usage
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read



Thread: robots.txt usage
View Single Post in the UNIX and Linux Forums - Click on the Thread or Permalink to View Entire Thread -->
  #2 (permalink)  
Old 06-25-2009
figaro figaro is offline
Registered User
  
 

Join Date: Jan 2007
Posts: 267
Almost everything there is to know about robots.txt can be found here: The Web Robots Pages
Hope this helps