The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Special Forums > Web Programming, Web 2.0 and Mashups
robots.txt usage robots.txt usage
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read


Web Programming, Web 2.0 and Mashups Discuss Web Programming and Web Server Administration, including LAMP, Apache, MySQL, Flash, HTML, SEO, Mashups and other Web APIs and topics.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
current CPU usage, memory usage, disk I/O oid(snmp) S_venkatesh SUN Solaris 2 12-13-2008 06:19 AM
how can I find cpu usage memory usage swap usage and logical volume usage alert0919 HP-UX 3 12-02-2008 02:38 PM
K-Robots 0.5 (Default branch) iBot Software Releases - RSS News 0 07-16-2008 09:10 PM
Monitor CPU usage and Memory Usage Gajanad Bihani High Level Programming 2 03-09-2005 07:35 AM

 
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Bulgarian Greek Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
Prev Previous Post   Next Post Next
  #1 (permalink)  
Old 06-25-2009
rickhlwong rickhlwong is offline
Registered User
  
 

Join Date: Jul 2008
Posts: 8
robots.txt usage
Dear all,

I want to use robots.txt to control the "spider". can i specify a IP address to ALLOW the website can be accessed by the "spider"??
thank you.

Rick
 

Bookmarks

« Finding which firefox thread is misbehaving | HTML help required »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Threaded Mode Threaded Mode
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 11:41 AM.

The UNIX and Linux Forums - Learn UNIX and UNIX Commands RSS Feeds - Contact Us - The UNIX and Linux Forums - Learn UNIX and UNIX Commands - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0