Hello, I have been searching for a way to extact urls from google cache url search results,

I have a file with a list of urls like this

""http://64.233.167.104/search?q=cache:ts2G04wctD0J:www.worldwidewords.org/qa/qa-shi3.htm+%22shit%22&hl=en&ct=clnk&cd=12&gl=ca&ie=UTF-8""

what i need to do is extract the actual url www.worldwidewords.org/qa/qa-shi3.htm which lies between the : and the + and remove the google cache url from the list so I will have a list of regular urls, I also have normal urls in the list which I would like to keep in the list.

any help would be appreciated

This is the bash script I am using, it searches google and give you a list of urls, takes out everything but the link and pipes them to a file

#!/bin/bash
#
# google.sh
# ---------
#  Automatic Google search from the command line.
#
#    Syntax : $ google {search terms}
#
if [ -z $1 ]
then
  # If no keyword is entered echo try again
  #
  echo "you didnt tell me what to search....try again"
else
  #url variable with the maximum search results (100) per page
  #
  url='http://google.ca/search?num=100&hl=en&safe=off&q='

  appended=0
  for searchTerm in "$@"
  do
    # Replace white spaces in the search terms
    #
    searchTerm=`echo $searchTerm | sed 's/ /%20/g'`

    url="$url%22$searchTerm%22"

    if [ $appended -lt `expr $# - 1` ]
    then
      url="$url"\+
    else
      url="$url"\&btnG\=Google\+Search\&meta\=
    fi

    let "appended+=1"
  done

  lynx -dump $url >> googleresult1
  sed 's/http/\^http/g' googleresult1 | tr -s "^" "\n" | grep http| sed 's/\ .*//g' >> googleresults2 #this command extract only the urs
  rm googleresult1
  cat googleresults2
  sed -e '/google/d' googleresults2 >> urls.txt
fi

Try this:

line='http://64.233.167.104/search?q=cache:ts2G04wctD0J:www.worldwidewords.org/qa/qa-shi3.htm+%22shit%22&hl=en&ct=clnk&cd=12&gl=ca&ie=UTF-8'

echo $line|sed 's/\(.*\):www\(.*\)+\(.*\)/www\2/'

Three absolute wildcards and no anchors? It may work my personally I would not consider using that code. Also why copy to buffer what you don't use?

echo $url_to_strip | awk -F'[:+]' '{print $4}'

I may wrong, I don't know how the adresses are formatted but this works only with exact three colons before the adresses.

Regards

That is the format of a google cache entry

However in the interest of being more flexible (note that the match is not a generic wildcard) :

echo $url_to_strip | sed -n 's_.*:\(www[^+]*\)+.*_\1_p'