Extracting the column containing URL from a text file

07-16-2014

Registered User

17, 0

Join Date: Apr 2012

Last Activity: 1 August 2014, 5:20 PM EDT

Posts: 17

Thanks Given: 6

Thanked 0 Times in 0 Posts

Extracting the column containing URL from a text file

I have the file like this:

Timestamp URL Text 1331635241000 http://example.com Peoples footage at www.test.com,http://example4.com 1331635231000 http://example1.net crack the nuts http://example6.com 1331635280000 http://example2.net Loving thisEach column is tab separated. I need to extract only the URLs from column 2 and column 3 if in case of the no URLs then leave it empty for example to get the result like this:

URL Text http://example.com www.test.com,http://example4.com http://example1.net http://example6.com http://example2.net

I tried this script

Code:

awk 'BEGIN {FS="\t"} {print $2,$3}' file | grep -oP '(((http|https|ftp|gopher)|mailto)[.:][^ >"\t]*|www\.[-a-z0-9.]+)[^ .,;\t>">\):]'

This script can give me the all URLS in a single column without the header. Any suggestion to resolve this.

csim_mohan

View Public Profile for csim_mohan

Find all posts by csim_mohan

IO::All::HTTP(3pm) User Contributed Perl Documentation IO::All::HTTP(3pm) NAME
IO::All::LWP - Extends IO::All to HTTP URLs SYNOPSIS
use IO::All; $content < io('http://example.org'); # GET webpage into scalar io('http://example.org') > io('index.html'); # GET to file "hello " > io('http://example.org/index.html'); # PUT webpage # two ways of getting a page with a password: $content < io('http://me:secret@example.org'); $content < io('http://example.org')->user('me')->password('secret'); DESCRIPTION
This module extends IO::All for dealing with HTTP URLs. Note that you don't need to use it explicitly, as it is autoloaded by IO::All whenever it sees something that looks like an HTTP URL. The SYNOPSIS shows some simple typical examples, but there are many other interesting combinations with other IO::All features! For example, you can get an HTTP URL and write the content to a socket, or to an FTP URL, of to a DBM file. METHODS
This is a subclass of IO::All::LWP. The only new method is "http", which can be used to create a blank IO::All::HTTP object; or it can also take an HTTP URL as a parameter. Note that in most cases it is simpler just to call io('http://example.com'), which calls the "http" method automatically. OPERATOR OVERLOADING
The same operators from IO::All may be used. < GETs an HTTP URL; > PUTs to an HTTP URL. SEE ALSO
IO::All, IO::All::LWP, LWP. AUTHORS
Ivan Tubert-Brohman <itub@cpan.org> and Brian Ingerson <ingy@cpan.org> COPYRIGHT
Copyright (c) 2007. Ivan Tubert-Brohman and Brian Ingerson. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See <http://www.perl.com/perl/misc/Artistic.html> perl v5.10.0 2007-03-29 IO::All::HTTP(3pm)

Shell Programming and Scripting