Extracting the column containing URL from a text file

07-16-2014

Registered User

17, 0

Join Date: Apr 2012

Last Activity: 1 August 2014, 5:20 PM EDT

Posts: 17

Thanks Given: 6

Thanked 0 Times in 0 Posts

Extracting the column containing URL from a text file

I have the file like this:

Timestamp URL Text 1331635241000 http://example.com Peoples footage at www.test.com,http://example4.com 1331635231000 http://example1.net crack the nuts http://example6.com 1331635280000 http://example2.net Loving thisEach column is tab separated. I need to extract only the URLs from column 2 and column 3 if in case of the no URLs then leave it empty for example to get the result like this:

URL Text http://example.com www.test.com,http://example4.com http://example1.net http://example6.com http://example2.net

I tried this script

Code:

awk 'BEGIN {FS="\t"} {print $2,$3}' file | grep -oP '(((http|https|ftp|gopher)|mailto)[.:][^ >"\t]*|www\.[-a-z0-9.]+)[^ .,;\t>">\):]'

This script can give me the all URLS in a single column without the header. Any suggestion to resolve this.

csim_mohan

View Public Profile for csim_mohan

Find all posts by csim_mohan

HTTPFS2(1) HTTPFS2(1) NAME
httpfs2 - mount a file from a http server into the filesystem SYNOPSIS
httpfs2 [OPTIONS] URL FUSE-OPTIONS httpfs2_ssl [OPTIONS] URL FUSE-OPTIONS DESCRIPTION
httpfs2 is a FUSE based filesystem for mounting http or https URLS as files in the filesystem. There is no notion of listable directories in http so only a single URL can be mounted. The server must be able to send byte ranges. OPTIONS
-c console Attempt to use the file ior device console for output after fork. The default is /dev/console. -f Do not fork, stay in foreground. -t timeout Use different timeout for connections. Default '30's. URL The url should specify the protocol as http or https, and it may specify basic authentication username and password. Currently special characters like whitespace are not handled so the URL cannot contain them. See a sample URL below: http://user:password@server.com/dir/file FUSE-OPTIONS These options are passed to the FUSE library. At the very least the mount point should be specified. EXIT STATUS
0 Successfully connected to the server other Failure (url parsing error, server error, FUSE setup error). Some FUSE errors may happen only after the process forks so they will not be returned in exit value. BUGS
The process can be stopped by typing ^Z on the terminal which may not be desirable under some circumstances. AUTHORS
Miklos Szeredi <miklos@szeredi.hu> hmb marionraven at users.sourceforge.net Michal Suchanek <hramrach@centrum.cz> COPYING
Free use of this software is granted under the terms of the GNU General Public License (GPL). 03/13/2010 HTTPFS2(1)

Shell Programming and Scripting