Sponsored Content
Top Forums UNIX for Advanced & Expert Users Parsing a file which contains urls from different sites Post 302357552 by radoulov on Wednesday 30th of September 2009 06:49:23 AM
Old 09-30-2009
With GNU AWK you can do something like this:

Code:
gawk -F'http://(www\\.)?|/' '!_[$2]++{print $2}' infile

Otherwise use Perl:

Code:
perl -nle'
  print $1 unless $_{(m|http://(?:www.)?([^/]*)|)[0]}++
' infile



---------- Post updated at 12:49 PM ---------- Previous update was at 12:48 PM ----------

To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums
 

8 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

find and replace urls

I need to archive a large website onto a DVD. Many of the links and image srcs are absolute URLs. As I don't want to alter them all manually, I'm looking for a perl or unix command that would remove: http://www.mydomain.com/mysubfolder/ and replace with: ./ Can anyone help me with this... (3 Replies)
Discussion started by: benkyma
3 Replies

2. Shell Programming and Scripting

Parsing of file for Report Generation (String parsing and splitting)

Hey guys, I have this file generated by me... i want to create some HTML output from it. The problem is that i am really confused about how do I go about reading the file. The file is in the following format: TID1 Name1 ATime=xx AResult=yyy AExpected=yyy BTime=xx BResult=yyy... (8 Replies)
Discussion started by: umar.shaikh
8 Replies

3. Shell Programming and Scripting

Remove external urls from .html file

Hi everyone. I have an html file with lines like so: link href="localFolder/..."> link href="htp://..."> img src="localFolder/..."> img src="htp://..."> I want to remove the links with http in the href and imgs with http in its src. I'm having trouble removing them because there... (4 Replies)
Discussion started by: CowCow339
4 Replies

4. Shell Programming and Scripting

Hashing URLs

So, I am writing a script that will read output from Bulk Extractor (which gathers data based on regular expressions). My script then reads the column that has the URL found, hashes it with MD5, then outputs the URL and hash to a file. Where I am stuck on is that I want to read the bulk... (7 Replies)
Discussion started by: twjolson
7 Replies

5. Web Development

How to redirect URLs in Apache?

I am a total newbie to Apache. I need to do this only for this weekend during an upgrade from old system to new system We have different URLs http://domain.name/xxx (xxx varies to any length and words - it can be /home, /login, /home/daily, /daily/report, etc). How do i redirect all those to... (0 Replies)
Discussion started by: GosarJunk
0 Replies

6. Post Here to Contact Site Administrators and Moderators

Not allowed to post URLs

Hi, I tried to post some perl code for discussion (wrapped in swaddling . However, a regex has an escaped backslash so the forum parser sees it as an URL? Had the same experience with the sample data that I tried to provide for the same discussion. It contains emails addresses,... (1 Reply)
Discussion started by: msutfin
1 Replies

7. Shell Programming and Scripting

Replacing urls from file

Hi ALL, I have a file A which contains A=www.google.com B=www.abcd.com C=www.nick.com D=567 file B Contains A=www.google1234.com B=www.bacd.com C=www.mick.com D=789 I wanted a script which can replace file A contents with B Contents (5 Replies)
Discussion started by: nikhil jain
5 Replies

8. Shell Programming and Scripting

Regex for URLs and files

Hi, I am looking for a regex that will validate a URL and files accessed in a browser. For example:http://www.google.co.uk http://www.google.com https://www.google.co.uk https://www.google.com ftp:// file:///somefile/on/a/server/accessed/from/browser/file.txt So far I have: ... (4 Replies)
Discussion started by: muay_tb
4 Replies
URLS.TXT(5)							File Formats Manual						       URLS.TXT(5)

NAME
urls.txt - URL database for regression testing INTRODUCTION
The urls.txt file is installed by default in /etc/siege/urls.txt. When siege is invoked without a command line reference to a URL, then by default it looks for urls in that file. The advantage of using the urls.txt file is two-fold: One, it frees you free retyping a url with every invocation. And two, it allows you to conduct full site regression testing. When the urls.txt file is used, siege reads all the urls in that file into memory and runs through the list in one of two ways, sequen- tially or randomly. The default run is sequential from start to finish and back again until the --reps or the --time option has been satis- fied. With the -i/--internet option selected, siege runs through the file randomly simulating the stress applied by a community of inter- net users. The -f/--file option allows you to select a file other then the default urls.txt file. You may also instruct siege to use a different file with the "file" directive in .siegerc, i.e., "file = /usr/local/etc/urls.txt" You may set and reference variables inside the urls.txt file. All variables must be declared BEFORE they are referenced. Variables are declared with the "=" operator, VARIABLE = VALUE. They are then referenced inside $() or ${}, example: $(HOST), ${HOST} HOST=joey.joedog.org http://${HOST}/browse.jsp?size=5 http://${HOST}/admin.jsp?name=ralph EXAMPLE FILE
This is an exmple urls.txt file. Lines beginning with a hash (#) are comments and ignored by siege. # # Example urls.txt file # URLs database for siege # http://www.haha.com/index.html http://www.haha.com/howto/index.html http://www.haha.com/cgi-bin/howto/display.cgi?1013 www.haha.com/cgi-bin/fm.cgi?first=j.&last=fulmer https://www.haha.com/index.shtml https://www.whoohoo.com/my_whoohoo.jsp # POST data requires a POST directive www.haha.com/cgi-bin/foo.cgi POST first=bart&last=simpson www.haha.com/hoho.jsp POST name=jeff&pass=secret # POST the contents of a file using the # line input character "<" http://www.haha.com/my.jsp POST </home/jeff/data.txt AUTHOR
Jeffrey Fulmer <jeff@joedog.org>, et al. BUGS
Report bugs to jeff@joedog.org. Give a detailed description of the problem and report the version of siege that you are using. COPYRIGHT
Copyright (C) 2007 Jeffrey Fulmer, et al. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MER- CHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. AVAILABILITY
The most recent released version of siege is available by anonymous FTP from ftp.joedog.org in the directory pub/siege. SEE ALSO
siege(1) siege.config(1) layingsiege(7) Siege v2.70 April-17-2012 URLS.TXT(5)
All times are GMT -4. The time now is 05:46 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy