Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Help with url filtering
Top Forums Shell Programming and Scripting Help with url filtering Post 302567180 by r4v3n on Saturday 22nd of October 2011 10:27:30 PM
Old 10-22-2011
Help with url filtering
Hi,

Here's my input file:

Code:
http://google.com/1.exe
http://google.com/2.exe
http://google.com/3.exe
http://rediff.jp/sample.zip
http://yahoo.com/1.exe

And here's what my output file should ideally look like:

Code:
http://google.com
http://rediff.jp/sample.zip
http://yahoo.com/1.exe

So what I'm trying to do is to take just 1 entry of those URLs which repeat themselves , and keep the once that don't repeat as it is, for URL blacklisting.

Can this be done using sed/grep/awk/uniq etc ?

Thanks in advance
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

2. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

3. UNIX for Dummies Questions & Answers

ReDirecting a URL to another URL - Linux

Hello, I need to redirect an existing URL, how can i do that? There's a current web address to a GUI that I have to redirect to another webaddress. Does anyone know how to do this? This is on Unix boxes Linux. example: https://m45.testing.address.net/host.php make it so the... (3 Replies)
Discussion started by: SkySmart
3 Replies

4. Web Development

Regex to rewrite URL to another URL based on HTTP_HOST?

I am trying to find a way to test some code, but I need to rewrite a specific URL only from a specific HTTP_HOST The call goes out to http://SUB.DOMAIN.COM/showAssignment/7bde10b45efdd7a97629ef2fe01f7303/jsmodule/Nevow.Athena The ID in the middle is always random due to the cookie. I... (5 Replies)
Discussion started by: EXT3FSCK
5 Replies

5. Shell Programming and Scripting

Please help me to do some filtering

I have to grep a pattern. scenario is like :- Suppose "/etc/sec/one" is a string, i need to check if this string contains "one" using any utility something like if /etc/sec/one | grep ; then Thanks in advance Renjesh Raju (3 Replies)
Discussion started by: Renjesh
3 Replies

6. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

7. AIX

Need help with filtering

Hi!! I have a bit of a task here and filtering/scripting not my strongest. I have to collect info of approx 1100 hdiskpower.so i have appended all the hdisk into a text file and i need it to run the command lscfg -vl to confirm if the drive is symmetrix. here's what i have so far at... (3 Replies)
Discussion started by: vpundit
3 Replies

8. Shell Programming and Scripting

Filtering

Hi I am interested in DNS resolving a set of sites and each time the output is different- $ host www.yahoo.com www.yahoo.com is an alias for fd-fp3.wg1.b.yahoo.com. fd-fp3.wg1.b.yahoo.com is an alias for ds-fp3.wg1.b.yahoo.com. ds-fp3.wg1.b.yahoo.com is an alias for... (1 Reply)
Discussion started by: jamie_123
1 Replies

9. Shell Programming and Scripting

Need help on filtering

Hi experts, I have a file image.csv as below: COMPUTERNAME,23/07/2013,22/07/2013,21/07/2013,20/07/2013,19/07/2013,18/07/2013,17/07/2013 AED03852180,3,3,3,3,3,3,3 AED03852181,3,3,3,3,3,3,1 AED09020382,3,0,3,0,3,3,3 AED09020383,1,3,3,3,2,1,3 AED09020386,3,3,0,3,3,0,3 ... (4 Replies)
Discussion started by: zaq1xsw2
4 Replies

10. Shell Programming and Scripting

Reading URL using Mechanize and dump all the contents of the URL to a file

Hello, Am very new to perl , please help me here !! I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file. below is the script which i have written so far , #!/usr/bin/perl use LWP::UserAgent; use... (2 Replies)
Discussion started by: scott_cog
2 Replies

LEARN ABOUT DEBIAN

gwakeonlan

GWAKEONLAN(1)						      General Commands Manual						     GWAKEONLAN(1)

NAME

       gWakeOnLan - Wake up your machines using Wake on LAN.

SYNOPSIS

       gwakeonlan [options]

DESCRIPTION

       gWakeOnLan is a A GTK+ utility to awake turned off computers through the Wake on LAN feature.

       The machines to turn on need to be shut off with the Wake on LAN magic packet enabled. Linux environments can enable this using:

       ethtool -s <nic device> wol g

OPTIONS

       This  program follow the usual GNU command line syntax, with long options starting with two dashes (`-').  A summary of options is included
       below.

       -h, --help
	      Show summary of options

       -v, --verbose
	      Show all error and information messages

       -q, --quiet
	      Hide all error and information messages

FILES

       Hosts list will be kept under ~/.gwakeonlan

REPORTING BUGS

       Report bugs to http://code.google.com/p/gwakeonlan/issues/list

AUTHORS

       gWakeOnLan was written by Fabio Castelli <muflone@vbsimple.net>

HOMEPAGE

       International project: http://code.google.com/p/gwakeonlan/

       Italian project: http://ubuntrucchi.wordpress.com/progetti-projects/gwakeonlan/

COPYRIGHT

       Copyright (C) 2010 Fabio Castelli.  License GPLv2+: GNU GPL version 2 or later <http://www.gnu.org/licenses/gpl-2.0.html>.

       This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.

								 January 16, 2010						     GWAKEONLAN(1)
All times are GMT -4. The time now is 02:52 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy