Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Get only domain from url file bind
Top Forums Shell Programming and Scripting Get only domain from url file bind Post 302959519 by Don Cragun on Wednesday 4th of November 2015 02:51:10 AM
Old 11-04-2015
Quote:
Originally Posted by omuhans123
RudiC, thank you very much for providing this solution, it is truly appreciated. I checked through the publicsuffix list and found that the longest domain is 4 as such added this to the script you provided. Now it works and provides all the different domains. Here is the code I am now using:
Code:
awk '
NR==FNR                 {C[$0]
                         next
                        }
$(NF-1) OFS $(NF) in C  {print $(NF-2) OFS $(NF-1) OFS $NF
                         next
                        }
$(NF-1) OFS $(NF) in C  {print $(NF-3) OFS $(NF-1) OFS $NF
                         next
                        }
$(NF-1) OFS $(NF) in C  {print $(NF-4) OFS $(NF-1) OFS $NF
                        }
                        {print $(NF-1) OFS $NF
                        }
' FS="." OFS="." public_suffix_list.dat url.txt

I'm surprised this is working for you. There seem to be a few problems:
  1. The code shown in red in your awk script will never be executed. Since the condition on the two red condition/action sets is identical to the orange condition and the action section with that condition ends with a next command, the actions shown in red cannot be executed.
  2. I believe your code should explicitly ignore blank lines and comment lines in public_suffix_list.dat (unless you have pruned those lines out of the public suffix list provided here) when you downloaded the public list into your file).
  3. I don't see how this code handles wildcards in rules (e.g., *.sch.uk).
  4. I don't see how this code handles exception rules (although there aren't any exception rules if you're just trying to process UK domains).
  5. And, according to the rules published for the public list, you should be loading values in your array with C[$1] instead of C[$0], but I don't see anything in the public list that includes a comment at the end of any rules so (if you ignored comment lines and blank lines) it might not matter.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

2. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

3. UNIX for Dummies Questions & Answers

ReDirecting a URL to another URL - Linux

Hello, I need to redirect an existing URL, how can i do that? There's a current web address to a GUI that I have to redirect to another webaddress. Does anyone know how to do this? This is on Unix boxes Linux. example: https://m45.testing.address.net/host.php make it so the... (3 Replies)
Discussion started by: SkySmart
3 Replies

4. Windows & DOS: Issues & Discussions

How to: Linux BOX in Windows Domain (w/out joining the domain)

Dear Expert, i have linux box that is running in the windows domain, BUT did not being a member of the domain. as I am not the System Administrator so I have no control on the server in the network, such as modify dns entry , add the linux box in AD and domain record and so on that relevant. ... (2 Replies)
Discussion started by: regmaster
2 Replies

5. Web Development

Regex to rewrite URL to another URL based on HTTP_HOST?

I am trying to find a way to test some code, but I need to rewrite a specific URL only from a specific HTTP_HOST The call goes out to http://SUB.DOMAIN.COM/showAssignment/7bde10b45efdd7a97629ef2fe01f7303/jsmodule/Nevow.Athena The ID in the middle is always random due to the cookie. I... (5 Replies)
Discussion started by: EXT3FSCK
5 Replies

6. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

7. Shell Programming and Scripting

Hit multiple URL from a text file and store result in other test file

Hi, I have a problem where i have to hit multiple URL that are stored in a text file (input.txt) and save their output in different text file (output.txt) somewhat like : cat input.txt http://192.168.21.20:8080/PPUPS/international?NUmber=917875446856... (3 Replies)
Discussion started by: mukulverma2408
3 Replies

8. Shell Programming and Scripting

Reading URL using Mechanize and dump all the contents of the URL to a file

Hello, Am very new to perl , please help me here !! I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file. below is the script which i have written so far , #!/usr/bin/perl use LWP::UserAgent; use... (2 Replies)
Discussion started by: scott_cog
2 Replies

9. UNIX for Dummies Questions & Answers

Putting the colon infront of the URL domain

I have a file like this: http://hello.com www.examplecom computer Company I wanted to keep dot (.) infront of com. to make the file like this http://hello.com www.example.com computer Company I applied this expression sed -r 's/com/.com/g'but what I get is: http://hello.com ... (4 Replies)
Discussion started by: csim_mohan
4 Replies

10. UNIX for Dummies Questions & Answers

Extracting URL with domain

I have a file like this: http://article.wn.com/view/2010/11/26/IV_drug_policy_feels_HIV_patients_Red_Cross/ http://aidsjournal.com/,www.cfpa.org.cn/page1/page2 , www.youtube.com http://seattletimes.nwsource.com/html/jerrybrewer/2013517803_brewer25.html... (1 Reply)
Discussion started by: csim_mohan
1 Replies

LEARN ABOUT OSX

urn-scheme

uri_urn(n)					    Tcl Uniform Resource Identifier Management						uri_urn(n)

__________________________________________________________________________________________________________________________________________________

NAME

       uri_urn - URI utilities, URN scheme

SYNOPSIS

       package require Tcl  8.2

       package require uri::urn  ?1.1.2?

       uri::urn::quote url

       uri::urn::unquote url

_________________________________________________________________

DESCRIPTION

       This  package  provides two commands to quote and unquote the disallowed characters for url using the urn scheme, registers the scheme with
       the package uri, and provides internal helpers which will be automatically used by the commands uri::split and uri::join of package uri	to
       handle urls using the urn scheme.

COMMANDS

       uri::urn::quote url
	      This command quotes the characters disallowed by the urn scheme (per RFC 2141 sec2.2) in the url and returns the modified url as its
	      result.

       uri::urn::unquote url
	      This commands performs the reverse of ::uri::urn::quote. It takes an urn url, removes the quoting from  all  disallowed  characters,
	      and returns the modified urls as its result.

BUGS, IDEAS, FEEDBACK
       This  document,	and the package it describes, will undoubtedly contain bugs and other problems.  Please report such in the category uri of
       the Tcllib SF Trackers [http://sourceforge.net/tracker/?group_id=12883].  Please also report any ideas for enhancements you  may  have  for
       either package and/or documentation.

KEYWORDS

       rfc 2141, uri, url, urn

CATEGORY

       Networking

uri								       1.1.2								uri_urn(n)
All times are GMT -4. The time now is 07:08 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy