Sponsored Content
Top Forums Shell Programming and Scripting Get only domain from url file bind Post 302959497 by RudiC on Tuesday 3rd of November 2015 04:15:51 PM
Old 11-03-2015
Code:
awk '
NR==FNR                 {C[$0]                  # read first file (= NR==FNR) into the indices of the associative array C
                         next                   # stop processing the actual line; proceed with next line
                        }
$(NF-1) OFS $(NF) in C  {print $(NF-2) OFS $(NF-1) OFS $NF
                                                # if second last ($(NF-1) and last ($NF) fields, joint by a dot, are found in C
                                                # print third last, second last, and last field
                         next                   # stop ... see above
                        }
                        {print $(NF-1) OFS $NF  # if above doesn't apply, print second last and last fields 
                        }
' FS="." OFS="." publicsuffix.lst raw           # supply the field separators and two files to awk

This code certainly is not perfect; e.g. the co.nz is missing in the publicsuffix.lst, but it may serve as a starting point...
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

2. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

3. UNIX for Dummies Questions & Answers

ReDirecting a URL to another URL - Linux

Hello, I need to redirect an existing URL, how can i do that? There's a current web address to a GUI that I have to redirect to another webaddress. Does anyone know how to do this? This is on Unix boxes Linux. example: https://m45.testing.address.net/host.php make it so the... (3 Replies)
Discussion started by: SkySmart
3 Replies

4. Windows & DOS: Issues & Discussions

How to: Linux BOX in Windows Domain (w/out joining the domain)

Dear Expert, i have linux box that is running in the windows domain, BUT did not being a member of the domain. as I am not the System Administrator so I have no control on the server in the network, such as modify dns entry , add the linux box in AD and domain record and so on that relevant. ... (2 Replies)
Discussion started by: regmaster
2 Replies

5. Web Development

Regex to rewrite URL to another URL based on HTTP_HOST?

I am trying to find a way to test some code, but I need to rewrite a specific URL only from a specific HTTP_HOST The call goes out to http://SUB.DOMAIN.COM/showAssignment/7bde10b45efdd7a97629ef2fe01f7303/jsmodule/Nevow.Athena The ID in the middle is always random due to the cookie. I... (5 Replies)
Discussion started by: EXT3FSCK
5 Replies

6. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

7. Shell Programming and Scripting

Hit multiple URL from a text file and store result in other test file

Hi, I have a problem where i have to hit multiple URL that are stored in a text file (input.txt) and save their output in different text file (output.txt) somewhat like : cat input.txt http://192.168.21.20:8080/PPUPS/international?NUmber=917875446856... (3 Replies)
Discussion started by: mukulverma2408
3 Replies

8. Shell Programming and Scripting

Reading URL using Mechanize and dump all the contents of the URL to a file

Hello, Am very new to perl , please help me here !! I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file. below is the script which i have written so far , #!/usr/bin/perl use LWP::UserAgent; use... (2 Replies)
Discussion started by: scott_cog
2 Replies

9. UNIX for Dummies Questions & Answers

Putting the colon infront of the URL domain

I have a file like this: http://hello.com www.examplecom computer Company I wanted to keep dot (.) infront of com. to make the file like this http://hello.com www.example.com computer Company I applied this expression sed -r 's/com/.com/g'but what I get is: http://hello.com ... (4 Replies)
Discussion started by: csim_mohan
4 Replies

10. UNIX for Dummies Questions & Answers

Extracting URL with domain

I have a file like this: http://article.wn.com/view/2010/11/26/IV_drug_policy_feels_HIV_patients_Red_Cross/ http://aidsjournal.com/,www.cfpa.org.cn/page1/page2 , www.youtube.com http://seattletimes.nwsource.com/html/jerrybrewer/2013517803_brewer25.html... (1 Reply)
Discussion started by: csim_mohan
1 Replies
makedbm(8yp)															      makedbm(8yp)

Name
       makedbm - make a yellow pages dbm file

Syntax
       makedbm [ -i yp_input_file ] [ -o yp_output_name ] [ -d yp_domain_name ] [ -m yp_master_name ] infile outfile
       makedbm [ -u dbmfilename ]

Description
       The command takes the file specified by the argument infile and converts it to a pair of files in format, namely and Each line of the input
       file is converted to a single record.  All characters up to the first tab or space form the key, and the rest of the line is defined as the
       key's  associated data.	If a line ends with a backslash (), the data for that record is continued onto the next line.	It is left for the
       clients of the yellow pages to interpret the number sign (#); does not treat it as a comment character.	The  infile  parameter	can  be  a
       hyphen (-), in which case reads the standard input.

       The  command  is  meant	to  be	used in generating files for the yellow pages service.	The command generates a special entry with the key
       yp_last_modified, which is the date of infile.

Options
       -i     Create a special entry with the key yp_input_file.

       -o     Create a special entry with the key yp_output_name.

       -d     Create a special entry with the key yp_domain_name.

       -m     Create a special entry with the key yp_master_name.  If no master host name is specified, yp_master_name will be set  to	the  local
	      host name.

       -u     Undo a file.  That is, print out a file one entry per line, with a single space separating keys from values.

Examples
       The  following  example shows how a combination of commands can be used to make the yellow pages files and from the file.  The percent sign
       (%) signifies the system prompt.

       % awk 'BEGIN { FS = ":"; OFS = ""; }
	{ print $1, $0 }' /etc/passwd > ptmp
       % makedbm ptmp passwd.byname
       % rm ptmp

       The command creates the file ptmp which is in a form usable by The command uses the ptmp file to create the yellow pages dbm files and  The
       command removes the ptmp file.

See Also
       yppasswd(1yp), dbm(3x), ypmake(8yp)

																      makedbm(8yp)
All times are GMT -4. The time now is 11:58 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy