Strip subdomains from domains list


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Strip subdomains from domains list
# 1  
Old 08-27-2010
Strip subdomains from domains list

Hello,

I'm trying to strip all the subdomains from a domains list.

Let's say the list is:

----------
http://add.my.yahoo.com
ad.doubleclick.net
admin.searchhippo.com
www.advertise.about.com.br
----------

I need to get

-----------
yahoo.com
doubleclick.net
searchhippo.com
about.com.br
------------
# 2  
Old 08-27-2010
One way:
Code:
awk -F\. '{print $(NF-1) FS $NF}' file

# 3  
Old 08-27-2010
Quote:
Originally Posted by Franklin52
One way:
Code:
awk -F\. '{print $(NF-1) FS $NF}' file

Thanks, this is almost good.

The problem is that it discards Country specific domains like
"bol.com.br" for example.
# 4  
Old 08-27-2010
Quote:
Originally Posted by rlopes
Thanks, this is almost good.

The problem is that it discards Country specific domains like
"bol.com.br" for example.
Then you need tell us in which way you need keep three sections in a domain name.
# 5  
Old 08-27-2010
#removed#
# 6  
Old 08-27-2010
According to the ISO standard:
Quote:
All ASCII ccTLD identifiers are two letters long, and all two-letter top-level domains are ccTLDs.
Thus, you could assume that any domain ending in a two character code should print the last two subdomain names in addition to the top level domain:

Code:
# assume url is first field; change if needed
awk '
        {
                gsub( "^.*://", "", $1 );      # ditch the http://  ftp:// etc
                n = split( $1, a, "." );
                if( length( a[n] ) == 2 )       # assuming all two character top level domains are country codes
                        printf( "%s.%s.%s\n", a[n-2], a[n-1], a[n] );
                else
                        printf( "%s.%s\n",  a[n-1], a[n] );
        }
' list-file-name

This also takes into account that
Code:
http://foo.com

should not print anything before the double slants. Caution may need to be exercised as some countries have licensed the use of their top level domain name (e.g. .tv and .fm).
# 7  
Old 08-27-2010
Code:
#!/bin/bash
exec 6<"myfile"
while read -r LINE<&6
do
   LINE=${LINE/www./}
   IFS="."
   LINE=(${LINE/http:\/\/})
   want="${LINE[@]:1}"
   echo "${want// /.}"
done
exec 6<&-

Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. IP Networking

VLANs and their domains

Hey everyone. I work in a data center, and I'm working on getting my CCNA. Now when I read articles on the idea of VLAN's it makes sense. Especially if you have multiple switches daisy chained in multiple locations. My two main questions though are that most of these examples use PC's as examples... (2 Replies)
Discussion started by: Lost in Cyberia
2 Replies

2. UNIX for Dummies Questions & Answers

Sub domains from report

Hi, I have a report containing severals organization's email address. The address contain several sub domains, and i need to pull those out. mail domain ( example.com) .................. The report column contain mail address in this format : john1@sub1.example.com... (2 Replies)
Discussion started by: john_prince
2 Replies

3. Shell Programming and Scripting

find out subdomains

I am the new user I want to find out sub domains in the URL using shell script. e.g http://abcd:8380/matcher This is the URL. The suburl is given below http://abcd:8380/matcher/servlet/viewnudatasource http://abcd:8380/matcher/servlet/addnudatasource... (4 Replies)
Discussion started by: mnmonu
4 Replies

4. UNIX for Advanced & Expert Users

Different Nameservers for Different Domains?

I have a system that is connected to a private network with its own DNS (call it "privnet."), and is also connected to the Internet on a separate interface. Is it possible to convince this server to query the private nameserver for the private network's domain (e.g. "host foo.privnet."), and the... (2 Replies)
Discussion started by: vertigo23
2 Replies

5. UNIX for Dummies Questions & Answers

Automating creation of sub-domains???

Edit, Sorry realised that muxtape.com is probably not creating a sub-domain for every accout but more than likely using a wildcard in the apache config and then using php/asp/java or similar to look at the url and point it in the right direction. So youraccount.muxtape.com maps to... (0 Replies)
Discussion started by: elduderino
0 Replies

6. UNIX for Dummies Questions & Answers

creating domains

ok i am setting up dns or going to do it with solaris 9 once u setup the domain what file can u look @ to see if it setup or not (4 Replies)
Discussion started by: rmuhammad
4 Replies

7. UNIX for Dummies Questions & Answers

blocking domains

Dear All , Kindly note I have sun solaries 7 . I want to block a domain who keep sending emails to my domain and users . thanks (1 Reply)
Discussion started by: tamemi
1 Replies

8. UNIX for Dummies Questions & Answers

multiple domains

Hello, I have 3 domains virtually hosted "name based" the first one "domain1.com" has its ServerName entered as domain1.com. this domain will load in a browser by www.domain1.com or simply domain1.com. the next two domains "domain2.com" and "domain3.com" ServerNames are listed as domain2.com and... (2 Replies)
Discussion started by: ericg
2 Replies

9. IP Networking

Setting up subdomains...

Hello, I'd like to set up a subdomain. For instance, me.mydomain.com, instead of mydomain.com/me. I have a dedicated unix server running Apache OS. How do I do this? Any help given will be greatly appreciated and thanks in advance. -Matt (1 Reply)
Discussion started by: matman
1 Replies
Login or Register to Ask a Question