Visit Our UNIX and Linux User Community


Get domain names from IP addresses of apache2 access.log


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Get domain names from IP addresses of apache2 access.log
# 1  
Old 09-09-2013
Get domain names from IP addresses of apache2 access.log

I am totally new to shell scripting. I want to see people from which domain access my website. I want to generate the domain names from IP addresses in the Apache access.log file.
There are around 54 log files. I concatenate all the files into one.

I am using Ubuntu 12.04 LTS.

So I used a script as follows in the file ips.sh -

Code:
#!/bin/bash
while read junk 
do
         echo -n "$junk "         
         dig +short -x $junk 
done

Now I execute this command -

Code:
cut -f 1 -d ' ' access.log | sort | uniq | ips.sh

But this does not do anything as of now and throws an error - ips.sh - command not found. Please help.

Last edited by Ronni; 09-09-2013 at 06:13 AM..
# 2  
Old 09-09-2013
Quote:
Originally Posted by Ronni
Code:
cut -f 1 -d ' ' access.log | sort | uniq | ips.sh

But this does not do anything as of now and throws an error - ips.sh - command not found.
Let is start with this last piece: The reason why the system complains about not finding "ips.sh" is probably because it is not in the PATH: issue the following command:

Code:
echo $PATH

to see a list of directories separated by colons. Files in these directories you can call without using their full path (which is why you could use "cut" and "sort", etc.. They are all in "/usr/bin", which is in the PATH.), for all the others (this includes your current directory) you have to write:

Code:
cut -f 1 -d ' ' access.log | sort | uniq | /full/path/to/ips.sh

Another thing: sort | uniq is not necessary, because sort can do that too. Use the -u switch.

Code:
cut -f 1 -d ' ' access.log | sort -u | /full/path/to/ips.sh

Quote:
Originally Posted by Ronni
Code:
#!/bin/bash
while read junk 
do
         echo -n "$junk "         
         dig +short -x $junk 
done

This should work. I have no idea what dig does, but probably it displays something. You might want to quote "$junk" in the dig-line too. It will perhaps work anyway, but it is never too early to begin developing good habits and quoting variables to protect against them containing blanks is such a good habit.

For the same reason, you may want to define a variable before using it.

Code:
#!/bin/bash

typeset junk=""

while read junk 
do
         echo -n "$junk "         
         dig +short -x "$junk" 
done

I hope this helps.

bakunin
This User Gave Thanks to bakunin For This Post:
# 3  
Old 09-09-2013
Thanks. It works but I was trying to execute the statement within the folder that contains ips.sh. This should work right?

dig is domain information groper. I am using it to return Domain names by taking IP address as input.

Now it prints the data in the terminal. How can i make it print just the resolved domain names in a text file?

Moreover it prints a lot of junk and lots of IPs as well which i dont need. Can I get rid of them as well?

Last edited by Ronni; 09-09-2013 at 06:55 AM..
# 4  
Old 09-09-2013
Quote:
Originally Posted by Ronni
Thanks. It works but I was trying to execute the statement within the folder that contains ips.sh. This should work right?
As said above, what is not included in "PATH" will not be found, even if it is in the current directory. You have to use a path name then. Either by using an absolute path:

Code:
some | process | /full/path/to/script

or a "relative path". Use "." fur the current directory, ".." for the one above, etc.:

Code:
some | process | ./script           # execute from current directory
some | process | ../script          # execute from directory above
some | process | ../../script       # execute from directory two levels above

This is why it is customary to create a folder "/your/HOME/bin" and include this directory into your PATH. Issue the following on command line or put it into your file "~/.bashrc" (btw., "~" is short for "my home directory):

Code:
PATH="$PATH":~/bin
export PATH

Then move all your scripts there (or create symbolic links there) and you can use them everywhere without specifying their path name.

Quote:
Originally Posted by Ronni
Now it prints the data in the terminal. How can i make it print everything in a text file?
Use redirection:

Code:
some | process                   # goes to screen
some | process > /path/to/file   # goes to /path/to/file

I hope this helps.

bakunin

Last edited by bakunin; 09-09-2013 at 07:02 AM.. Reason: typo
# 5  
Old 09-09-2013
I haven't seen the last part of your second post:

Quote:
Moreover it prints a lot of junk and lots of IPs as well which i dont need. Can I get rid of them as well?
perhaps, but this means you have to show us a qualified example of what your data looks like. Unix is full of very flexible, programmable text filters (you already found "cut", "sort" and "uniq", but there are more, lots more!) but to find out which one is best suited to your problem one has to analyse your data.

I hope this helps.

bakunin
 

Previous Thread | Next Thread
Test Your Knowledge in Computers #436
Difficulty: Easy
React is an open source JavaScript library.
True or False?

9 More Discussions You Might Find Interesting

1. Solaris

Certain domain cannot access

hi there, im beginner in solaris 10. we are using solaris 10 as our DNS server Solaris 10 8/07 s10s_u4wos_12b SPARC Copyright 2007 Sun Microsystems, Inc. All Rights Reserved. Use is subject to license terms. ... (0 Replies)
Discussion started by: tappetmus
0 Replies

2. Shell Programming and Scripting

Filter out IP addresses in domain resolve script

Hello everyone, I have written a one liner which looks in a configuration file for remote hosts to connect to. It then resolves them by first ping, and nslookup if it fails, then places the info in the hosts file. The whole thing works, seemingly, apart from one slight issue. I would like to... (0 Replies)
Discussion started by: haggismn
0 Replies

3. AIX

How do I restrict TFTP access to certain hosts/IP addresses?

Hi Everyone, I searched for an answer to this but couldn't find one so I'm hoping someone can lend some advice. My issue is that I have an AIX server running Sysback (for TSM backup/restore) and Sysback uses TFTP for sending the boot image to the client during a restore. A recent penetration... (2 Replies)
Discussion started by: need2bageek
2 Replies

4. Shell Programming and Scripting

How can i send mail to multiple addresses in same domain in bash?

Suppose i have a txt file that is the list of the addresses,something like: lala0045 john james lala0234 george james and i want to send an email to lala0045@blabla.com and lala0234@blabla.com,the same domain...what is the exact syntax i should use in my script? there is a command... (10 Replies)
Discussion started by: bashuser2
10 Replies

5. Shell Programming and Scripting

extracting domain names out of a text file

I am needing to extract and list domain names out of a very large text file. The text file contains tlds .com .net .org and others as well as third level domains e.g. host1.domain.com and the names are placed within paragraphs of text. Domains do not have a http:// prefix so I'm thinking the... (6 Replies)
Discussion started by: totus
6 Replies

6. Shell Programming and Scripting

please help, find domain names in string

Hello, i have a file contains the information like below /home/username/domain.com/log/access /home/username/domain23.net/log/access /home/reseller/username/domain.com/log/access using a loop i can read every line of the file but i wants to extract domain name like(domain.com,... (3 Replies)
Discussion started by: eyes_drinker
3 Replies

7. IP Networking

find computer names from IP addresses?

Arright, here's what I'm trying to do. I want to dig up currently active IP addresses on my subnet, and my present strategy is to ping every address until I find active ones, then ping them more often to verify their status. Next, I want to find the names of the computers associated with those... (1 Reply)
Discussion started by: sladuuch
1 Replies

8. UNIX for Dummies Questions & Answers

Using Sendmail for multiple domain names

Hi, We're an internet company with several domain names. Our mail server was originally set up to deal with xxx@domain1.com email addresses which works fine. The problem I have is that we're now also using a domain2.com, and sales@domain1.com isn't the same as sales@domain2.com. I've added... (1 Reply)
Discussion started by: captainash
1 Replies

9. IP Networking

using unregistered domain names

hey what the hell happens if you make sure (as best one can) that a domain name like anything.com is not used at all, and you set up your own DNS and use that name without registering with a registrar, i know if the address is in use you will make some people very upset and give many internet users... (2 Replies)
Discussion started by: norsk hedensk
2 Replies

Featured Tech Videos