UNIX for Dummies Questions & Answers

I am totally new to shell scripting. I want to see people from which domain access my website. I want to generate the domain names from IP addresses in the Apache access.log file.
There are around 54 log files. I concatenate all the files into one.

I am using Ubuntu 12.04 LTS.

So I used a script as follows in the file ips.sh -

Code:

#!/bin/bash
while read junk 
do
         echo -n "$junk "         
         dig +short -x $junk 
done

Now I execute this command -

Code:

cut -f 1 -d ' ' access.log | sort | uniq | ips.sh

But this does not do anything as of now and throws an error - ips.sh - command not found. Please help.

Let is start with this last piece: The reason why the system complains about not finding "ips.sh" is probably because it is not in the PATH: issue the following command:

Code:

echo $PATH

to see a list of directories separated by colons. Files in these directories you can call without using their full path (which is why you could use "cut" and "sort", etc.. They are all in "/usr/bin", which is in the PATH.), for all the others (this includes your current directory) you have to write:

Code:

cut -f 1 -d ' ' access.log | sort | uniq | /full/path/to/ips.sh

Another thing: sort | uniq is not necessary, because sort can do that too. Use the -u switch.

Code:

cut -f 1 -d ' ' access.log | sort -u | /full/path/to/ips.sh

This should work. I have no idea what dig does, but probably it displays something. You might want to quote "$junk" in the dig-line too. It will perhaps work anyway, but it is never too early to begin developing good habits and quoting variables to protect against them containing blanks is such a good habit.

For the same reason, you may want to define a variable before using it.

Code:

#!/bin/bash

typeset junk=""

while read junk 
do
         echo -n "$junk "         
         dig +short -x "$junk" 
done

I hope this helps.

bakunin

Thanks. It works but I was trying to execute the statement within the folder that contains ips.sh. This should work right?

dig is domain information groper. I am using it to return Domain names by taking IP address as input.

Now it prints the data in the terminal. How can i make it print just the resolved domain names in a text file?

Moreover it prints a lot of junk and lots of IPs as well which i dont need. Can I get rid of them as well?

As said above, what is not included in "PATH" will not be found, even if it is in the current directory. You have to use a path name then. Either by using an absolute path:

Code:

some | process | /full/path/to/script

or a "relative path". Use "." fur the current directory, ".." for the one above, etc.:

Code:

some | process | ./script           # execute from current directory
some | process | ../script          # execute from directory above
some | process | ../../script       # execute from directory two levels above

This is why it is customary to create a folder "/your/HOME/bin" and include this directory into your PATH. Issue the following on command line or put it into your file "~/.bashrc" (btw., "~" is short for "my home directory):

Code:

PATH="$PATH":~/bin
export PATH

Then move all your scripts there (or create symbolic links there) and you can use them everywhere without specifying their path name.

Use redirection:

Code:

some | process                   # goes to screen
some | process > /path/to/file   # goes to /path/to/file

I hope this helps.

bakunin

I haven't seen the last part of your second post:

perhaps, but this means you have to show us a qualified example of what your data looks like. Unix is full of very flexible, programmable text filters (you already found "cut", "sort" and "uniq", but there are more, lots more!) but to find out which one is best suited to your problem one has to analyse your data.

I hope this helps.

bakunin