Shell Programming and Scripting

Hi All,

We have a file "Customers.lst". It contains list of all the Customers.

There is directory which has number of text files and each file containing name of defaulter customers.

We want to search for all the customers available in "Customers.lst" file against the list of files containing the name of default customers and then
print the Default customer name and the file in which it is available.

Currently we are reading Customers.lst file line by line and then doing grep in the directory.

Code:

for customername in `cat ${customerlistfile}`
do
   cd $dir
   echo "Searching for $customername in the $dir" >> $mainlog
   find . -name "*"|xargs grep -il $customername |sed 's/.*/'"${customername}"', &/' >> $mainout
done

Is there a better way to do this operation or any suggestions on the above command to make it faster.

REgards,
Arun M

Please use code tags.
A few minor suggestions to your code:
= Useless Use Of Cat; no need
= '-name "*"' is redundant, be more specific or omit, if searching through all files
= sed doesn't need to do matching; use anchor '^'
= do use double quotes in grep pattern variable (no need inside sed command though)

Is there a reason why you 'cd' in each loop iteration? Does $dir change?

Code:

while read customername ; do 
   echo "Searching for $customername in the $dir" >> $mainlog
   find $dir | xargs grep -il "$customername" | 
     sed 's/^/'${customername}', /' >> $mainout
done <  ${customerlistfile}

Thanks a lot for your reply. It helps a lot.

I have follow up questions for my understanding as still novice in shell scripting.

a) Which looping construct is more efficient in case the file to loop through is big one. In our case customer list file will be 25k-35k records ??

b) do use double quotes in grep pattern variable (no need inside sed command though)

Could you plz explain this suggestion.

Thanks once again.

Arun

a) <file redirection is not too much more efficient than your for loop. It really just saves one process (cat). The best is to avoid shell loop and run the file through a filter, like awk.
And since you're not using find(1) to do anything super useful, it can be omitted as well. Something along these lines:

Code:

awk -v "dir=$someDir" -v "log=some.log" '{print "Searching for "$0 >> log; printf("grep -il \"%s\" * | sed \"s/^/%s, /\"\n", $0,$0); }' Customers.lst

This will print the commands to stdout. Look at it, take one, and execute it, and if everythinh seems well, pipe it to bash and capture the output to a file:

Code:

awk -v "dir=$someDir" -v "log=some.log" '{print "Searching for "$0  >> log; printf("grep -il \"%s\" * | sed \"s/^/%s, /\"\n", $0,$0);  }' Customers.lst | sh > output.log

b) if you omit double quotes and you're searching for a 2-word pattern, grep takes second word as the file to do search on:

Code:

$ cat file 
John Doe
John OtherDoe
$ grep John Doe file 
grep: Doe: No such file or directory
file:John Doe
file:John OtherDoe
$ grep "John Doe" file 
John Doe

Hi,

Thanks for the reply.

I tried using the example above it is throwing syntaxt error whenever trying to use log or dir.

Could you please explain :

a) "grep -il \"%s\" * " : How this will allow search of $0 in the current directories and its subsirectories ??

b) sed \"s/^/%s, /\"\n", $0,$0); : This sed ??


Regards,
Arun M

Sorry, the redirection into variable doesn't work as expected. The following, inserting it straight from shell, works fine.
Let me format it nicer:

Code:

sh$ logFile=someLog.txt
sh$ awk '{
  print "Searching for "$0 > "'$logFile'"; 
  printf("grep -il \"%s\" * | sed \"s/^/%s, /\"\n", $0,$0); 
}' Customers.lst

Input:

Code:

sh$ cat Customers.lst
John Doe
Alice In Wonderland
John Other

Outputs:

Code:

sh$ logFile=somOther.log; awk '
{print "Searching for " $0 > "'$logFile'" ; printf("grep -il \"%s\" * | sed \"s/^/%s, /\"\n", $0,$0); 
}' Customers.lst
grep -il "John Doe" * | sed "s/^/John Doe, /"
grep -il "Alice In Wonderland" * | sed "s/^/Alice In Wonderland, /"
grep -il "John Other" * | sed "s/^/John Other, /"

So awk is used to print out a bunch of commands with your names from Customers.lst. Nicely each command on one line.
Now what does each command do when you execute it:

Code:

grep -il "John Doe" *

will search for "John Doe", case insensitive (-i), in all files present (*) and output the filename only if string is found (-l).
So if you have 3 log files present in current directory:

Code:

sh$ ls
file1.log
file2.log
file3.log

and only one -- file2.log -- contains "John Doe", then:

Code:

sh$ grep -il "John Doe" *
file2.log

that's the filename that grep returns.
Now you want <name>, <filenameThatContains_name>
so that's what sed does; it inserts 'John Doe, ' in the beginning of line ('^'), so:

Code:

sh$ echo file2.log | sed "s/^/John Doe, /"
John Doe, file2.log

Those '%s' are format specifiers for printf, it's telling printf to print string, and $0,$0 are arguments to printf, telling it to substitute $0 (which is the whole record -- the name (if you don't have anything else on line of Customers.lst) in awk) for %s.

So the awk magic is just to format and print out a nice command to the screen. Then you can test out one of those commands, if you wish, or just assess them visually (useful for debugging, before you actually run them).
Then pipe to bash to execute, and redirect to capture output.
You might need to insert a directory name for grep's argument, instead of just plain '*' to adjust for details of your dir structure.

I know it's not quite elegant, but should be faster.
Loops in shell are not nearly as efficient. Awk excels in the speed of reading lines from input -- it's a filter, after all, well crafted for this particular purpose. Then you are gonna launch a process for each grep; this can be optimized further (parallelized onto the CPUs, e.g).

If you want, you can do a little benchmark, take a subset of your logfiles, and process them with our script and mine.
Run it with 'time' like:

Code:

time ./scriptToRun.sh

which will spit out the time it takes for script to finish. I'd be curious...
Also I can't wait for answers of more experienced *nix people.

Hi,

Thanks for the detailed reply.

awk is faster than the looping at present based on current run. I have follow up questions :

a) We provided the list with 5000 customers, it ran pretty fast for first 500-800 then it is taking long time as it is running though the list.

Is it related to buffer or memory allocated ??

Regards,
Arun M