Shell Programming and Scripting

Good evening

Im new at unix shell scripting and im planning to script a shell that removes headers for about 120 files in a directory and each file contains about 200000
lines in average.


i know i will loop files to process each one and ive found in this great forum different solutions using grep, sed, awk, head, etc.

But according to the above scenario and your experince and knowledge which command is the best for performance and does the homework faster ?

Thanks in advance

Assuming header is first line, I compared execution speed of sed and awk on a 450,251 line file. Here are the results:-

Code:

HP-UX B.11.31 U ia64

Code:

wc -l infile
450251 infile

Code:

time awk 'FNR>1' infile > out

real    0m5.45s
user    0m2.21s
sys     0m2.86s

Code:

time sed '1d' infile > out

real    0m2.90s
user    0m1.22s
sys     0m1.43s

In this case sed won But it depends on what you are trying to do.

Did sed win? Or did file caching speed up sed? Modern controllers and RAM cache - HPUX - can cache 100 MB of a single file without really using up system resources.

I vote for caching. The only fair test is two separate files.

BTW: programs like sed, awk, head, tail, grep are all highly optimized for their respective jobs. There are several of external factors like: caching, I/O load (I/O request queue length), SAN vs disk, that distort these kinds of tests. So, by the time you have runs some tests, any time differences between the commands will likely have been eaten up by testing.

Your best bet is to parallelize, use the cpu and disk I/O to the max. With a quad core maybe you want to consider 4 simultaneous child processes, for example:

Code:

cd /directory
cnt=1
for fname in $(find . -type f)
do
   (awk 'FNR>1' $fname > tmp.${cnt}; mv tmp.${cnt} $fname)  &
   cnt=$(( $cnt + 1  ))
   [  $(( $cnt % 4 )) -eq 0 ]  && wait
done
wait

Jim, do you mean file caching helped sed because of the sequence of execution I chose? If yes, I tried the other way and still sed took less time to complete this particular task.

Code:

# time sed '1d' infile > out

real    0m3.41s
user    0m1.22s
sys     0m1.62s

Code:

# time awk 'FNR>1' infile > out

real    0m5.60s
user    0m2.20s
sys     0m3.21s

Yes, that was what I meant. And yes it is very likely the grep, egrep, and sed are better at massive I/O than awk, which is running interpreted. The point, I think, is that a lot of tests like this are a lot of fun, but they may not be informative. Unless you understand why results can be set askew.

On my large m4000 Solaris boxes sed always outperforms awk on simple stream editing of massive files. On cygwin they come out really close.

However, by the time I've set up a fair test and run several candidates through, I could have coded and already processed 24 files in parallel, using any reasonable method.

Which is a lot less fun, I admit.

OK GREAT, thanks you very much all of you for your time and knowledge. ill start working with the script and then testing it.