Performance issue

02-27-2011

Moderator

12,296, 3,792

Join Date: Nov 2008

Last Activity: 1 January 2021, 1:47 AM EST

Location: Amsterdam

Posts: 12,296

Thanks Given: 679

Thanked 3,792 Times in 3,282 Posts

The open/close operations can be limited if we simply use this instead, so writes can be done more efficiently (option B):

Code:

while read first
do
  while read second
  do
    while read third
    do
       printf "%-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s\n" $first $second $third 4 5 1 2 1 5 2 
    done < $3
  done < $2
done < $1 > file.txt

Also reads can be optimized in shell by reading the files at once (option C):

Code:

file1=$(cat "$1")
file2=$(cat "$2")
file3=$(cat "$3")
for first in $file1
do
  for second in $file2
  do
    for third in $file3
    do
       printf "%-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s\n" "$first" "$second" "$third" 4 5 1 2 1 5 2
    done
  done
done  > file.txt

Since there is only 1 variable per line we can do it like this, otherwise we would need to adjust IFS.

Also the shell used can have quite an impact. I did some speed test with files with 32 lines each with methods A, B and C. I tested with bash, ksh93 and dash.
Also I used Pludi's perl script of this thread (without the system date calls) and the following awk script for comparison:

Code:

awk 'BEGIN{
       f1="file1";f2="file2";f3="file3"
       while(getline x<f1) {
         while(getline y<f2) {
           while(getline z<f3)
             printf "%-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s\n",x,y,z,4,5,1,2,1,5,2
           close(f3)
         }
         close(f2)
       }
      }' > file.out

These are the results (time in seconds):

		bash	dash	ksh93	perl	awk
A	line by line	60,34	10,67	14,69
B	line by line	50,93	8,14	8,09		0,87
C	read into memory	31,23	5,50	7,50	1,55

Quote:

Originally Posted by pludi

[..] Each call to printf costs the time of a process spawn. And the file has to be opened, written, and closed on each call.

printf is a shell builtin, so in this case no spawn is needed...

Last edited by Scrutinizer; 02-27-2011 at 11:40 AM..

Scrutinizer

View Public Profile for Scrutinizer

Find all posts by Scrutinizer

UNIX for Dummies Questions & Answers