Performance issue


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Performance issue
# 1  
Old 02-24-2011
Performance issue

hi I am having a performance issue with the following requirement
i have to create a permutation and combination on a set of three files
such that each record in each file is picked and the output is redirected in
a specific format but it is taking around 70 odd hours to prepare a
combination of 6, 1500, 2351 records in files
file 1 - 6 records
file 2 - 1500 records
file3 - 2351 records
total record in file.txt below is 6*1500*2351=21159000
script i am using is

Code:
 
echo 'date'
while read first
 do
  while read second
      do
        while read third
         do
            printf "%-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s\n" $first $second $third 4 5 1 2 1 5 2 >> file.txt
          done < $3
        done < $2
   done < $1
date

Please Advice

Moderator's Comments:
Mod Comment It's [CODE], not <code>

Last edited by pludi; 02-24-2011 at 09:37 AM.. Reason: code tags
# 2  
Old 02-24-2011
You're probably losing most of the time on the actual output. Each call to printf costs the time of a process spawn. And the file has to be opened, written, and closed on each call.

The below Perl script creates the permutations (same number of records as in your example) in just over 2 minutes on an average sized laptop
Code:
# ./perm.pl
Thu Feb 24 15:28:41 CET 2011
Thu Feb 24 15:30:56 CET 2011
# cat perm.pl
#!/usr/bin/perl

use strict;
use warnings;

my ( $fh1, $fh2, $fh3, $out );
my ( @file1, @file2, @file3 );
system("date");
{
    local $/;
    open $fh1, '<', 'file1';
    my $file1 = <$fh1>;
    @file1 = split /\n/, $file1;
    close $fh1;
}
{
    local $/;
    open $fh2, '<', 'file2';
    my $file2 = <$fh2>;
    @file2 = split /\n/, $file2;
    close $fh2;
}
{
    local $/;
    open $fh3, '<', 'file3';
    my $file3 = <$fh3>;
    @file3 = split /\n/, $file3;
    close $fh3;
}

chomp @file1;
chomp @file2;
chomp @file3;

open $out, '>', 'file2.txt';
foreach my $first (@file1) {
    foreach my $second (@file2) {
        foreach my $third (@file3) {
            printf $out "%-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s\n",
              $first, $second, $third, 4, 5, 1, 2, 1, 5, 2;
        }
    }
}
close $out;
system("date");

---------- Post updated at 15:46 ---------- Previous update was at 15:38 ----------

Addendum: on the very same setup, the script as you have given it managed to produce ~50% of all combinations in about 17 minutes. There has to be something entirely different wrong with your setup for it to take 70 hours.
# 3  
Old 02-24-2011
giving error while executing

Thanks a lot but but to be frank i dont have any knowledge about perl
so what i did was to put the script and file1, file2 file3 in a directory and
ran the script but it gives me a syntax error

Code:
 
./format.pl [4] use: not found
./format.pl [5] use: not found
./format.pl: 0403-057 Syntax error at line 7 : `(' not expected

Please Advice
# 4  
Old 02-24-2011
If you get that error, you've left out the shebang line (#!/usr/bin/perl), because your shell is trying to interpret it.

Second, the shell script finished too by now, after 33 minutes. So if you didn't exaggerate and it really needs 70 hours to finish on your system there's something different very very wrong.
# 5  
Old 02-24-2011
Thanks a Lot Pludi, the script ran wonder and completed in 1hour 36 min
Much Appreciate the help.Smilie
# 6  
Old 02-27-2011
Quote:
Originally Posted by pludi
You're probably losing most of the time on the actual output. Each call to printf costs the time of a process spawn. And the file has to be opened, written, and closed on each call.

The below Perl script creates the permutations (same number of records as in your example) in just over 2 minutes on an average sized laptop
Code:
# ./perm.pl
Thu Feb 24 15:28:41 CET 2011
Thu Feb 24 15:30:56 CET 2011
# cat perm.pl
#!/usr/bin/perl

use strict;
use warnings;

my ( $fh1, $fh2, $fh3, $out );
my ( @file1, @file2, @file3 );
system("date");
{
    local $/;
    open $fh1, '<', 'file1';
    my $file1 = <$fh1>;
    @file1 = split /\n/, $file1;
    close $fh1;
}
{
    local $/;
    open $fh2, '<', 'file2';
    my $file2 = <$fh2>;
    @file2 = split /\n/, $file2;
    close $fh2;
}
{
    local $/;
    open $fh3, '<', 'file3';
    my $file3 = <$fh3>;
    @file3 = split /\n/, $file3;
    close $fh3;
}

chomp @file1;
chomp @file2;
chomp @file3;

open $out, '>', 'file2.txt';
foreach my $first (@file1) {
    foreach my $second (@file2) {
        foreach my $third (@file3) {
            printf $out "%-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s\n",
              $first, $second, $third, 4, 5, 1, 2, 1, 5, 2;
        }
    }
}
close $out;
system("date");

---------- Post updated at 15:46 ---------- Previous update was at 15:38 ----------

Addendum: on the very same setup, the script as you have given it managed to produce ~50% of all combinations in about 17 minutes. There has to be something entirely different wrong with your setup for it to take 70 hours.
I would suggest to use perl builtin for time elapsed in seconds instead of using system call for date. But really minor.
# 7  
Old 02-27-2011
True, if I wanted to time specific sections and/or do further calculations on the timings. But for a simple comparison of (logically) equal code I think this suffices.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

Performance issue in Linux

IN solaris, for network high-availability we are using IPMP concept, can u tell me in REDHAT LINUX what we are using... also pls share good step to read & understand the that concept... Also performance issue in linux what are step & cmd can u tell me??? (2 Replies)
Discussion started by: tiger09
2 Replies

2. AIX

Performance issue

Hi We have an AIX5.3 server with application which is written in C. We are facing server (lpar) hangs intermediately. If we open new telnet window prompts for user and takes hell of a time to authenticate, not only that if we run ps -aef then also it takes lot of time. surprisingly there is no... (2 Replies)
Discussion started by: powerAIX
2 Replies

3. AIX

Performance issue

Hi, We have 2 lpars on p6 blade. One of the lpar is having 3 core cpu with 5gb memory running sybase as database. An EOD process takes 25 min. to complete. Now we have an lpar on P7 server with entitled cpu capacity of 2 with 16 Gb memory and sybase as database. The EOD process which takes... (17 Replies)
Discussion started by: vjm
17 Replies

4. Shell Programming and Scripting

Performance issue or something else?

Hi All, I have the following script which I use in Nagios to check the health of the applications, the problem with it is that the curl part ($TOTAL) does not return anything after running for 2-3 hrs, even though from command line the script runs fine but not from Nagios. There are 17... (1 Reply)
Discussion started by: jacki
1 Replies

5. Solaris

Performance issue

Hi Gurus, I am beginner in solaris and want to know what are the things we need to check for performance monitoring on our solairs OS. for DISK,CPU and MEMORY. Also how we do ipforwarding in slaris Many thanks for your help Pradeep P (4 Replies)
Discussion started by: ppandey21
4 Replies

6. UNIX for Advanced & Expert Users

Performance issue!

In my C program i am using very large file(approx 400MB) to read parts of it frequently. But due to large file the performance of the program goes down very badly. It shows very high I/O usage and I/O wait time. My question is, What are the ways to optimize or tune I/O on linux or how i can get... (10 Replies)
Discussion started by: mavens
10 Replies

7. Shell Programming and Scripting

performance issue

I want to read a file. is it good to use File I/O or shell script?? which one is the best option? (1 Reply)
Discussion started by: vishwaraj
1 Replies

8. UNIX for Advanced & Expert Users

performance issue

Hi, on a linux server I have the following : vmstat 2 10 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 4 0 675236 39836 206060 1617660 3 3 3 6 8 7 1 1 ... (1 Reply)
Discussion started by: big123456
1 Replies

9. AIX

performance issue

We have a AIX v5.3 on a p5 system with a poor performing Ingres database. We added one CPU to the system to see if this would help. Now there are two CPU's. with sar and topas -P I see good results: CPU usage around 30% with topas I only see good results in the process output screen, the... (1 Reply)
Discussion started by: rein
1 Replies

10. UNIX for Advanced & Expert Users

Performance issue

Hello all, I just stuck up in an uncertain situation related to network performance... I am trying to access one of my remote client unix machine from a distant location.. The client machine is Ultra-5_10 , with SunOS 5.5.1 The ndd result ( hme1 )shows that the machine is hooked to a... (5 Replies)
Discussion started by: shibz
5 Replies
Login or Register to Ask a Question