Sponsored Content
Top Forums Shell Programming and Scripting Perl sum really inefficient!! Post 302313352 by pludi on Tuesday 5th of May 2009 11:39:44 AM
Old 05-05-2009
First, for future reference, please put your formatted code inside [code][/code] tags

Second, there are quite a few things wrong with your code.
Code:
open(DAT3, $input_file3) || print "Could not open count file!";
@raw_data3=<DAT3>;
close(DAT3);

Instead of reading the whole file at once, process it line by line. This will save you a huge amount of memory and time (since the OS won't have to allocate that memory)

Code:
($VAR,$Name,$S1,$F1,$S2,$F2,$S3,$F3,$S4,$F4,$S5,$F5,$S6,$F6,$S7,$F7,$S8,$F8,$S9,$F9,$S10,$F10,$S11,$ F11,$S12,$F12,$S13,$F13,$S14,
$F14,$S15,$F15,$S16,$F16,$S17,$F17,$S18,$F18,$S19,$F19,$S20,$F20,$S21,$F21,$S22,$F22,$S23,$F23,$S24, $F24)=split(/,/,$line);

Why don't you just split into an array? That way your code would still work if you ever need more fields, without needing a rewrite.

Code:
$suc = $S1 + $S2 + $S3 + $S4 + $S5 + $S6 + $S7 + $S8 + $S9 + $S10 +$S11 + $S12 + $S13 + $S14 + $S15 + $S16 + $S17 + $S18 + $S19
+ $S20 + $S21 + $S22 + $S23 + $S24;

$fail = $F1 + $F2 + $F3 + $F4 + $F5 +$F6 + $F7 + $F8 + $F9 + $F10 + $F11 + $F12 + $F13 + $F14 + $F15 + $F16 + $F17 + $F18 +$F19 +
$F20 + $F21 + $F22 +$F23 +$F24;

See above, with an array those could be reduced to two for loops (for maintainability)

Code:
`echo "$CC,$Name,$suc,$fail" >> $tmp_path/suc_and_fail`

This way, Perl has to create a shell process which runs echo, has to open the file for appending, and close it again. If you open the file inside Perl before you start processing, write directly to it, and close it afterwards you'll probably shave off even more seconds.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sum

Hello everyone I need to write a script that sums numbers passed to it as arguments on the command line and displays the results. I must use a for loop and then rewrite it using a while loop. It would have to output something like 10+20+30=60 this is what I have so far fafountain@hfc:~$ vi sum... (1 Reply)
Discussion started by: Blinky85
1 Replies

2. Shell Programming and Scripting

Sum value from selected lines script (awk,perl)

Hello. I face this (2 side) problem. Some lines with this structure. ........... 12345678 4 12345989 13 12346356 205 12346644 74 12346819 22 ......... The first field (timestamp) is growing (or at least equal). 1)Sum the second fields if the first_field/500 are... (8 Replies)
Discussion started by: paolfili
8 Replies

3. Shell Programming and Scripting

Perl script to find particular field and sum it

Hi, I have a file with format a b c d e 1 1 2 2 2 1 2 2 2 3 1 1 1 1 2 1 1 1 1 4 1 1 1 1 6 in column e i want to find all similar fields ( with perl script )and sum it how many are there for instance in format above. 2 - 2 times 4 - 1 time 6 - 1 time what i use is ... (14 Replies)
Discussion started by: Learnerabc
14 Replies

4. Shell Programming and Scripting

Print sum and relative value of the sum

Hi i data looks like this: student 1 Subject1 45 55 Subject2 44 55 Subject3 33 44 // student 2 Subject1 45 55 Subject2 44 55 Subject3 33 44 i would like to sum $2, $3 (marks) and divide each entry in $2 and $3 with their respective sums and print for each student as $4 and... (2 Replies)
Discussion started by: saint2006
2 Replies

5. Shell Programming and Scripting

perl sum 2nd field in an array

Hi Everyone, ($total+=$_) for @record; assume @record=(1,2,3), so the result is 6. if @record=("1 3","2 3","3 3"), would like to sum up the 2nd field of this array, the result is 9. i tried " ($total+=$) for @record ", cannot, please advice. Thanks ---------- Post updated at 03:45... (1 Reply)
Discussion started by: jimmy_y
1 Replies

6. Shell Programming and Scripting

PERL : Group & Sum in hash

Hi, I have a hash which is to be populated by reading data lines from a flat file. I am supposed to read fields 1-5 from the file and load them on to the hash such that fields 1-4 are going to be the hash key-set and field 5 is the hash value. Field 5 is a monetary amount and is supposed... (1 Reply)
Discussion started by: sinpeak
1 Replies

7. Solaris

How to Sum

Hi I need to incorporate a 'sum' as follows into a script and not sure how. I have a variable per line and I need them to be summed, e.g below 1 23 1,456 1 1 34 46 How do I calculate the sum of all these numbers to ouptut the answer ( 1,562) Thanks in advance (3 Replies)
Discussion started by: rob171171
3 Replies

8. UNIX for Dummies Questions & Answers

Getting the sum

I am trying to get the sum of the first column of a file. When I use the same method for other files it works just fine... for some reason for the file below it gives me an error that I don't understand... I tried looking at different lines of the file and tried different things, but I still... (7 Replies)
Discussion started by: cosmologist
7 Replies

9. Shell Programming and Scripting

Incredibly inefficient cat | grep script

Hi there, I have 2 files that I am trying to work on. File 1 contains a reference list of unique subscriber numbers ( 7 million entries in total) File 2 contains a list of the subscriber numbers and their tariff (15 million entries in total). This file is in the production system and... (12 Replies)
Discussion started by: Cludgie
12 Replies

10. Programming

PERL: In a perl-scripttTrying to execute another perl-script that SETS SOME VARIABLES !

I have reviewed many examples on-line about running another process (either PERL or shell command or a program), but do not find any usefull for my needs way. (Reviewed and not useful the system(), 'back ticks', exec() and open()) I would like to run another PERL-script from first one, not... (1 Reply)
Discussion started by: alex_5161
1 Replies
Perl::Critic::Policy::InputOutput::RequireBriefOpen(3pm)User Contributed Perl DocumentatioPerl::Critic::Policy::InputOutput::RequireBriefOpen(3pm)

NAME
Perl::Critic::Policy::InputOutput::RequireBriefOpen - Close filehandles as soon as possible after opening them. AFFILIATION
This Policy is part of the core Perl::Critic distribution. DESCRIPTION
One way that production systems fail unexpectedly is by running out of filehandles. Filehandles are a finite resource on every operating system that I'm aware of, and running out of them is virtually impossible to recover from. The solution is to not run out in the first place. What causes programs to run out of filehandles? Usually, it's leaks: you open a filehandle and forget to close it, or just wait a really long time before closing it. This problem is rarely exposed by test systems, because the tests rarely run long enough or have enough load to hit the filehandle limit. So, the best way to avoid the problem is 1) always close all filehandles that you open and 2) close them as soon as is practical. This policy takes note of calls to "open()" where there is no matching "close()" call within "N" lines of code. If you really need to do a lot of processing on an open filehandle, then you can move that processing to another method like this: sub process_data_file { my ($self, $filename) = @_; open my $fh, '<', $filename or croak 'Failed to read datafile ' . $filename . '; ' . $OS_ERROR; $self->_parse_input_data($fh); close $fh; return; } sub _parse_input_data { my ($self, $fh) = @_; while (my $line = <$fh>) { ... } return; } As a special case, this policy also allows code to return the filehandle after the "open" instead of closing it. Just like the close, however, that "return" has to be within the right number of lines. From there, you're on your own to figure out whether the code is promptly closing the filehandle. The STDIN, STDOUT, and STDERR handles are exempt from this policy. CONFIGURATION
This policy allows "close()" invocations to be up to "N" lines after their corresponding "open()" calls, where "N" defaults to 9. You can override this to set it to a different number with the "lines" setting. To do this, put entries in a .perlcriticrc file like this: [InputOutput::RequireBriefOpen] lines = 5 CAVEATS
"IO::File->new" This policy only looks for explicit "open" calls. It does not detect calls to "CORE::open" or "IO::File->new" or the like. Is it the right lexical? We don't currently check for redeclared filehandles. So the following code is false negative, for example, because the outer scoped filehandle is not closed: open my $fh, '<', $file1 or croak; if (open my $fh, '<', $file2) { print <$fh>; close $fh; } This is a contrived example, but it isn't uncommon for people to use $fh for the name of the filehandle every time. Perhaps it's time to think of better variable names... CREDITS
Initial development of this policy was supported by a grant from the Perl Foundation. AUTHOR
Chris Dolan <cdolan@cpan.org> COPYRIGHT
Copyright (c) 2007-2011 Chris Dolan. Many rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. The full text of this license can be found in the LICENSE file included with this module perl v5.14.2 2012-06-07 Perl::Critic::Policy::InputOutput::RequireBriefOpen(3pm)
All times are GMT -4. The time now is 09:57 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy