Speed up this script! Post: 302350112

Sponsored Content

Top Forums Shell Programming and Scripting Speed up this script! Post 302350112 by CRGreathouse on Wednesday 2nd of September 2009 09:05:42 PM

09-02-2009

Registered User

Speed up this script!

I have a script that processes a fair amount of data -- say, 25-50 megs per run. I'd like ideas on speeding it up. The code is actually just a preprocessor -- I'm using another language to do the heavy lifting. But as it happens, the preprocessing takes much more time than the final processing so I'm optimizing this rather than that.

Here's the code. The basic idea is that, for each line of input (redirected to stdin), the program checks to see if the sequence number is in $mult and, if so, prints a line asking the other program to validate that sequence:

Code:

#!/usr/bin/perl -w

open(MULT, "mult.txt") or die("Can't find list of multiplicative sequences in mult.txt");
my $terminator = $/;
undef $/;
$mult = <MULT>;
$/ = $terminator;

// Print application-specific code -- snipped for brevity

$total = 0;
while(<>) {
	if (m/(A\d\d\d\d\d\d) ,((-?\d+,)*-?\d+),/) {
		$nm = $1;
		$seq = $2;
		if ($mult =~ /$nm/) { # Replace this line?
			print "go(\"$nm\", [$seq]);\n";
			$total++;
		}
	} else {
		print "print(\"Error reading line: $_\");\n";
	}
}

// Print application-specific code -- snipped for brevity

The file mult.txt is a short file of about a thousand lines, each of which is guaranteed to contain at most (exactly?) one line of the form A\d\d\d\d\d\d; the rest of the line is irrelevant here.

My thought for optimizing this: make an array of the \d\d\d\d\d\d values, sort, and do a binary search rather than a regular expression at the spot marked "Replace this line?". But I'm not sure how to go about that, or even if that's the 'right' optimization. Thoughts?

Also, any suggestions on making better idiomatic use of Perl would be appreciated. I'm not at all accustomed to the language.

CRGreathouse

View Public Profile for CRGreathouse

Find all posts by CRGreathouse

10 More Discussions You Might Find Interesting

1. Filesystems, Disks and Memory

dmidecode, RAM speed = "Current Speed: Unknown"

Hello, I have a Supermicro server with a P4SCI mother board running Debian Sarge 3.1. This is the "dmidecode" output related to RAM info: RAM speed information is incomplete.. "Current Speed: Unknown", is there anyway/soft to get the speed of installed RAM modules? thanks!! Regards :)...

2. Shell Programming and Scripting

any way to speed up calculations in bash script

hi i have a script that is taking the difference of multiple columns in a file from a value from a single row..so far i have a loop to do that.. all the data is floating point..fin has the difference between array1 and array2..array1 has 700 x 300= 210000 values and array2 has 700 values.. ...

3. Shell Programming and Scripting

Help to improve speed of text processing script

Hey together, You should know, that I'am relatively new to shell scripting, so my solution is probably a little awkward. Here is the script: #!/bin/bash live_dir=/var/lib/pokerhands/live for limit in `find $live_dir/ -type d | sed -e s#$live_dir/##`; do cat $live_dir/$limit/*...

4. Filesystems, Disks and Memory

data from blktrace: read speed V.S. write speed

I analysed disk performance with blktrace and get some data: read: 8,3 4 2141 2.882115217 3342 Q R 195732187 + 32 8,3 4 2142 2.882116411 3342 G R 195732187 + 32 8,3 4 2144 2.882117647 3342 I R 195732187 + 32 8,3 4 2145 ...

5. Shell Programming and Scripting

Any trick to speed up script?

Hi Guys, I have a script that I am using to convert some text files to xls files. I create multiple temp. files in the process of conversion. Other than reducing the temp. files, are there any general tricks to help speed up the script? I am running it in the bash shell. Thanks.

6. Shell Programming and Scripting

Slow Perl script: how to speed up?

I had written a perl script to compare two files: new and master and get the output of the first file i.e. the first file: words that are not in the master file STRUCTURE OF THE TWO FILES The first file is a series of names ramesh sushil jonga sudesh lugdi whereas the second file (could be...

7. Shell Programming and Scripting

How can i speed this script up?

Hi, Im quite new to scripting and would like a bit of assistance with trying to speed up the following script. At the moment it is quite slow.... Any way to improve it? total=111120 while do total=`expr $total + 1` INCREMENT=$total firstline = "blablabla" secondline = "blablabla"...

8. Shell Programming and Scripting

Help me with speed up this script

hey guys i have a perl script wich use to compare hashes but it tookes a long time to do that so i wich i will have the soulition to do it soo fast he is the code <redacted>

9. Shell Programming and Scripting

Speed up the loop in shell script

Hi I have written a shell script which will test 300 to 500 IPs to find which are pinging and which are not pinging. the script which give output as 10.x.x.x is pining 10.x.x.x. is not pining - - - 10.x.x.x is pining like above. But, this script is taking...

10. Shell Programming and Scripting

Need to Speed up shell script

Hello, I am basic level shell script developer. I have developed the following script. The shell script basically tracking various files containing certain strings. I am finding options to make the script run more faster. Any help/suggestion would be appreciated :) #! /bin/bash # Greps for...

LEARN ABOUT DEBIAN

service

service(8)						      System Manager's Manual							service(8)

NAME

       service - run a System V init script

SYNOPSIS

       service SCRIPT COMMAND [OPTIONS]

       service --status-all

       service --help | -h | --version

DESCRIPTION

       service	runs a System V init script in as predictable an environment as possible, removing most environment variables and with the current
       working directory set to /.

       The SCRIPT parameter specifies a System V init script, located in /etc/init.d/SCRIPT.  The  supported  values  of  COMMAND  depend  on  the
       invoked	script.  service passes COMMAND and OPTIONS to the init script unmodified.  All scripts should support at least the start and stop
       commands.  As a special case, if COMMAND is --full-restart, the script is run twice, first with the stop command, then with the start  com-
       mand.

       service --status-all runs all init scripts, in alphabetical order, with the status command.

EXIT CODES

       service calls the init script and returns the status returned by it.

FILES

       /etc/init.d
	      The directory containing System V init scripts.

ENVIRONMENT

       LANG, TERM
	      The only environment variables passed to the init scripts.

SEE ALSO

       /etc/init.d/skeleton,
       update-rc.d(8),
       init(8),
       invoke-rc.d(8).

								     Jan 2006								service(8)