Programming

So it takes a while. So what? Is there any reason you need the data faster?

You're probably costing your employer $100-200 an hour when everything is added up. If not more.

There really needs to be a good reason to spend lots of time simply speeding up something that's already working.

Code:

it takes some time to process millions of data

It depends on how "some time" is defined. Minutes? Hours? Days?

Well, the username, ribosome, suggests that the data may involve bioinformatics. If that's so, then very large datasets may be involved and a small improvement could yield significant monetary savings over its lifetime.

Regards,
Alister

---------- Post updated at 03:24 PM ---------- Previous update was at 03:04 PM ----------

By the way, Ribosome, if you would like some concrete advice, it's about time you posted a sample of the logs that you are processing, a description of how they ought to be processed, a sample of the processed output, the perl code that you are currently using to process them, and how that script is called. Only with that information in hand will a competent perl coder be able to make useful recommendations. If this is too much to reasonably accomodate in a forum post, attach it in a file (archived if necessary).

Given the paucity of details, it's possible that your bottleneck is an inefficient shell script wrapper. Who knows.

Regards,
Alister

Now we're getting somewhere. Rebuilding a log scanner in C sounds much more doable than rebuilding an entire application. Can you show this script?

Also, tell us the size of the dataset. How large are the files (file size in bytes) and how many are there?

Regards.
Alister

Your own code and their run on a different platform:

Code:

$ ll
total 31856
-rw-r--r--  1 pkumarpr  ipg       144 Apr 21 15:27 sha1_.pl
-rw-r--r--  1 pkumarpr  ipg  32544556 Apr 21 15:36 usr.bin

Code:

$ cat sha1.pl
use Digest::SHA1;
binmode(STDIN);
my $s = Digest::SHA1->new;
while (read(STDIN, $buf, 1024)) {
    $s->add($buf);
}
print $s->hexdigest . "\n"

Code:

$ time perl sha1_.pl < usr.bin
bad603823684e49866e5c744d560992d34743338

real    0m1.520s
user    0m0.368s
sys     0m0.196s

$ time sha1 < usr.bin
bad603823684e49866e5c744d560992d34743338

real    0m0.181s
user    0m0.128s
sys     0m0.053s

The above run itself is around 8X performance difference in performance with standard sha1sum utility.



The above was run on a FreeBSD7.1 machine; -a very busy build machine, we have in lab!!!

One should not conclude for the overall scenario with few runs on simple tests performed on a type of H/w & one platform only; because this might differ drastically in other environment especially a controlled ones like appliances/devices;
wherein something might be configured to run with 100% CPU utilization and with hi-priority (in multi-core/CPU environment). I think you should have considered this too!!!


Now about 440x performance: You calculated it with the data I described, great!!! You still don't know the environment my c-program ran and I got the kind of performance difference.


Fact:

IBM ISS Proventia GX-series IPS devices, under FIPS mode, run a boot-time integrity check (with SHA1 hash), of the entire disk, in just around 3 seconds (around 5 seconds on low end the models).

The devices are in public domain and you should have a look at it.

Under similar conditions that Perl script based utility had a 22-mintues of run!!!

Praveen_218:

Respectfully, whatever hardware you or the OP may be using is of no concern to me. I have not made any assertions whatsoever, and have no predictions in need of substantiation.

The point of my previous post is simply that your expectation that if the OP switches implementation language they will see a performance improvement similar to that which you described (440x) is without merit. I stand by that.

Even if that statement were to be proven correct, it would still have been meritless. There is simply insufficient information to make such a prediction.

Regards,
Alister