Programming

The other point is that Perl is written in C/C++ too... If the Perl code spends most of its time using external modules, or just doing I/O, its performance may be quite comparable to a pure C++ program. If it spends most of its time doing complex data processing via perl statements, perhaps not so much.

So there's one easy benchmark for you. If this Perl program uses lots and lots of CPU power while running, it may benefit from optimization. If it doesn't, it's spending most of its time waiting for data to arrive or be sent... A C++ version wouldn't be able to wait much faster

You know what -There was a Perl script I had got from CPAN to generate SHA1 checksum of files in a directory, I modified to suite my needs, for the project I was working at, and the script was taking around 21-22 minutes just to generate SHA1 sum of the required files across the system disk (which was most of the files of the disk except a list of certain category of files filtered by it's extension) and later would verify at boot time.

I displayed the same to my boss, the moment I conveyed him that I started out modification of a public domain script from CPAN; he didn't allow me to use the same for a commercial project, although he wanted originally a script kind of thing only; I was made to write a C version of the same again -that too giving me just 48 hrs only, as I had already eaten up much of the scheduled time with my R & D and modifications job with the Perl script (even learning Perl too ).

I somehow managed myself to come out of the soup by really creating a C program from scratch.

The pleasant part was that the job which took 22 minutes (approx) was taking only around 3 seconds.

This took away all the pain I faced in the last 48 hrs while I wrote the C version of the program.

No one expected this much performance improvement (even myself & my boss) but that was it!!!

I really do expect your C++ program, if you really create, will get you to see the same kind of performance improvements.

The porting issues and all with your C++ program would be immaterial if your project has got a build system which create images for different supported platforms.

Please do let us know here, if you chose to create the C/C++ version of the same? And how was the performance improvement?

---------- Post updated at 03:20 PM ---------- Previous update was at 03:16 PM ----------

I'd be eager to know your experience too!!!

Happy coding!! :-)

---------- Post updated at 03:31 PM ---------- Previous update was at 03:20 PM ----------

That's absolutely true!!

However a CPU intensive program implemented into C/C++ (even with exactly the same algorithm -however bad that might be) runs not just 2x or 3x time faster but much much more than that of a Perl script (of course the bottleneck is on the percentage of I/O wait, in the overall run time, that you have mentioned).

Hey Thank you all for enlightening me!

I've not worked much on perl, but in every C++ project I've worked on I've seen the reporting scripts were done especially in PERL and thats why I was confused.
The task which needs to be done here is to read some log files, extract the data such as folder id, number of files it contains, processing time, no. of successfully processed files and no. of failed files etc and put them in an excel file.
The script is good and doesnt seem to need any fix but it takes some time to process millions of data. So the only concern here is to get this thing done in any other language for optimization.

I do not intend any personal offense, but I am compelled to say that it is absurd to tell someone that they will realize a 440x runtime improvement without any knowledge of their task, their code, or their hardware.

If the code isn't CPU-bound, the performance gain will be negligible.

Your example, by the way, of cryptographic hashing, is just the type of task that does benefit greatly from using a lower-level language. However, switching to C won't yield the massive improvements you describe because the folks who implement perl's widely-used crypto modules are already using C.

To demonstrate, on an ancient Pentium II system running at 350 MHz, let's hash all the executables in my /usr/bin directory.

Version of sha1sum (a C program):

Code:

$ sha1sum --version | head -n1
sha1sum (GNU coreutils) 5.97

Perl script:

Code:

$ cat sha1.pl
use Digest::SHA1;
binmode(STDIN);
my $s = Digest::SHA1->new;
while (read(STDIN, $buf, 1024)) {
    $s->add($buf);
}
print $s->hexdigest . "\n"

Creating data file:

Code:

$ for f in /usr/bin/*; do cat "$f"; done > usr.bin 2> /dev/null
$ stat --printf '%s\n' usr.bin  # ~ 181 MB
189537836

Comparison:

Code:

$ time sha1sum < usr.bin
bb02b0a0c08a1b4070f97035f0ad5f7f2c491c78  -

real    0m13.622s
user    0m10.217s
sys     0m1.868s
$ time perl sha1.pl < usr.bin
bb02b0a0c08a1b4070f97035f0ad5f7f2c491c78

real    0m17.131s
user    0m12.733s
sys     0m1.604s

The tests were repeated several times in different orders. The times never varied significantly. 1.25x faster is a far, far cry from 440x. The small improvement is expected since both are hashing in C.

It is reasonable to conclude that whatever perl script you were using (or the modifications made to it) was seriously deficient.

Regards,
Alister

---------- Post updated at 12:43 PM ---------- Previous update was at 12:34 PM ----------

For i/o and string comparisons and the like, the improvement ratio (perl-time/c-time) will be small. However, if the dataset is massive, that ratio could amount to a substantial speedup. The 1.25x improvement in the sha1 hashing in my previous post isn't much for 181 MB of data, 3.5 secs or so, but it's about 20 seconds per gigabyte and 5.5 hours per terabyte.

Regards,
Alister

Implementing crypto in raw perl is absolutely ludicrous, it's like trying to accomplish serious work in intercal... It's just not made for that kind of work. It's bound to have a module to do that natively instead, which they should have used.

Or perhaps they did use it but badly -- reading and processing single characters at time, or consuming huge gobs of ram to slurp in entire huge files... Perl lends itself to certain abuses.

In any case, it must be possible to write faster perl code for even that task. Perl is no excuse for a poor algorithm.

It's been done, but it is only intended for situations where a C compiler is unavailable: Digest::SHA::PurePerl

Regards,
Alister

I didn't say it hadn't, only that it was ludicrous to do so.

People have also done things like entire X servers in native Perl. It baffles.