I've not worked much on perl, but in every C++ project I've worked on I've seen the reporting scripts were done especially in PERL and thats why I was confused.
The task which needs to be done here is to read some log files, extract the data such as folder id, number of files it contains, processing time, no. of successfully processed files and no. of failed files etc and put them in an excel file.
The script is good and doesnt seem to need any fix but it takes some time to process millions of data. So the only concern here is to get this thing done in any other language for optimization.
So it takes a while. So what? Is there any reason you need the data faster?
You're probably costing your employer $100-200 an hour when everything is added up. If not more.
There really needs to be a good reason to spend lots of time simply speeding up something that's already working.
So it takes a while. So what? Is there any reason you need the data faster?
Well, the username, ribosome, suggests that the data may involve bioinformatics. If that's so, then very large datasets may be involved and a small improvement could yield significant monetary savings over its lifetime.
Regards,
Alister
---------- Post updated at 03:24 PM ---------- Previous update was at 03:04 PM ----------
By the way, Ribosome, if you would like some concrete advice, it's about time you posted a sample of the logs that you are processing, a description of how they ought to be processed, a sample of the processed output, the perl code that you are currently using to process them, and how that script is called. Only with that information in hand will a competent perl coder be able to make useful recommendations. If this is too much to reasonably accomodate in a forum post, attach it in a file (archived if necessary).
Given the paucity of details, it's possible that your bottleneck is an inefficient shell script wrapper. Who knows.
The script is good and doesnt seem to need any fix but it takes some time to process millions of data. So the only concern here is to get this thing done in any other language for optimization.
Now we're getting somewhere. Rebuilding a log scanner in C sounds much more doable than rebuilding an entire application. Can you show this script?
Your own code and their run on a different platform:
Quote:
Originally Posted by alister
...
The tests were repeated several times in different orders. The times never varied significantly. 1.25x faster is a far, far cry from 440x.
...
The above run itself is around 8X performance difference in performance with standard sha1sum utility.
Quote:
Originally Posted by alister
...
... I am compelled to say that it is absurd to tell someone that they will realize a 440x run time improvement without any knowledge of their task, their code, or their hardware.
...
The above was run on a FreeBSD7.1 machine; -a very busy build machine, we have in lab!!!
One should not conclude for the overall scenario with few runs on simple tests performed on a type of H/w & one platform only; because this might differ drastically in other environment especially a controlled ones like appliances/devices;
wherein something might be configured to run with 100% CPU utilization and with hi-priority (in multi-core/CPU environment). I think you should have considered this too!!!
Now about 440x performance: You calculated it with the data I described, great!!! You still don't know the environment my c-program ran and I got the kind of performance difference.
Fact:
IBM ISS Proventia GX-series IPS devices, under FIPS mode, run a boot-time integrity check (with SHA1 hash), of the entire disk, in just around 3 seconds (around 5 seconds on low end the models).
The devices are in public domain and you should have a look at it.
Under similar conditions that Perl script based utility had a 22-mintues of run!!!
Last edited by Praveen_218; 04-21-2013 at 03:36 PM..
Respectfully, whatever hardware you or the OP may be using is of no concern to me. I have not made any assertions whatsoever, and have no predictions in need of substantiation.
The point of my previous post is simply that your expectation that if the OP switches implementation language they will see a performance improvement similar to that which you described (440x) is without merit. I stand by that.
Quote:
Originally Posted by Praveen_218
I really do expect your C++ program, if you really create, will get you to see the same kind of performance improvements.
Even if that statement were to be proven correct, it would still have been meritless. There is simply insufficient information to make such a prediction.
Hello Folks
I am writing this simple program but I am stuck at this point. Here is the snippet from my script where I have issues.
3) echo "Current Directory: $(pwd) Menu 3"
echo -e "Enter a file name\n"
read fname
if
then
... (5 Replies)
HI,
I am using the windows 2003 server R2 in there we are using the putty as to access the different AP's
now from the primary AP i want to login to several different AP's using a script
what the script will do is :-
input a text file in which list of different ap's and the corresponding... (0 Replies)
Hi
I'm quite new with linux.
Very simple, I need to swap every 2 lines in a file.
Example
INPUT:
a a a
b b b
x x x
y y y
s s s
t t t
OUTPUT:
b b b
a a a
y y y
x x x
t t t (5 Replies)
greetings,
i hope this hasn't been covered previously. has anyone heard of a .kext or daemon that would allow linux or (open)solaris-like vt switching? googling didn't help much..
i know os x allows a '>console' login from loginwindow.app, but i'm mainly interested in this because there are... (0 Replies)
Hi
I want to write a script which can switch between super users.But it asks for the password at the prompt.How can I manage in the script so that it didnt ask me for the password at the prompt. (1 Reply)
Hi all,
we have running some linux servers with sles9 and we have some problems with our dns servers. Sometimes they don't like to work. However, is there a parameter to enable faster switching between two ore more dns servers?
Thx for your help in front
Regards
frank (5 Replies)
Can any one tell me :
How we can switch between two users without prompting for the password. (In the SHELL SCRIPT can we fetch the USERID and PASSWORD from a specified file, without using SUDO command)? (2 Replies)
Hi,
what is the use of the double quotes and !! in the following code segment:
su - user1 << ""!! > /dev/null 2>&1
echo "welcome user1"
EOF
!!
also what is the difference between below:
su - user1 << ""!! > /dev/null 2>&1 and
su - $USER << ""!!!> /dev/null 2>&1.
Note: $USER =... (2 Replies)
Hi,
what is the use of the double quotes and !! in the following code segment:
su - user1 << ""!! > /dev/null 2>&1
echo "welcome user1"
EOF
!!
also what is the difference between below:
su - user1 << ""!! > /dev/null 2>&1 and
su - $USER << ""!!!> /dev/null 2>&1.
Note: $USER =... (1 Reply)
Hi folks,
could anyone please tell me how can i switch between two users without going thru the su(i.e. root)?
is there any such command?
thanks in advance,
thell (1 Reply)