Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Slow performance filtering file
Top Forums Shell Programming and Scripting Slow performance filtering file Post 302495680 by markhahn on Friday 11th of February 2011 12:25:45 AM
Old 02-11-2011
use the right tool
I have to admit I'm pretty turned off by this kind of shell script - spawning way too many trivial processes to perform minor processing. for instance, in perl:

Code:
#!/usr/bin/perl
use strict;
while (<>) {
    my ($date,$time,$rest) = /^(.{10}) (.{8}),(.+)$/;
    $rest = $1 if ($rest =~ /^"(.+)"$/);
    my ($prefix,$num,$type,$message) = ($rest =~ /^(...)(....)(.) (.+)$/);
    print "$date,$time,$prefix,$num,$type,$message\n" if ($type eq 'E');
}

could be down to ~3 lines with perl -n and integrating the 'E' test into the pattern match.

to me, shell is ok for command-y stuff (not processing). I turn to awk mainly for quick processing of simple, delimited tables on the commandline. for anything more than trivial (real parsing, any math), I use perl.
 

8 More Discussions You Might Find Interesting

1. Post Here to Contact Site Administrators and Moderators

Help! Slow Performance

Is the performance now very, very slow (pages take a very long time to load)? Or is it just me? Neo (6 Replies)
Discussion started by: Neo
6 Replies

2. Shell Programming and Scripting

egrep is very slow : How to improve performance

We have an egrep search in a while loop. egrep -w "$key" ${PICKUP_DIR}/new_update >> ${PICKUP_DIR}/update_record_new ${PICKUP_DIR}/new_update is 210 MB file In each iteration, the egrep on an average takes around 50-60 seconds to search. Ther'es nothing significant in the loop other... (7 Replies)
Discussion started by: hidnana
7 Replies

3. Filesystems, Disks and Memory

Slow Copy(CP) performance

Hi all We have got issues with copying a 2.6 GB file from one folder to another folder. Well, this is not the first issue we are having on the box currently, i will try to explain everything we have done from the past 2 days. We got a message 2 days back saying that our Production is 98%... (3 Replies)
Discussion started by: b_sri
3 Replies

4. UNIX for Dummies Questions & Answers

Slow copy/performance... between volumes

hi guys We are seeing weird issues on my Linux Suse 10, it has lotus 8.5 and 1 filesystem for OS and another for Lotus Database. the issue is when the Lotus service starts wait on top is very high about 25% percent and in general CPU usage is very high we found that when this happens if we... (0 Replies)
Discussion started by: kopper
0 Replies

5. Infrastructure Monitoring

99% performance wa, slow server.

There is a big problem with the server (VPS based on OpenVZ, CentOS 5, 3GB RAM). The problem is the following. The first 15-20 minutes after starting the server is operating normally, the load average is less than or about 1.0, but then begins to increase sharply% wa, then hovers around 95-99%.... (2 Replies)
Discussion started by: draiphod
2 Replies

6. Red Hat

GFS file system performance is very slow

My code Hi All, I am having redhat linux 5.3 (Tikanga) with GFS file system and its very very slow for executing ls -ls command also.Please see the below for 2minits 12 second takes. Please help me to fix the issue. $ sudo time ls -la BadFiles |wc -l 0.01user 0.26system... (3 Replies)
Discussion started by: susindram
3 Replies

7. Solaris

Solaris 11.1 Slow Network Performance

I have identical M5000 machines that are needing to transfer very large amounts of data between them. These are fully loaded machines, and I've already checked IO, memory usage, etc... I get poor network performance even when the machines are idle or copying via loopback. The 10 GB NICs are... (7 Replies)
Discussion started by: christr
7 Replies

8. Filesystems, Disks and Memory

Slow copy (cp) performance when overwriting files

I have a lot of binary files I need to copy to a folder. The folder is already filled with files of the same name. Copying on top of the old files takes MUCH longer than if I were to delete the old files then copy the new files to the now-empty folder. This result is specific to one system -... (3 Replies)
Discussion started by: ces55
3 Replies

LEARN ABOUT MOJAVE

moosex::getopt::basic5.18

MooseX::Getopt::Basic(3pm)				User Contributed Perl Documentation				MooseX::Getopt::Basic(3pm)

NAME

       MooseX::Getopt::Basic - MooseX::Getopt::Basic - role to implement the Getopt::Long functionality

VERSION

       version 0.59

SYNOPSIS

	 ## In your class
	 package My::App;
	 use Moose;

	 with 'MooseX::Getopt::Basic';

	 has 'out' => (is => 'rw', isa => 'Str', required => 1);
	 has 'in'  => (is => 'rw', isa => 'Str', required => 1);

	 # ... rest of the class here

	 ## in your script
	 #!/usr/bin/perl

	 use My::App;

	 my $app = My::App->new_with_options();
	 # ... rest of the script here

	 ## on the command line
	 % perl my_app_script.pl --in file.input --out file.dump

DESCRIPTION

       This is like MooseX::Getopt and can be used instead except that it doesn't make use of Getopt::Long::Descriptive (or "GLD" for short).

METHODS

   new_with_options
       See "new_with_options" in MooseX::Getopt.

   process_argv
       See "process_argv" in MooseX::Getopt.

AUTHOR

       Stevan Little <stevan@iinteractive.com>

COPYRIGHT AND LICENSE

       This software is copyright (c) 2007 by Infinity Interactive, Inc.

       This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

perl v5.18.2							    2013-11-30						MooseX::Getopt::Basic(3pm)
All times are GMT -4. The time now is 07:14 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy