Sponsored Content
Top Forums Shell Programming and Scripting How to make awk command faster for large amount of data? Post 303024262 by disedorgue on Thursday 4th of October 2018 08:45:17 PM
Old 10-04-2018
Hi, here a perl script that output first and last line, maybe most faster but i'm not sure...
Code:
#!/usr/bin/env perl

use strict;
use warnings;
use IO::Uncompress::Gunzip qw(gunzip $GunzipError);

my $line;
for my $input ( glob "./*.gz" )
{
    print $input."\n";
    my $gz = new IO::Uncompress::Gunzip $input;
    my $mm=$gz->getHeaderInfo();
    print $gz->getline();
    $gz->seek(%$mm{"ISIZE"}-80,0); # here, max size of line is 80
    $line=$gz->getline() while(!eof($gz));
    print $line;
    close($gz)
}

Example:
Code:
$ perl unco.pl 
./bbrr.gz
"";"Gender";"FSIQ";"VIQ";"PIQ";"Weight";"Height";"MRI_Count"
"40";"Male";89;91;89;"179";"75.5";935863
./brain_size.csv.gz
"";"Gender";"FSIQ";"VIQ";"PIQ";"Weight";"Height";"MRI_Count"
"40";"Male";89;91;89;"179";"75.5";935863

Regards.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk help to make my work faster

hii everyone , i have a file in which i have line numbers.. file name is file1.txt aa bb cc "12" qw xx yy zz "23" we bb qw we "123249" jh here 12,23,123249. is the line number now according to this line numbers we have to print lines from other file named... (11 Replies)
Discussion started by: kumar_amit
11 Replies

2. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies

3. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

4. Shell Programming and Scripting

How to tar large amount of files?

Hello I have the following files VOICE_hhhh SUBSCR_llll DEL_kkkk Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files When i try to tar these files using tar -cvf backup.tar VOICE* SUBSCR* DEL* i get the error: ksh: /usr/bin/tar: arg list too long How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies

5. Emergency UNIX and Linux Support

Help to make awk script more efficient for large files

Hello, Error awk: Internal software error in the tostring function on TS1101?05044400?.0085498227?0?.0011041461?.0034752266?.00397045?0?0?0?0?0?0?11/02/10?09/23/10???10?no??0??no?sct_det3_10_20110516_143936.txt What it is It is a unix shell script that contains an awk program as well as... (4 Replies)
Discussion started by: script_op2a
4 Replies

6. Shell Programming and Scripting

Running rename command on large files and make it faster

Hi All, I have some 80,000 files in a directory which I need to rename. Below is the command which I am currently running and it seems, it is taking fore ever to run this command. This command seems too slow. Is there any way to speed up the command. I have have GNU Parallel installed on my... (6 Replies)
Discussion started by: shoaibjameel123
6 Replies

7. Shell Programming and Scripting

Faster way to use this awk command

awk "/May 23, 2012 /,0" /var/tmp/datafile the above command pulls out information in the datafile. the information it pulls is from the date specified to the end of the file. now, how can i make this faster if the datafile is huge? even if it wasn't huge, i feel there's a better/faster way to... (8 Replies)
Discussion started by: SkySmart
8 Replies

8. Shell Programming and Scripting

awk changes to make it faster

I have script like below, who is picking number from one file and and searching in another file, and printing output. Bu is is very slow to be run on huge file.can we modify it with awk #! /bin/ksh while read line1 do echo "$line1" a=`echo $line1` if then echo "$num" cat file1|nawk... (6 Replies)
Discussion started by: mirwasim
6 Replies

9. Shell Programming and Scripting

Perl : Large amount of data put into an array

This basic code works. I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in". It... (5 Replies)
Discussion started by: sumguy
5 Replies

10. Shell Programming and Scripting

How to make awk command faster?

I have the below command which is referring a large file and it is taking 3 hours to run. Can something be done to make this command faster. awk -F ',' '{OFS=","}{ if ($13 == "9999") print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12 }' ${NLAP_TEMP}/hist1.out|sort -T ${NLAP_TEMP} |uniq>... (13 Replies)
Discussion started by: Peu Mukherjee
13 Replies
IO::Uncompress::Bunzip2(3pm)				 Perl Programmers Reference Guide			      IO::Uncompress::Bunzip2(3pm)

NAME
IO::Uncompress::Bunzip2 - Read bzip2 files/buffers SYNOPSIS
use IO::Uncompress::Bunzip2 qw(bunzip2 $Bunzip2Error) ; my $status = bunzip2 $input => $output [,OPTS] or die "bunzip2 failed: $Bunzip2Error "; my $z = new IO::Uncompress::Bunzip2 $input [OPTS] or die "bunzip2 failed: $Bunzip2Error "; $status = $z->read($buffer) $status = $z->read($buffer, $length) $status = $z->read($buffer, $length, $offset) $line = $z->getline() $char = $z->getc() $char = $z->ungetc() $char = $z->opened() $data = $z->trailingData() $status = $z->nextStream() $data = $z->getHeaderInfo() $z->tell() $z->seek($position, $whence) $z->binmode() $z->fileno() $z->eof() $z->close() $Bunzip2Error ; # IO::File mode <$z> read($z, $buffer); read($z, $buffer, $length); read($z, $buffer, $length, $offset); tell($z) seek($z, $position, $whence) binmode($z) fileno($z) eof($z) close($z) DESCRIPTION
This module provides a Perl interface that allows the reading of bzip2 files/buffers. For writing bzip2 files/buffers, see the companion module IO::Compress::Bzip2. Functional Interface A top-level function, "bunzip2", is provided to carry out "one-shot" uncompression between buffers and/or files. For finer control over the uncompression process, see the "OO Interface" section. use IO::Uncompress::Bunzip2 qw(bunzip2 $Bunzip2Error) ; bunzip2 $input => $output [,OPTS] or die "bunzip2 failed: $Bunzip2Error "; The functional interface needs Perl5.005 or better. bunzip2 $input => $output [, OPTS] "bunzip2" expects at least two parameters, $input and $output. The $input parameter The parameter, $input, is used to define the source of the compressed data. It can take one of the following forms: A filename If the $input parameter is a simple scalar, it is assumed to be a filename. This file will be opened for reading and the input data will be read from it. A filehandle If the $input parameter is a filehandle, the input data will be read from it. The string '-' can be used as an alias for standard input. A scalar reference If $input is a scalar reference, the input data will be read from $$input. An array reference If $input is an array reference, each element in the array must be a filename. The input data will be read from each file in turn. The complete array will be walked to ensure that it only contains valid filenames before any data is uncompressed. An Input FileGlob string If $input is a string that is delimited by the characters "<" and ">" "bunzip2" will assume that it is an input fileglob string. The input is the list of files that match the fileglob. If the fileglob does not match any files ... See File::GlobMapper for more details. If the $input parameter is any other type, "undef" will be returned. The $output parameter The parameter $output is used to control the destination of the uncompressed data. This parameter can take one of these forms. A filename If the $output parameter is a simple scalar, it is assumed to be a filename. This file will be opened for writing and the uncompressed data will be written to it. A filehandle If the $output parameter is a filehandle, the uncompressed data will be written to it. The string '-' can be used as an alias for standard output. A scalar reference If $output is a scalar reference, the uncompressed data will be stored in $$output. An Array Reference If $output is an array reference, the uncompressed data will be pushed onto the array. An Output FileGlob If $output is a string that is delimited by the characters "<" and ">" "bunzip2" will assume that it is an output fileglob string. The output is the list of files that match the fileglob. When $output is an fileglob string, $input must also be a fileglob string. Anything else is an error. If the $output parameter is any other type, "undef" will be returned. Notes When $input maps to multiple compressed files/buffers and $output is a single file/buffer, after uncompression $output will contain a concatenation of all the uncompressed data from each of the input files/buffers. Optional Parameters Unless specified below, the optional parameters for "bunzip2", "OPTS", are the same as those used with the OO interface defined in the "Constructor Options" section below. "AutoClose => 0|1" This option applies to any input or output data streams to "bunzip2" that are filehandles. If "AutoClose" is specified, and the value is true, it will result in all input and/or output filehandles being closed once "bunzip2" has completed. This parameter defaults to 0. "BinModeOut => 0|1" When writing to a file or filehandle, set "binmode" before writing to the file. Defaults to 0. "Append => 0|1" TODO "MultiStream => 0|1" If the input file/buffer contains multiple compressed data streams, this option will uncompress the whole lot as a single data stream. Defaults to 0. "TrailingData => $scalar" Returns the data, if any, that is present immediately after the compressed data stream once uncompression is complete. This option can be used when there is useful information immediately following the compressed data stream, and you don't know the length of the compressed data stream. If the input is a buffer, "trailingData" will return everything from the end of the compressed data stream to the end of the buffer. If the input is a filehandle, "trailingData" will return the data that is left in the filehandle input buffer once the end of the compressed data stream has been reached. You can then use the filehandle to read the rest of the input file. Don't bother using "trailingData" if the input is a filename. If you know the length of the compressed data stream before you start uncompressing, you can avoid having to use "trailingData" by setting the "InputLength" option. Examples To read the contents of the file "file1.txt.bz2" and write the compressed data to the file "file1.txt". use strict ; use warnings ; use IO::Uncompress::Bunzip2 qw(bunzip2 $Bunzip2Error) ; my $input = "file1.txt.bz2"; my $output = "file1.txt"; bunzip2 $input => $output or die "bunzip2 failed: $Bunzip2Error "; To read from an existing Perl filehandle, $input, and write the uncompressed data to a buffer, $buffer. use strict ; use warnings ; use IO::Uncompress::Bunzip2 qw(bunzip2 $Bunzip2Error) ; use IO::File ; my $input = new IO::File "<file1.txt.bz2" or die "Cannot open 'file1.txt.bz2': $! " ; my $buffer ; bunzip2 $input => $buffer or die "bunzip2 failed: $Bunzip2Error "; To uncompress all files in the directory "/my/home" that match "*.txt.bz2" and store the compressed data in the same directory use strict ; use warnings ; use IO::Uncompress::Bunzip2 qw(bunzip2 $Bunzip2Error) ; bunzip2 '</my/home/*.txt.bz2>' => '</my/home/#1.txt>' or die "bunzip2 failed: $Bunzip2Error "; and if you want to compress each file one at a time, this will do the trick use strict ; use warnings ; use IO::Uncompress::Bunzip2 qw(bunzip2 $Bunzip2Error) ; for my $input ( glob "/my/home/*.txt.bz2" ) { my $output = $input; $output =~ s/.bz2// ; bunzip2 $input => $output or die "Error compressing '$input': $Bunzip2Error "; } OO Interface Constructor The format of the constructor for IO::Uncompress::Bunzip2 is shown below my $z = new IO::Uncompress::Bunzip2 $input [OPTS] or die "IO::Uncompress::Bunzip2 failed: $Bunzip2Error "; Returns an "IO::Uncompress::Bunzip2" object on success and undef on failure. The variable $Bunzip2Error will contain an error message on failure. If you are running Perl 5.005 or better the object, $z, returned from IO::Uncompress::Bunzip2 can be used exactly like an IO::File filehandle. This means that all normal input file operations can be carried out with $z. For example, to read a line from a compressed file/buffer you can use either of these forms $line = $z->getline(); $line = <$z>; The mandatory parameter $input is used to determine the source of the compressed data. This parameter can take one of three forms. A filename If the $input parameter is a scalar, it is assumed to be a filename. This file will be opened for reading and the compressed data will be read from it. A filehandle If the $input parameter is a filehandle, the compressed data will be read from it. The string '-' can be used as an alias for standard input. A scalar reference If $input is a scalar reference, the compressed data will be read from $$output. Constructor Options The option names defined below are case insensitive and can be optionally prefixed by a '-'. So all of the following are valid -AutoClose -autoclose AUTOCLOSE autoclose OPTS is a combination of the following options: "AutoClose => 0|1" This option is only valid when the $input parameter is a filehandle. If specified, and the value is true, it will result in the file being closed once either the "close" method is called or the IO::Uncompress::Bunzip2 object is destroyed. This parameter defaults to 0. "MultiStream => 0|1" Allows multiple concatenated compressed streams to be treated as a single compressed stream. Decompression will stop once either the end of the file/buffer is reached, an error is encountered (premature eof, corrupt compressed data) or the end of a stream is not immediately followed by the start of another stream. This parameter defaults to 0. "Prime => $string" This option will uncompress the contents of $string before processing the input file/buffer. This option can be useful when the compressed data is embedded in another file/data structure and it is not possible to work out where the compressed data begins without having to read the first few bytes. If this is the case, the uncompression can be primed with these bytes using this option. "Transparent => 0|1" If this option is set and the input file/buffer is not compressed data, the module will allow reading of it anyway. In addition, if the input file/buffer does contain compressed data and there is non-compressed data immediately following it, setting this option will make this module treat the whole file/bufffer as a single data stream. This option defaults to 1. "BlockSize => $num" When reading the compressed input data, IO::Uncompress::Bunzip2 will read it in blocks of $num bytes. This option defaults to 4096. "InputLength => $size" When present this option will limit the number of compressed bytes read from the input file/buffer to $size. This option can be used in the situation where there is useful data directly after the compressed data stream and you know beforehand the exact length of the compressed data stream. This option is mostly used when reading from a filehandle, in which case the file pointer will be left pointing to the first byte directly after the compressed data stream. This option defaults to off. "Append => 0|1" This option controls what the "read" method does with uncompressed data. If set to 1, all uncompressed data will be appended to the output parameter of the "read" method. If set to 0, the contents of the output parameter of the "read" method will be overwritten by the uncompressed data. Defaults to 0. "Strict => 0|1" This option is a no-op. "Small => 0|1" When non-zero this options will make bzip2 use a decompression algorithm that uses less memory at the expense of increasing the amount of time taken for decompression. Default is 0. Examples TODO Methods read Usage is $status = $z->read($buffer) Reads a block of compressed data (the size the the compressed block is determined by the "Buffer" option in the constructor), uncompresses it and writes any uncompressed data into $buffer. If the "Append" parameter is set in the constructor, the uncompressed data will be appended to the $buffer parameter. Otherwise $buffer will be overwritten. Returns the number of uncompressed bytes written to $buffer, zero if eof or a negative number on error. read Usage is $status = $z->read($buffer, $length) $status = $z->read($buffer, $length, $offset) $status = read($z, $buffer, $length) $status = read($z, $buffer, $length, $offset) Attempt to read $length bytes of uncompressed data into $buffer. The main difference between this form of the "read" method and the previous one, is that this one will attempt to return exactly $length bytes. The only circumstances that this function will not is if end-of-file or an IO error is encountered. Returns the number of uncompressed bytes written to $buffer, zero if eof or a negative number on error. getline Usage is $line = $z->getline() $line = <$z> Reads a single line. This method fully supports the use of of the variable $/ (or $INPUT_RECORD_SEPARATOR or $RS when "English" is in use) to determine what constitutes an end of line. Paragraph mode, record mode and file slurp mode are all supported. getc Usage is $char = $z->getc() Read a single character. ungetc Usage is $char = $z->ungetc($string) getHeaderInfo Usage is $hdr = $z->getHeaderInfo(); @hdrs = $z->getHeaderInfo(); This method returns either a hash reference (in scalar context) or a list or hash references (in array context) that contains information about each of the header fields in the compressed data stream(s). tell Usage is $z->tell() tell $z Returns the uncompressed file offset. eof Usage is $z->eof(); eof($z); Returns true if the end of the compressed input stream has been reached. seek $z->seek($position, $whence); seek($z, $position, $whence); Provides a sub-set of the "seek" functionality, with the restriction that it is only legal to seek forward in the input file/buffer. It is a fatal error to attempt to seek backward. The $whence parameter takes one the usual values, namely SEEK_SET, SEEK_CUR or SEEK_END. Returns 1 on success, 0 on failure. binmode Usage is $z->binmode binmode $z ; This is a noop provided for completeness. opened $z->opened() Returns true if the object currently refers to a opened file/buffer. autoflush my $prev = $z->autoflush() my $prev = $z->autoflush(EXPR) If the $z object is associated with a file or a filehandle, this method returns the current autoflush setting for the underlying filehandle. If "EXPR" is present, and is non-zero, it will enable flushing after every write/print operation. If $z is associated with a buffer, this method has no effect and always returns "undef". Note that the special variable $| cannot be used to set or retrieve the autoflush setting. input_line_number $z->input_line_number() $z->input_line_number(EXPR) Returns the current uncompressed line number. If "EXPR" is present it has the effect of setting the line number. Note that setting the line number does not change the current position within the file/buffer being read. The contents of $/ are used to to determine what constitutes a line terminator. fileno $z->fileno() fileno($z) If the $z object is associated with a file or a filehandle, "fileno" will return the underlying file descriptor. Once the "close" method is called "fileno" will return "undef". If the $z object is is associated with a buffer, this method will return "undef". close $z->close() ; close $z ; Closes the output file/buffer. For most versions of Perl this method will be automatically invoked if the IO::Uncompress::Bunzip2 object is destroyed (either explicitly or by the variable with the reference to the object going out of scope). The exceptions are Perl versions 5.005 through 5.00504 and 5.8.0. In these cases, the "close" method will be called automatically, but not until global destruction of all live objects when the program is terminating. Therefore, if you want your scripts to be able to run on all versions of Perl, you should call "close" explicitly and not rely on automatic closing. Returns true on success, otherwise 0. If the "AutoClose" option has been enabled when the IO::Uncompress::Bunzip2 object was created, and the object is associated with a file, the underlying file will also be closed. nextStream Usage is my $status = $z->nextStream(); Skips to the next compressed data stream in the input file/buffer. If a new compressed data stream is found, the eof marker will be cleared and $. will be reset to 0. Returns 1 if a new stream was found, 0 if none was found, and -1 if an error was encountered. trailingData Usage is my $data = $z->trailingData(); Returns the data, if any, that is present immediately after the compressed data stream once uncompression is complete. It only makes sense to call this method once the end of the compressed data stream has been encountered. This option can be used when there is useful information immediately following the compressed data stream, and you don't know the length of the compressed data stream. If the input is a buffer, "trailingData" will return everything from the end of the compressed data stream to the end of the buffer. If the input is a filehandle, "trailingData" will return the data that is left in the filehandle input buffer once the end of the compressed data stream has been reached. You can then use the filehandle to read the rest of the input file. Don't bother using "trailingData" if the input is a filename. If you know the length of the compressed data stream before you start uncompressing, you can avoid having to use "trailingData" by setting the "InputLength" option in the constructor. Importing No symbolic constants are required by this IO::Uncompress::Bunzip2 at present. :all Imports "bunzip2" and $Bunzip2Error. Same as doing this use IO::Uncompress::Bunzip2 qw(bunzip2 $Bunzip2Error) ; EXAMPLES
Working with Net::FTP See IO::Uncompress::Bunzip2::FAQ SEE ALSO
Compress::Zlib, IO::Compress::Gzip, IO::Uncompress::Gunzip, IO::Compress::Deflate, IO::Uncompress::Inflate, IO::Compress::RawDeflate, IO::Uncompress::RawInflate, IO::Compress::Bzip2, IO::Compress::Lzop, IO::Uncompress::UnLzop, IO::Compress::Lzf, IO::Uncompress::UnLzf, IO::Uncompress::AnyInflate, IO::Uncompress::AnyUncompress Compress::Zlib::FAQ File::GlobMapper, Archive::Zip, Archive::Tar, IO::Zlib The primary site for the bzip2 program is http://www.bzip.org. See the module Compress::Bzip2 AUTHOR
This module was written by Paul Marquess, pmqs@cpan.org. MODIFICATION HISTORY
See the Changes file. COPYRIGHT AND LICENSE
Copyright (c) 2005-2008 Paul Marquess. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.18.2 2014-01-06 IO::Uncompress::Bunzip2(3pm)
All times are GMT -4. The time now is 08:41 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy