Optimize/speed-up perl extraction


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Optimize/speed-up perl extraction
# 1  
Old 08-04-2007
Optimize/speed-up perl extraction

Hi,

Is there a way I can extract my data faster. You know my data is 1.2 GB text file with 8Million rows with 38 columns/fields. Imagine how huge this is.

How I can optimized the data extraction using perl. That is why I'm creating a script to filter only those informations that I need. Is there any modules available or any way to speed up the process of extraction? Tnx in advance.

Cheers! Smilie

Br, Pete
# 2  
Old 08-04-2007
Quote:
Imagine how huge this is.
This is an absolute relative term.

What is that you are interested in extraction ?

What is the problem with the current approach that you have ?

Could you please post sample input and output ? That would help us a lot better to understand what is needed to do ! Smilie
# 3  
Old 08-04-2007
Quote:
Originally Posted by pinpe
Imagine how huge this is.
We don't need to, you have just told us. Smilie

Quote:
Originally Posted by pinpe
How I can optimized the data extraction using perl.
How are you determining what you extract?

How do you extract it?

The fastest way would be to use a C program that can read each line into a single buffer, do the determination without any memory allocation/deallocation, then print the required sections again without memory allocation/deallocation.
# 4  
Old 08-04-2007
Quote:
The fastest way would be to use a C program that can read each line into a single buffer, do the determination without any memory allocation/deallocation, then print the required sections again without memory allocation/deallocation.
Reply With Quote
How is this possible ?

Isn't there any size restriction on the program buffer, kernel buffer ?

If there is a feasibility to have single buffer to hold the contents whatever be the size, there could be just one flush that could do the job ( this is purely subjective )

Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Speed up extraction od tar.bz2 files using bash

The below bash will untar each tar.bz2 folder in the directory, then remove the tar.bz2. Each of the tar.bz2 folders ranges from 40-75GB and currently takes ~2 hours to extract. Is there a way to speed up the extraction process? I am using a xeon processor with 12 cores. Thank you :). ... (7 Replies)
Discussion started by: cmccabe
7 Replies

2. Shell Programming and Scripting

Perl-extraction using windows cmd

Hi, I need to extract Password expires from the output of windows command print `net user %USERNAME% /domain`; in perl. So i want to redirect the output of this win-cmd to a file and try extracting Password expires along with its value. i'm trying with this code but getting errors. #!usr/bin/perl... (1 Reply)
Discussion started by: sam_bd
1 Replies

3. Shell Programming and Scripting

perl extraction of code between comments /* .. */

Hi, I am using the following code to retrieve the contents between C-style comments "/* .. */". perl -lne 'while(/(\/\*.*?\*\/)/g) {print "$1";}' This works fine when the commented section of code is present in a single line. But I also need to extract the data which is present inside... (3 Replies)
Discussion started by: royalibrahim
3 Replies

4. Shell Programming and Scripting

Data Extraction problem in perl

Hello, I want to extract the words from a file which starts with SRD-R or SRD-DR. I have written a script which is able to trace the word but it is printing the whole line. sub extract_SRD_tag{ my ($tag, $app, $path, @data, $word ); $path = shift; $app = shift; open (FILE, $path) or... (2 Replies)
Discussion started by: suvendu4urs
2 Replies

5. Shell Programming and Scripting

Data extraction in perl variable

HI, i have variable in perl like below $tmp="/home/sai/automation/work/TFP_GN.txt" it can conatain any path its filled from config file. now i want extarct the path upto this /home/sai/automation/work/ and put it in another variable say... (4 Replies)
Discussion started by: raghavendra.nsn
4 Replies

6. Shell Programming and Scripting

Slow Perl script: how to speed up?

I had written a perl script to compare two files: new and master and get the output of the first file i.e. the first file: words that are not in the master file STRUCTURE OF THE TWO FILES The first file is a series of names ramesh sushil jonga sudesh lugdi whereas the second file (could be... (4 Replies)
Discussion started by: gimley
4 Replies

7. Shell Programming and Scripting

Function extraction in PERL

the log contains mathematical operation as follows fm_void_mathematics : PCM_OP_MATHS input function PIN_FLD_NUM1 INT 1 PIN_FLD_NUM2 INT 2 PIN_FLD_RESULTS int PIN_FLD_OUT INT * D Wed Sep 16 05:40:22 2009 solaris_testing fm_void_add : PIN_FLD_SUM int 3 D Wed Sep 16 05:40:22 2009... (1 Reply)
Discussion started by: vkca
1 Replies

8. Shell Programming and Scripting

Perl function extraction

The log file reads as follows. D function_add() ADD input data 1021214 0 VAR1 STR 10 0 VAR2 STR 20 0 VAR3 STR 1 SUM=VAR1+VAR2 D function_add() ADD output data 1021267 0 DISPLAY SUM D function_sub() SUB input data 1021214 0 VAR1 STR 10 0 VAR2 STR 20 0 VAR3 STR 1 sub=VAR1-VAR2 D... (2 Replies)
Discussion started by: vkca
2 Replies

9. Shell Programming and Scripting

String Extraction in Perl

I have a string stored in a variable. For instance, $str = " Opcode called is : CM_OP_xxx " where xxx changes dynamically and can be either LOGIN or SEARCH..... depends on runtime. For example : $str = " Opcode called is : CM_OP_SEARCH " $str = " Opcode called is : CM_OP_LOGIN " I... (3 Replies)
Discussion started by: vkca
3 Replies

10. Shell Programming and Scripting

metapattern extraction in PERL

Hi, I want to extract some part of a pattern that matches my requirement in a string with PERL. A case in point is a string like: $eqtst="abh nmae res = 10 s abh nmae req = 10 s"; from which I want the words preceding the "=" symbol. Previously I was assured that there would be only 2 such... (4 Replies)
Discussion started by: Abhishek Ghose
4 Replies
Login or Register to Ask a Question