Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Optimize/speed-up perl extraction

Shell Programming and Scripting


Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 08-04-2007   -   Original Discussion by pinpe
pinpe pinpe is offline
Registered User
 
Join Date: Jul 2007
Last Activity: 18 December 2011, 2:24 PM EST
Posts: 60
Thanks: 11
Thanked 0 Times in 0 Posts
Optimize/speed-up perl extraction

Hi,

Is there a way I can extract my data faster. You know my data is 1.2 GB text file with 8Million rows with 38 columns/fields. Imagine how huge this is.

How I can optimized the data extraction using perl. That is why I'm creating a script to filter only those informations that I need. Is there any modules available or any way to speed up the process of extraction? Tnx in advance.

Cheers! Linux

Br, Pete
Sponsored Links
    #2  
Old Unix and Linux 08-04-2007   -   Original Discussion by pinpe
matrixmadhan matrixmadhan is offline Forum Advisor  
Technorati Master
 
Join Date: Mar 2005
Last Activity: 10 January 2017, 8:10 AM EST
Location: classification algos
Posts: 3,215
Thanks: 19
Thanked 31 Times in 28 Posts
Quote:
Imagine how huge this is.
This is an absolute relative term.

What is that you are interested in extraction ?

What is the problem with the current approach that you have ?

Could you please post sample input and output ? That would help us a lot better to understand what is needed to do ! Linux
Sponsored Links
    #3  
Old Unix and Linux 08-04-2007   -   Original Discussion by pinpe
porter porter is offline Forum Advisor  
Registered User
 
Join Date: Jan 2007
Last Activity: 8 January 2008, 6:50 PM EST
Posts: 2,965
Thanks: 0
Thanked 5 Times in 5 Posts
Quote:
Originally Posted by pinpe View Post
Imagine how huge this is.
We don't need to, you have just told us. Linux

Quote:
Originally Posted by pinpe View Post
How I can optimized the data extraction using perl.
How are you determining what you extract?

How do you extract it?

The fastest way would be to use a C program that can read each line into a single buffer, do the determination without any memory allocation/deallocation, then print the required sections again without memory allocation/deallocation.
    #4  
Old Unix and Linux 08-04-2007   -   Original Discussion by pinpe
matrixmadhan matrixmadhan is offline Forum Advisor  
Technorati Master
 
Join Date: Mar 2005
Last Activity: 10 January 2017, 8:10 AM EST
Location: classification algos
Posts: 3,215
Thanks: 19
Thanked 31 Times in 28 Posts
Quote:
The fastest way would be to use a C program that can read each line into a single buffer, do the determination without any memory allocation/deallocation, then print the required sections again without memory allocation/deallocation.
Reply With Quote
How is this possible ?

Isn't there any size restriction on the program buffer, kernel buffer ?

If there is a feasibility to have single buffer to hold the contents whatever be the size, there could be just one flush that could do the job ( this is purely subjective )

Linux
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Data Extraction problem in perl suvendu4urs Shell Programming and Scripting 2 08-08-2011 07:11 AM
Function extraction in PERL vkca Shell Programming and Scripting 1 12-10-2009 01:24 AM
Perl function extraction vkca Shell Programming and Scripting 2 12-04-2009 01:34 AM
String Extraction in Perl vkca Shell Programming and Scripting 3 12-03-2009 12:54 AM
metapattern extraction in PERL Abhishek Ghose Shell Programming and Scripting 4 10-19-2007 05:21 AM



All times are GMT -4. The time now is 06:36 AM.