Perl code Out of memory problem


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Perl code Out of memory problem
# 1  
Old 09-11-2011
Perl code Out of memory problem

Hi I am working with a 3.0 GB file and running the following script to process it. After running for 15 minutes, the script stops saying Out of memory or Killed.

Can the experts suggest where my code can be improved to save memory?

Thanks in advance.

Code:
#! usr/bin/perl
use warnings;
open RAW, "$ARGV[1]";
open ADAPTER, "$ARGV[0]";
@reads =();
@ada =();
while (<RAW>) {
      if (($. -2) % 4 == 0) {push (@reads, $_);             
                            }
              }
while (<ADAPTER>) {                                                    @adapter = split (//);
            $i= -1;
        while (++$i < $#adapter-8) {
                            $_ = "$adapter[$i]$adapter[$i+1]$adapter[$i+2]$adapter[$i+3]$adapter[$i+4]$adapter[$i+5]$adapter[$i+6]$adapter[$i+7]$adapter[$i+8]";
                             push (@ada, $_);                             
                                   }
                  }
@reads1 = ();
foreach $reads(@reads) {                              
             foreach $ada(@ada) {
                                  $reads =~ s/$ada.*|(A{9}|G{9}|C{9}|T{9})[A-Z]+//g; 
                                }                 
                       push (@reads1, $reads);  
                       }
%duplicates = ();
chomp (@reads1);
$i = 0;
open Fasta_reads, ">/home/Desktop/fasta_reads";
foreach (@reads1) { 
            if (length $_ >14) {                                      
                     $duplicates{$_}++;
                  }            }
foreach $count (keys %duplicates) {$i +=1;
               $read_length = length $count;
               $_ = "> $i $read_length $duplicates{$count}\n$count\n";
               print Fasta_reads;
                                  } close Fasta_reads;

# 2  
Old 09-11-2011
It's hard and unpleasant to read code without the description of your problem, comments and sample input/output. I'm going to admire a man who can help you.
# 3  
Old 09-11-2011
Quote:
Originally Posted by yazu
It's hard and unpleasant to read code without the description of your problem, comments and sample input/output. I'm going to admire a man who can help you.
sorry - I didnt think the purpose of code is relevant here. But these are the steps I am following in the code.

I have a 3 GB text file with DNA sequence reads.
1. I extract reads at specific line numbers
2. substitute reads containing some expressions. These expressions are chopped from a single DNA adapter sequence.
3. count the duplicates
4. extract reads of only >14 length.
5. add > sign and serial number and length to each read to make it suitable for downstream purposes.

It appears that the problem is stemming from too much usage of arrays. If I split the code and use filehandles, it is able to execute the code but still its taking a very long time.

I started to think that AWK is much faster than Perl.Smilie

thanks.Smilie
# 4  
Old 09-11-2011
What do those two files you pass as parameters look like?

tyler_durden
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Out of memory issue in perl

I am getting a out of memory issue while executing the perl program. Per version : /opt/acc_perl/lib/site_perl/5.14.2 Read in 54973 total records Read in 54973 table records from table. Out of memory! so the job get failed due to out of memory. need to get rid of the out of memory... (3 Replies)
Discussion started by: ramkumar15
3 Replies

2. Shell Programming and Scripting

Perl code- word count problem

Hi, I am having .csv files contains some row - Info: Value of field name 'SecurityExchange' is not supported ","Original Order Tuple Please see the below perl code carefully- /Info: (+),Original (\w+) Tuple/ and do { ($category, $type) = ($1, $2); if($type eq 'Execution')... (1 Reply)
Discussion started by: pspriyanka
1 Replies

3. Shell Programming and Scripting

Out Of Memory in Perl

Hi , when one of the perl script is being executed i am getting the following error as out of memory, when i check memory utilisation it is showing as 60% . cp utilisation as 11%. swap is free for 19 gb. can anyone guide me to sort the above isssue.i want to know the root cause. (2 Replies)
Discussion started by: Ajoy
2 Replies

4. Shell Programming and Scripting

User perl to get memory installed in a machine

I currently have a statistics gathering script i run on my Linux servers. One of the stat i gather is total memory in the machine. The script is all perl with the exception of gathering the memory for that i use the following command: $ram = (`cat /proc/meminfo | grep "MemTotal" | awk... (1 Reply)
Discussion started by: insania
1 Replies

5. Shell Programming and Scripting

bash/perl script that will use up all memory

Hi, Is there any known trick that will consume all of my memory and force my machine to swap? I'm scripting a monitor in perl that will alert me when available memory is dangerously low and system is swapping, and I would like to simulate the scenario. I am clueless how to make my system lack... (4 Replies)
Discussion started by: marcpascual
4 Replies

6. Shell Programming and Scripting

Perl - FileHandles that crash memory

I have a program that opens a file handle that reads a lot of logs, it uses globbing, for example ba* matches bananas.log, bank.log, etc. I have seven different names, each with several logs that I run to a grep pattern. I have subroutines for each match and a loop w/o args that processes this. ... (5 Replies)
Discussion started by: nj78
5 Replies

7. UNIX for Dummies Questions & Answers

perl memory leak detector

Could somebody give links for memory detector for perl? Thanks :) (4 Replies)
Discussion started by: matrixmadhan
4 Replies

8. Shell Programming and Scripting

Out of memory in PERL

I dont know whether this is a common error but it seems my PERL program is running out of buffer memory. Inside a loop I am processing an array (reading elements and displaying in it--note: for every iteration of the loop the WHOLE array is being scanned once--...the array was created by reading a... (5 Replies)
Discussion started by: Abhishek Ghose
5 Replies

9. Programming

how to round up a memory address(memory alignment problem)

Hi, I try to marshal a unsigned int and a char * into a buffer, and then unmarshal them later to get them out. I need to put the char * in the front and unsigned int at the end of the buffer. However, my system always give me "BUS ERROR". I am using Sun Sparcs Sloris 2.10. My code to marshal... (6 Replies)
Discussion started by: nj302
6 Replies

10. UNIX for Advanced & Expert Users

Weakness in Perl CGI causes memory dump ??

I have discovered a curious phenomenon in GCI. I need some advice from someone far more adept. Is this a bug or potential security weakness? Context: Redhat 8.0 on xx86 (pentium 3) Apache 2.0, Perl 5.80 Background: I've been trying to harden a web application which accepts user uploaded... (0 Replies)
Discussion started by: andyj
0 Replies
Login or Register to Ask a Question