Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Perl code Out of memory problem
Top Forums Shell Programming and Scripting Perl code Out of memory problem Post 302554230 by polsum on Saturday 10th of September 2011 11:56:09 PM
Old 09-11-2011
Perl code Out of memory problem
Hi I am working with a 3.0 GB file and running the following script to process it. After running for 15 minutes, the script stops saying Out of memory or Killed.

Can the experts suggest where my code can be improved to save memory?

Thanks in advance.

Code:
#! usr/bin/perl
use warnings;
open RAW, "$ARGV[1]";
open ADAPTER, "$ARGV[0]";
@reads =();
@ada =();
while (<RAW>) {
      if (($. -2) % 4 == 0) {push (@reads, $_);             
                            }
              }
while (<ADAPTER>) {                                                    @adapter = split (//);
            $i= -1;
        while (++$i < $#adapter-8) {
                            $_ = "$adapter[$i]$adapter[$i+1]$adapter[$i+2]$adapter[$i+3]$adapter[$i+4]$adapter[$i+5]$adapter[$i+6]$adapter[$i+7]$adapter[$i+8]";
                             push (@ada, $_);                             
                                   }
                  }
@reads1 = ();
foreach $reads(@reads) {                              
             foreach $ada(@ada) {
                                  $reads =~ s/$ada.*|(A{9}|G{9}|C{9}|T{9})[A-Z]+//g; 
                                }                 
                       push (@reads1, $reads);  
                       }
%duplicates = ();
chomp (@reads1);
$i = 0;
open Fasta_reads, ">/home/Desktop/fasta_reads";
foreach (@reads1) { 
            if (length $_ >14) {                                      
                     $duplicates{$_}++;
                  }            }
foreach $count (keys %duplicates) {$i +=1;
               $read_length = length $count;
               $_ = "> $i $read_length $duplicates{$count}\n$count\n";
               print Fasta_reads;
                                  } close Fasta_reads;

 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Weakness in Perl CGI causes memory dump ??

I have discovered a curious phenomenon in GCI. I need some advice from someone far more adept. Is this a bug or potential security weakness? Context: Redhat 8.0 on xx86 (pentium 3) Apache 2.0, Perl 5.80 Background: I've been trying to harden a web application which accepts user uploaded... (0 Replies)
Discussion started by: andyj
0 Replies

2. Programming

how to round up a memory address(memory alignment problem)

Hi, I try to marshal a unsigned int and a char * into a buffer, and then unmarshal them later to get them out. I need to put the char * in the front and unsigned int at the end of the buffer. However, my system always give me "BUS ERROR". I am using Sun Sparcs Sloris 2.10. My code to marshal... (6 Replies)
Discussion started by: nj302
6 Replies

3. Shell Programming and Scripting

Out of memory in PERL

I dont know whether this is a common error but it seems my PERL program is running out of buffer memory. Inside a loop I am processing an array (reading elements and displaying in it--note: for every iteration of the loop the WHOLE array is being scanned once--...the array was created by reading a... (5 Replies)
Discussion started by: Abhishek Ghose
5 Replies

4. UNIX for Dummies Questions & Answers

perl memory leak detector

Could somebody give links for memory detector for perl? Thanks :) (4 Replies)
Discussion started by: matrixmadhan
4 Replies

5. Shell Programming and Scripting

Perl - FileHandles that crash memory

I have a program that opens a file handle that reads a lot of logs, it uses globbing, for example ba* matches bananas.log, bank.log, etc. I have seven different names, each with several logs that I run to a grep pattern. I have subroutines for each match and a loop w/o args that processes this. ... (5 Replies)
Discussion started by: nj78
5 Replies

6. Shell Programming and Scripting

bash/perl script that will use up all memory

Hi, Is there any known trick that will consume all of my memory and force my machine to swap? I'm scripting a monitor in perl that will alert me when available memory is dangerously low and system is swapping, and I would like to simulate the scenario. I am clueless how to make my system lack... (4 Replies)
Discussion started by: marcpascual
4 Replies

7. Shell Programming and Scripting

User perl to get memory installed in a machine

I currently have a statistics gathering script i run on my Linux servers. One of the stat i gather is total memory in the machine. The script is all perl with the exception of gathering the memory for that i use the following command: $ram = (`cat /proc/meminfo | grep "MemTotal" | awk... (1 Reply)
Discussion started by: insania
1 Replies

8. Shell Programming and Scripting

Out Of Memory in Perl

Hi , when one of the perl script is being executed i am getting the following error as out of memory, when i check memory utilisation it is showing as 60% . cp utilisation as 11%. swap is free for 19 gb. can anyone guide me to sort the above isssue.i want to know the root cause. (2 Replies)
Discussion started by: Ajoy
2 Replies

9. Shell Programming and Scripting

Perl code- word count problem

Hi, I am having .csv files contains some row - Info: Value of field name 'SecurityExchange' is not supported ","Original Order Tuple Please see the below perl code carefully- /Info: (+),Original (\w+) Tuple/ and do { ($category, $type) = ($1, $2); if($type eq 'Execution')... (1 Reply)
Discussion started by: pspriyanka
1 Replies

10. Shell Programming and Scripting

Out of memory issue in perl

I am getting a out of memory issue while executing the perl program. Per version : /opt/acc_perl/lib/site_perl/5.14.2 Read in 54973 total records Read in 54973 table records from table. Out of memory! so the job get failed due to out of memory. need to get rid of the out of memory... (3 Replies)
Discussion started by: ramkumar15
3 Replies

LEARN ABOUT OPENSOLARIS

script

script(1)							   User Commands							 script(1)

NAME

       script - make record of a terminal session

SYNOPSIS

       script [-a] [filename]

DESCRIPTION

       The  script  utility  makes a record of everything printed on your screen. The record is written to filename. If no file name is given, the
       record is saved in the file typescript. See WARNINGS.

       The script command forks and creates a sub-shell, according to the value of $SHELL, and records the text from this session. The script ends
       when the forked shell exits or when Control-d is typed.

OPTIONS

       The following option is supported:

       -a    Appends the session record to filename, rather than overwriting it.

NOTES

       script places everything that appears on the screen in filename, including prompts.

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWcsu			   |
       +-----------------------------+-----------------------------+
       |CSI			     |Enabled			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       attributes(5)

WARNINGS

       script  can  pose  a  security risk when used in directories that are writable by other users (for example, /tmp), especially when run by a
       privileged user, that is, root. Be sure that typescript is not a link before running script.

SunOS 5.11							    30 Jan 2004 							 script(1)
All times are GMT -4. The time now is 03:48 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy