Perl - FileHandles that crash memory

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Perl - FileHandles that crash memory
# 1  
Old 02-19-2008
Perl - FileHandles that crash memory

I have a program that opens a file handle that reads a lot of logs, it uses globbing, for example ba* matches bananas.log, bank.log, etc. I have seven different names, each with several logs that I run to a grep pattern. I have subroutines for each match and a loop w/o args that processes this. When I run it I get this:

Out of memory!

I went to the perl doc and tried the suggestions, but still doesn't work. Just wondering if anyone here had any experience with "Out of memory". Even if it isn't relevent to above, it would help so we don't "reinvent the wheel" (even though that's what most of us do with scripts, hehe).
# 2  
Old 02-19-2008
Likely this is the <b>ulimit</b> or actual kernel resource rather than perl.

You can easily create a while loop that { push @foo, rand(); } and run out of memory. Testing for memory used is sort of tricky depending on perspective, but perlmonks has plenty of threads on it.
# 3  
Old 02-20-2008
grep returns a list, which means it reads the entire file/folder of whatever you grep. If the files are big you are just simply running out of physical memory because perl will use all the memory the system has available. In this case you probably do have to rewrite some code to use less memory, such as:

   while(<FH>) {
       search for whatever
close FH;

# 4  
Old 02-20-2008
perl -e "fork while 1" &

this used to exhaust all the system available inodes, most kernels are patched against it.
# 5  
Old 02-20-2008
You are all rock stars, grep was not needed at all as mentioned. I'm smacking head on desk now.
# 6  
Old 02-20-2008
Another problem with this type of work while { match } is that often the regex is rebuilt over and over. if you an build the regex outside the while that will speed things up ... DRAMATICALLY.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Out of memory issue in perl

I am getting a out of memory issue while executing the perl program. Per version : /opt/acc_perl/lib/site_perl/5.14.2 Read in 54973 total records Read in 54973 table records from table. Out of memory! so the job get failed due to out of memory. need to get rid of the out of memory... (3 Replies)
Discussion started by: ramkumar15
3 Replies

2. UNIX for Dummies Questions & Answers

Fork resource unavailable error, max # filehandles open?

I wrote a perl program that simultaneously reads in data from 691 tar.gz files using zcat. I can run one instance of the program without any issues and the memory and swap sizes are negligible. However, when I attempt to run more than 1 I start to get fork: resource unavailable messages. Are... (6 Replies)
Discussion started by: aquinom85
6 Replies

3. Shell Programming and Scripting

Perl code Out of memory problem

Hi I am working with a 3.0 GB file and running the following script to process it. After running for 15 minutes, the script stops saying Out of memory or Killed. Can the experts suggest where my code can be improved to save memory? Thanks in advance. #! usr/bin/perl use warnings; open... (3 Replies)
Discussion started by: polsum
3 Replies

4. Shell Programming and Scripting

Out Of Memory in Perl

Hi , when one of the perl script is being executed i am getting the following error as out of memory, when i check memory utilisation it is showing as 60% . cp utilisation as 11%. swap is free for 19 gb. can anyone guide me to sort the above isssue.i want to know the root cause. (2 Replies)
Discussion started by: Ajoy
2 Replies

5. Shell Programming and Scripting

User perl to get memory installed in a machine

I currently have a statistics gathering script i run on my Linux servers. One of the stat i gather is total memory in the machine. The script is all perl with the exception of gathering the memory for that i use the following command: $ram = (`cat /proc/meminfo | grep "MemTotal" | awk... (1 Reply)
Discussion started by: insania
1 Replies

6. Shell Programming and Scripting

bash/perl script that will use up all memory

Hi, Is there any known trick that will consume all of my memory and force my machine to swap? I'm scripting a monitor in perl that will alert me when available memory is dangerously low and system is swapping, and I would like to simulate the scenario. I am clueless how to make my system lack... (4 Replies)
Discussion started by: marcpascual
4 Replies

7. Programming

Process crash when allocating memory

Hi I'm currently using C++ on a HP-UX 11i system (upgrading some libraries) and am encountering a problem with the process crashing when allocating memory via a call to new (a rather large array of objects are being created). Is there a way to find out what the sizes of the stack and heap are?... (1 Reply)
Discussion started by: themezzaman
1 Replies

8. UNIX for Dummies Questions & Answers

perl memory leak detector

Could somebody give links for memory detector for perl? Thanks :) (4 Replies)
Discussion started by: matrixmadhan
4 Replies

9. Shell Programming and Scripting

Out of memory in PERL

I dont know whether this is a common error but it seems my PERL program is running out of buffer memory. Inside a loop I am processing an array (reading elements and displaying in it--note: for every iteration of the loop the WHOLE array is being scanned once--...the array was created by reading a... (5 Replies)
Discussion started by: Abhishek Ghose
5 Replies

10. UNIX for Advanced & Expert Users

Weakness in Perl CGI causes memory dump ??

I have discovered a curious phenomenon in GCI. I need some advice from someone far more adept. Is this a bug or potential security weakness? Context: Redhat 8.0 on xx86 (pentium 3) Apache 2.0, Perl 5.80 Background: I've been trying to harden a web application which accepts user uploaded... (0 Replies)
Discussion started by: andyj
0 Replies
Login or Register to Ask a Question