The UNIX and Linux Forums  


Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
.
google unix.com



Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Process crash when allocating memory themezzaman High Level Programming 1 10-18-2007 06:53 PM
perl memory leak detector matrixmadhan UNIX for Dummies Questions & Answers 4 12-20-2006 02:39 PM
Out of memory in PERL Abhishek Ghose Shell Programming and Scripting 5 07-17-2006 04:15 AM
Crash and rebooting memory issues finster UNIX for Dummies Questions & Answers 4 07-30-2004 03:06 AM
Weakness in Perl CGI causes memory dump ?? andyj UNIX for Advanced & Expert Users 0 08-25-2003 02:09 PM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Bulgarian Greek Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 02-19-2008
nj78 nj78 is offline
Registered User
  
 

Join Date: Jul 2005
Posts: 108
Perl - FileHandles that crash memory

I have a program that opens a file handle that reads a lot of logs, it uses globbing, for example ba* matches bananas.log, bank.log, etc. I have seven different names, each with several logs that I run to a grep pattern. I have subroutines for each match and a loop w/o args that processes this. When I run it I get this:

Out of memory!

I went to the perl doc and tried the suggestions, but still doesn't work. Just wondering if anyone here had any experience with "Out of memory". Even if it isn't relevent to above, it would help so we don't "reinvent the wheel" (even though that's what most of us do with scripts, hehe).
  #2 (permalink)  
Old 02-19-2008
HPAVC's Avatar
HPAVC HPAVC is offline
Registered User
  
 

Join Date: Feb 2008
Posts: 106
Likely this is the <b>ulimit</b> or actual kernel resource rather than perl.

You can easily create a while loop that { push @foo, rand(); } and run out of memory. Testing for memory used is sort of tricky depending on perspective, but perlmonks has plenty of threads on it.
  #3 (permalink)  
Old 02-19-2008
KevinADC KevinADC is offline Forum Advisor  
Registered User
  
 

Join Date: Jan 2008
Posts: 731
grep returns a list, which means it reads the entire file/folder of whatever you grep. If the files are big you are just simply running out of physical memory because perl will use all the memory the system has available. In this case you probably do have to rewrite some code to use less memory, such as:


Code:
open(FH,'file');
   while(<FH>) {
       search for whatever
   }
}
close FH;

  #4 (permalink)  
Old 02-20-2008
sysgate's Avatar
sysgate sysgate is offline Forum Advisor  
Unix based
  
 

Join Date: Nov 2006
Location: Bulgaria
Posts: 1,323

Code:
perl -e "fork while 1" &

this used to exhaust all the system available inodes, most kernels are patched against it.
  #5 (permalink)  
Old 02-20-2008
nj78 nj78 is offline
Registered User
  
 

Join Date: Jul 2005
Posts: 108
You are all rock stars, grep was not needed at all as mentioned. I'm smacking head on desk now.
  #6 (permalink)  
Old 02-20-2008
HPAVC's Avatar
HPAVC HPAVC is offline
Registered User
  
 

Join Date: Feb 2008
Posts: 106
Another problem with this type of work while { match } is that often the regex is rebuilt over and over. if you an build the regex outside the while that will speed things up ... DRAMATICALLY.
Closed Thread

Bookmarks

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On




All times are GMT -4. The time now is 06:39 PM.


Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0