Visit Our UNIX and Linux User Community


Is this "Out of Memory!" error due to sort() of perl how to resolve it?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Is this "Out of Memory!" error due to sort() of perl how to resolve it?
# 1  
Old 07-02-2010
Is this "Out of Memory!" error due to sort() of perl how to resolve it?

I am running a program written in perl script and it is stopped with "Out of memory!" error. This is very strange because at the time then the program is aborted, it only used 4GB RAM and there are still 30GB free physical memory left in the system. I check the perl script and found the program stopped at the place when it tried to sort some string set. Look at this:
"foreach my $string (sort keys %{$stringSet->{'strings'}}) {"
where the stringSet is composed of 4731718 string and the length of each string is around 30 bytes.

Do you think the Out of memory issue is due to the sort function?
if yes, then how does the sort in perl is implemented and how can I rewrite it?

Thanks,

---------- Post updated at 05:37 PM ---------- Previous update was at 05:20 PM ----------

Some more details FYI:
The program calls sort function because it got the data and tried to sort them and then store them to an xml file.

Here is the prstat and vmstat result when the program stopped:
prstat -a |grep <this program>
15901 root 3949M 3517M cpu13 50 0 1:12:26 6.2% <this program>./1
it shows the program used 3517M RAM when it is dead;
note this machine has 16 CPUs and this program used 6.2% of them;

vmstat 5 5
Vmstat:
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs us sy id
0 0 0 24500544 28493564 717 5122 0 0 0 0 0 0 0 0 0 27693 22769 32615 8 5 88
it shows the machine in which the program run still had 28.49GB free memory left when the program is dead;

this is the result of ulimit -a:
# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16357
virtual memory (kbytes, -v) unlimited
# 2  
Old 07-02-2010
You need the 64bit version of perl.
# 3  
Old 07-03-2010
Quote:
Originally Posted by binlib
You need the 64bit version of perl.
What do you mean 64bit perl? can you give me more details? what's wrong with the perl script? how to check how many bits is the perl based on?

The OS of the machine is solaries U8.
the perl of the machine is perl5/5.8.4

Thanks.

# /usr/bin/perl -v

This is perl, v5.8.4 built for i86pc-solaris-64int
(with 32 registered patches, see perl -V for more detail)

---------- Post updated at 11:33 PM ---------- Previous update was at 09:31 PM ----------

ok, maybe you are right. I saw this:By default, perl will be compiled as a 32-bit application. Unless you want to allocate more than ~ 4GB of memory inside perl, or unless you need more than 255 open file descriptors, you probably don't need perl to be a 64-bit app.
in README.solaris - search.cpan.org.

Last edited by lilili07; 07-03-2010 at 02:55 AM..
# 4  
Old 07-07-2010
Quote:
Originally Posted by binlib
You need the 64bit version of perl.

Hi, see this:
"By default, perl will be compiled as> a 32-bit application. Unless you want to allocate more than ~ 4GB of> memory inside perl, or unless you need more than 255 open file d> escriptors, you probably don't need perl to be a 64-bit app"

Does 4GB here mean single chunk of memory allocated for once or the total number of memory allocated by the program so far? Thanks,

Previous Thread | Next Thread
Test Your Knowledge in Computers #664
Difficulty: Medium
DOS was initially released with the IBM PC in 1985.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Perl: Can someone please explain this code "sort { $a <=> $b } @unsorted"

@sorted = sort { $a <=> $b } @unsorted; I am having hard time understanding how this works? I know the output but interested to know the working. Thanks in advance. (2 Replies)
Discussion started by: Tanu
2 Replies

2. UNIX for Beginners Questions & Answers

Extract delta records using with "comm" and "sort" commands combination

Hi All, I have 2 pipe delimited files viz., file_old and file_new. I'm trying to compare these 2 files, and extract all the different rows between them into a new_file. comm -3 < sort file_old < sort file_new > new_file I am getting the below error: -ksh: sort: cannot open But if I do... (7 Replies)
Discussion started by: njny
7 Replies

3. Ubuntu

What is solution for this error "tar: Exiting with failure status due to previous errors"?

Does anyone know what is solution for this error ?tar: Exiting with failure status due to previous errors from last 3 days I am trying to take backup of home/user directory getting again and again same error please anyone give me solution (8 Replies)
Discussion started by: Akshay Hegde
8 Replies

4. Shell Programming and Scripting

Perl failure with "main::$fn" used only once:" in error logs

Hi all, Can anyone guess why this is happening? I am new to Perl, so please help me to fix this error: - I have a static html file which calls the cgi-perl script. HTML Code: <html> <head> <title> Hey Dude! </title> </head> <body> <form method="POST"... (3 Replies)
Discussion started by: bashily
3 Replies

5. Shell Programming and Scripting

"Syntax Error sometimes due to corruption of variable value "

Hi , I have script as follows , #!/usr/bin/ksh -x if then alias echo="echo -e" fi MAX_ENTRIES=1024 nb_of_entries=`echo "$list_of_entries" | wc -w` # Set number of tables eval nb_of_tables=\`expr `expr $nb_of_entries / $MAX_ENTRIES` + 1 \` # Output the number of tables echo... (6 Replies)
Discussion started by: breezevinay
6 Replies

6. Shell Programming and Scripting

Meaning of "b" modifier in "sort" command

I need to sort the following file by the rhdiskpower devices in the last column: Total_MB Free_MB OS_MB Name Failgroup Library Label UDID Product Redund Path 1024 851 1024 OCRVOT1_0000 OCRVOT1_0000 System UNKNOWN ... (3 Replies)
Discussion started by: wjssj
3 Replies

7. Shell Programming and Scripting

"Cannot allocate memory" error when run from script

hi in my application there is a line with open(/dev/mydevice,0); it work good when it run manually, but when try to run it within script /////////////////////////////////////////// #!/bin/sh ./device_test 0 0 0 1 1 1 0 0 1 ///////////////////////////////////////// I receive 'Failed... (3 Replies)
Discussion started by: majeed
3 Replies

8. Solaris

How to resolve error "INIT: Cannot stat /etc/inittab, errno: 2"

Hi All, I am getting an error message when I execute command “zlogin -C sunsrv4z5” on my root server. INIT: Cannot stat /etc/inittab, errno: 2 INIT: Cannot stat /etc/inittab, errno: 2 As per my analysis it seems that some files inside /etc folder are deleted. This server was... (14 Replies)
Discussion started by: surbhit4u
14 Replies

9. Forum Support Area for Unregistered Users & Account Problems

Cannot register due to "spam" error message

Hi all, I am trying to register but it seems my IP address is being seen or black listed as a spam address. I get the following message: "Registration denied. Sorry, The UNIX and Linux Forums runs an active policy of not allowing spammers. Please contact us via by posting in this forum if... (0 Replies)
Discussion started by: codenjanod
0 Replies

10. AIX

"too big" and "not enough memory" errors in shell script

Hi, This is odd, however here goes. There are several shell scripts that run in our production environment AIX 595 LPAR m/c, which has sufficient memory 14GB (physical memory) and horsepower 5CPUs. However from time to time we get the following errors in these shell scripts. The time when these... (11 Replies)
Discussion started by: jerardfjay
11 Replies

Featured Tech Videos