Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Is this "Out of Memory!" error due to sort() of perl how to resolve it?
Top Forums Shell Programming and Scripting Is this "Out of Memory!" error due to sort() of perl how to resolve it? Post 302434489 by lilili07 on Friday 2nd of July 2010 08:37:13 PM
Old 07-02-2010
Is this "Out of Memory!" error due to sort() of perl how to resolve it?
I am running a program written in perl script and it is stopped with "Out of memory!" error. This is very strange because at the time then the program is aborted, it only used 4GB RAM and there are still 30GB free physical memory left in the system. I check the perl script and found the program stopped at the place when it tried to sort some string set. Look at this:
"foreach my $string (sort keys %{$stringSet->{'strings'}}) {"
where the stringSet is composed of 4731718 string and the length of each string is around 30 bytes.

Do you think the Out of memory issue is due to the sort function?
if yes, then how does the sort in perl is implemented and how can I rewrite it?

Thanks,

---------- Post updated at 05:37 PM ---------- Previous update was at 05:20 PM ----------

Some more details FYI:
The program calls sort function because it got the data and tried to sort them and then store them to an xml file.

Here is the prstat and vmstat result when the program stopped:
prstat -a |grep <this program>
15901 root 3949M 3517M cpu13 50 0 1:12:26 6.2% <this program>./1
it shows the program used 3517M RAM when it is dead;
note this machine has 16 CPUs and this program used 6.2% of them;

vmstat 5 5
Vmstat:
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs us sy id
0 0 0 24500544 28493564 717 5122 0 0 0 0 0 0 0 0 0 27693 22769 32615 8 5 88
it shows the machine in which the program run still had 28.49GB free memory left when the program is dead;

this is the result of ulimit -a:
# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16357
virtual memory (kbytes, -v) unlimited
 

10 More Discussions You Might Find Interesting

1. AIX

"too big" and "not enough memory" errors in shell script

Hi, This is odd, however here goes. There are several shell scripts that run in our production environment AIX 595 LPAR m/c, which has sufficient memory 14GB (physical memory) and horsepower 5CPUs. However from time to time we get the following errors in these shell scripts. The time when these... (11 Replies)
Discussion started by: jerardfjay
11 Replies

2. Forum Support Area for Unregistered Users & Account Problems

Cannot register due to "spam" error message

Hi all, I am trying to register but it seems my IP address is being seen or black listed as a spam address. I get the following message: "Registration denied. Sorry, The UNIX and Linux Forums runs an active policy of not allowing spammers. Please contact us via by posting in this forum if... (0 Replies)
Discussion started by: codenjanod
0 Replies

3. Solaris

How to resolve error "INIT: Cannot stat /etc/inittab, errno: 2"

Hi All, I am getting an error message when I execute command “zlogin -C sunsrv4z5” on my root server. INIT: Cannot stat /etc/inittab, errno: 2 INIT: Cannot stat /etc/inittab, errno: 2 As per my analysis it seems that some files inside /etc folder are deleted. This server was... (14 Replies)
Discussion started by: surbhit4u
14 Replies

4. Shell Programming and Scripting

"Cannot allocate memory" error when run from script

hi in my application there is a line with open(/dev/mydevice,0); it work good when it run manually, but when try to run it within script /////////////////////////////////////////// #!/bin/sh ./device_test 0 0 0 1 1 1 0 0 1 ///////////////////////////////////////// I receive 'Failed... (3 Replies)
Discussion started by: majeed
3 Replies

5. Shell Programming and Scripting

Meaning of "b" modifier in "sort" command

I need to sort the following file by the rhdiskpower devices in the last column: Total_MB Free_MB OS_MB Name Failgroup Library Label UDID Product Redund Path 1024 851 1024 OCRVOT1_0000 OCRVOT1_0000 System UNKNOWN ... (3 Replies)
Discussion started by: wjssj
3 Replies

6. Shell Programming and Scripting

"Syntax Error sometimes due to corruption of variable value "

Hi , I have script as follows , #!/usr/bin/ksh -x if then alias echo="echo -e" fi MAX_ENTRIES=1024 nb_of_entries=`echo "$list_of_entries" | wc -w` # Set number of tables eval nb_of_tables=\`expr `expr $nb_of_entries / $MAX_ENTRIES` + 1 \` # Output the number of tables echo... (6 Replies)
Discussion started by: breezevinay
6 Replies

7. Shell Programming and Scripting

Perl failure with "main::$fn" used only once:" in error logs

Hi all, Can anyone guess why this is happening? I am new to Perl, so please help me to fix this error: - I have a static html file which calls the cgi-perl script. HTML Code: <html> <head> <title> Hey Dude! </title> </head> <body> <form method="POST"... (3 Replies)
Discussion started by: bashily
3 Replies

8. Ubuntu

What is solution for this error "tar: Exiting with failure status due to previous errors"?

Does anyone know what is solution for this error ?tar: Exiting with failure status due to previous errors from last 3 days I am trying to take backup of home/user directory getting again and again same error please anyone give me solution (8 Replies)
Discussion started by: Akshay Hegde
8 Replies

9. UNIX for Beginners Questions & Answers

Extract delta records using with "comm" and "sort" commands combination

Hi All, I have 2 pipe delimited files viz., file_old and file_new. I'm trying to compare these 2 files, and extract all the different rows between them into a new_file. comm -3 < sort file_old < sort file_new > new_file I am getting the below error: -ksh: sort: cannot open But if I do... (7 Replies)
Discussion started by: njny
7 Replies

10. UNIX for Beginners Questions & Answers

Perl: Can someone please explain this code "sort { $a <=> $b } @unsorted"

@sorted = sort { $a <=> $b } @unsorted; I am having hard time understanding how this works? I know the output but interested to know the working. Thanks in advance. (2 Replies)
Discussion started by: Tanu
2 Replies

LEARN ABOUT LINUX

pidof

PIDOF(8)						Linux System Administrator's Manual						  PIDOF(8)

NAME

       pidof -- find the process ID of a running program.

SYNOPSIS

       pidof [-s] [-c] [-n] [-x] [-o omitpid[,omitpid..]]  [-o omitpid[,omitpid..]..]  program [program..]

DESCRIPTION

       Pidof  finds  the  process  id's (pids) of the named programs. It prints those id's on the standard output. This program is on some systems
       used in run-level change scripts, especially when the system has a System-V like rc structure. In that case these scripts  are  located	in
       /etc/rc?.d, where ? is the runlevel. If the system has a start-stop-daemon (8) program that should be used instead.

OPTIONS

       -s     Single shot - this instructs the program to only return one pid.

       -c     Only  return  process ids that are running with the same root directory.	This option is ignored for non-root users, as they will be
	      unable to check the current root directory of processes they do not own.

       -n     Avoid stat(2) system function call on all binaries which are located on network based file systems like NFS.  Instead of using  this
	      option the the variable PIDOF_NETFS may be set and exported.

       -x     Scripts too - this causes the program to also return process id's of shells running the named scripts.

       -o omitpid
	      Tells  pidof  to omit processes with that process id. The special pid %PPID can be used to name the parent process of the pidof pro-
	      gram, in other words the calling shell or shell script.

EXIT STATUS

       0      At least one program was found with the requested name.

       1      No program was found with the requested name.

NOTES

       pidof is actually the same program as killall5; the program behaves according to the name under which it is called.

       When pidof is invoked with a full pathname to the program it should find the pid of, it is reasonably safe. Otherwise it is  possible  that
       it  returns  pids  of  running programs that happen to have the same name as the program you're after but are actually other programs. Note
       that that the executable name of running processes is calculated with readlink(2), so symbolic links to executables will also match.

SEE ALSO

       shutdown(8), init(8), halt(8), reboot(8), killall5(8)

AUTHOR

       Miquel van Smoorenburg, miquels@cistron.nl

								    01 Sep 1998 							  PIDOF(8)
All times are GMT -4. The time now is 02:45 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy