Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Is this "Out of Memory!" error due to sort() of perl how to resolve it?
Top Forums Shell Programming and Scripting Is this "Out of Memory!" error due to sort() of perl how to resolve it? Post 302434489 by lilili07 on Friday 2nd of July 2010 08:37:13 PM
Old 07-02-2010
Is this "Out of Memory!" error due to sort() of perl how to resolve it?
I am running a program written in perl script and it is stopped with "Out of memory!" error. This is very strange because at the time then the program is aborted, it only used 4GB RAM and there are still 30GB free physical memory left in the system. I check the perl script and found the program stopped at the place when it tried to sort some string set. Look at this:
"foreach my $string (sort keys %{$stringSet->{'strings'}}) {"
where the stringSet is composed of 4731718 string and the length of each string is around 30 bytes.

Do you think the Out of memory issue is due to the sort function?
if yes, then how does the sort in perl is implemented and how can I rewrite it?

Thanks,

---------- Post updated at 05:37 PM ---------- Previous update was at 05:20 PM ----------

Some more details FYI:
The program calls sort function because it got the data and tried to sort them and then store them to an xml file.

Here is the prstat and vmstat result when the program stopped:
prstat -a |grep <this program>
15901 root 3949M 3517M cpu13 50 0 1:12:26 6.2% <this program>./1
it shows the program used 3517M RAM when it is dead;
note this machine has 16 CPUs and this program used 6.2% of them;

vmstat 5 5
Vmstat:
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs us sy id
0 0 0 24500544 28493564 717 5122 0 0 0 0 0 0 0 0 0 27693 22769 32615 8 5 88
it shows the machine in which the program run still had 28.49GB free memory left when the program is dead;

this is the result of ulimit -a:
# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16357
virtual memory (kbytes, -v) unlimited
 

10 More Discussions You Might Find Interesting

1. AIX

"too big" and "not enough memory" errors in shell script

Hi, This is odd, however here goes. There are several shell scripts that run in our production environment AIX 595 LPAR m/c, which has sufficient memory 14GB (physical memory) and horsepower 5CPUs. However from time to time we get the following errors in these shell scripts. The time when these... (11 Replies)
Discussion started by: jerardfjay
11 Replies

2. Forum Support Area for Unregistered Users & Account Problems

Cannot register due to "spam" error message

Hi all, I am trying to register but it seems my IP address is being seen or black listed as a spam address. I get the following message: "Registration denied. Sorry, The UNIX and Linux Forums runs an active policy of not allowing spammers. Please contact us via by posting in this forum if... (0 Replies)
Discussion started by: codenjanod
0 Replies

3. Solaris

How to resolve error "INIT: Cannot stat /etc/inittab, errno: 2"

Hi All, I am getting an error message when I execute command “zlogin -C sunsrv4z5” on my root server. INIT: Cannot stat /etc/inittab, errno: 2 INIT: Cannot stat /etc/inittab, errno: 2 As per my analysis it seems that some files inside /etc folder are deleted. This server was... (14 Replies)
Discussion started by: surbhit4u
14 Replies

4. Shell Programming and Scripting

"Cannot allocate memory" error when run from script

hi in my application there is a line with open(/dev/mydevice,0); it work good when it run manually, but when try to run it within script /////////////////////////////////////////// #!/bin/sh ./device_test 0 0 0 1 1 1 0 0 1 ///////////////////////////////////////// I receive 'Failed... (3 Replies)
Discussion started by: majeed
3 Replies

5. Shell Programming and Scripting

Meaning of "b" modifier in "sort" command

I need to sort the following file by the rhdiskpower devices in the last column: Total_MB Free_MB OS_MB Name Failgroup Library Label UDID Product Redund Path 1024 851 1024 OCRVOT1_0000 OCRVOT1_0000 System UNKNOWN ... (3 Replies)
Discussion started by: wjssj
3 Replies

6. Shell Programming and Scripting

"Syntax Error sometimes due to corruption of variable value "

Hi , I have script as follows , #!/usr/bin/ksh -x if then alias echo="echo -e" fi MAX_ENTRIES=1024 nb_of_entries=`echo "$list_of_entries" | wc -w` # Set number of tables eval nb_of_tables=\`expr `expr $nb_of_entries / $MAX_ENTRIES` + 1 \` # Output the number of tables echo... (6 Replies)
Discussion started by: breezevinay
6 Replies

7. Shell Programming and Scripting

Perl failure with "main::$fn" used only once:" in error logs

Hi all, Can anyone guess why this is happening? I am new to Perl, so please help me to fix this error: - I have a static html file which calls the cgi-perl script. HTML Code: <html> <head> <title> Hey Dude! </title> </head> <body> <form method="POST"... (3 Replies)
Discussion started by: bashily
3 Replies

8. Ubuntu

What is solution for this error "tar: Exiting with failure status due to previous errors"?

Does anyone know what is solution for this error ?tar: Exiting with failure status due to previous errors from last 3 days I am trying to take backup of home/user directory getting again and again same error please anyone give me solution (8 Replies)
Discussion started by: Akshay Hegde
8 Replies

9. UNIX for Beginners Questions & Answers

Extract delta records using with "comm" and "sort" commands combination

Hi All, I have 2 pipe delimited files viz., file_old and file_new. I'm trying to compare these 2 files, and extract all the different rows between them into a new_file. comm -3 < sort file_old < sort file_new > new_file I am getting the below error: -ksh: sort: cannot open But if I do... (7 Replies)
Discussion started by: njny
7 Replies

10. UNIX for Beginners Questions & Answers

Perl: Can someone please explain this code "sort { $a <=> $b } @unsorted"

@sorted = sort { $a <=> $b } @unsorted; I am having hard time understanding how this works? I know the output but interested to know the working. Thanks in advance. (2 Replies)
Discussion started by: Tanu
2 Replies

LEARN ABOUT CENTOS

test::script

Test::Script(3) 					User Contributed Perl Documentation					   Test::Script(3)

NAME

       Test::Script - Basic cross-platform tests for scripts

DESCRIPTION

       The intent of this module is to provide a series of basic tests for 80% of the testing you will need to do for scripts in the script (or
       bin as is also commonly used) paths of your Perl distribution.

       Further, it aims to provide this functionality with perfect platform-compatibility, and in a way that is as unobtrusive as possible.

       That is, if the program works on a platform, then Test::Script should always work on that platform as well. Anything less than 100% is
       considered unacceptable.

       In doing so, it is hoped that Test::Script can become a module that you can safely make a dependency of all your modules, without risking
       that your module won't on some platform because of the dependency.

       Where a clash exists between wanting more functionality and maintaining platform safety, this module will err on the side of platform
       safety.

FUNCTIONS

   script_compiles
	   script_compiles( 'script/foo.pl', 'Main script compiles' );

       The "script_compiles" test calls the script with "perl -c script.pl", and checks that it returns without error.

       The path it should be passed is a relative unix-format script name. This will be localised when running "perl -c" and if the test fails the
       local name used will be shown in the diagnostic output.

       Note also that the test will be run with the same perl interpreter that is running the test script (and not with the default system perl).
       This will also be shown in the diagnostic output on failure.

   script_runs
	   script_runs( 'script/foo.pl', 'Main script runs' );

       The "script_runs" test executes the script with "perl script.pl" and checks that it returns success.

       The path it should be passed is a relative unix-format script name. This will be localised when running "perl -c" and if the test fails the
       local name used will be shown in the diagnostic output.

       The test will be run with the same perl interpreter that is running the test script (and not with the default system perl). This will also
       be shown in the diagnostic output on failure.

SUPPORT

       All bugs should be filed via the bug tracker at

       <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Test-Script>

       For other issues, or commercial enhancement and support, contact the author.

AUTHOR

       Adam Kennedy <adamk@cpan.org>

SEE ALSO

       prove, <http://ali.as/>

COPYRIGHT

       Copyright 2006 - 2009 Adam Kennedy.

       This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

       The full text of the license can be found in the LICENSE file included with this module.

perl v5.16.3							    2009-11-24							   Test::Script(3)
All times are GMT -4. The time now is 09:11 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy