Sponsored Content
Top Forums Shell Programming and Scripting Is this "Out of Memory!" error due to sort() of perl how to resolve it? Post 302434489 by lilili07 on Friday 2nd of July 2010 08:37:13 PM
Old 07-02-2010
Is this "Out of Memory!" error due to sort() of perl how to resolve it?

I am running a program written in perl script and it is stopped with "Out of memory!" error. This is very strange because at the time then the program is aborted, it only used 4GB RAM and there are still 30GB free physical memory left in the system. I check the perl script and found the program stopped at the place when it tried to sort some string set. Look at this:
"foreach my $string (sort keys %{$stringSet->{'strings'}}) {"
where the stringSet is composed of 4731718 string and the length of each string is around 30 bytes.

Do you think the Out of memory issue is due to the sort function?
if yes, then how does the sort in perl is implemented and how can I rewrite it?

Thanks,

---------- Post updated at 05:37 PM ---------- Previous update was at 05:20 PM ----------

Some more details FYI:
The program calls sort function because it got the data and tried to sort them and then store them to an xml file.

Here is the prstat and vmstat result when the program stopped:
prstat -a |grep <this program>
15901 root 3949M 3517M cpu13 50 0 1:12:26 6.2% <this program>./1
it shows the program used 3517M RAM when it is dead;
note this machine has 16 CPUs and this program used 6.2% of them;

vmstat 5 5
Vmstat:
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs us sy id
0 0 0 24500544 28493564 717 5122 0 0 0 0 0 0 0 0 0 27693 22769 32615 8 5 88
it shows the machine in which the program run still had 28.49GB free memory left when the program is dead;

this is the result of ulimit -a:
# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16357
virtual memory (kbytes, -v) unlimited
 

10 More Discussions You Might Find Interesting

1. AIX

"too big" and "not enough memory" errors in shell script

Hi, This is odd, however here goes. There are several shell scripts that run in our production environment AIX 595 LPAR m/c, which has sufficient memory 14GB (physical memory) and horsepower 5CPUs. However from time to time we get the following errors in these shell scripts. The time when these... (11 Replies)
Discussion started by: jerardfjay
11 Replies

2. Forum Support Area for Unregistered Users & Account Problems

Cannot register due to "spam" error message

Hi all, I am trying to register but it seems my IP address is being seen or black listed as a spam address. I get the following message: "Registration denied. Sorry, The UNIX and Linux Forums runs an active policy of not allowing spammers. Please contact us via by posting in this forum if... (0 Replies)
Discussion started by: codenjanod
0 Replies

3. Solaris

How to resolve error "INIT: Cannot stat /etc/inittab, errno: 2"

Hi All, I am getting an error message when I execute command “zlogin -C sunsrv4z5” on my root server. INIT: Cannot stat /etc/inittab, errno: 2 INIT: Cannot stat /etc/inittab, errno: 2 As per my analysis it seems that some files inside /etc folder are deleted. This server was... (14 Replies)
Discussion started by: surbhit4u
14 Replies

4. Shell Programming and Scripting

"Cannot allocate memory" error when run from script

hi in my application there is a line with open(/dev/mydevice,0); it work good when it run manually, but when try to run it within script /////////////////////////////////////////// #!/bin/sh ./device_test 0 0 0 1 1 1 0 0 1 ///////////////////////////////////////// I receive 'Failed... (3 Replies)
Discussion started by: majeed
3 Replies

5. Shell Programming and Scripting

Meaning of "b" modifier in "sort" command

I need to sort the following file by the rhdiskpower devices in the last column: Total_MB Free_MB OS_MB Name Failgroup Library Label UDID Product Redund Path 1024 851 1024 OCRVOT1_0000 OCRVOT1_0000 System UNKNOWN ... (3 Replies)
Discussion started by: wjssj
3 Replies

6. Shell Programming and Scripting

"Syntax Error sometimes due to corruption of variable value "

Hi , I have script as follows , #!/usr/bin/ksh -x if then alias echo="echo -e" fi MAX_ENTRIES=1024 nb_of_entries=`echo "$list_of_entries" | wc -w` # Set number of tables eval nb_of_tables=\`expr `expr $nb_of_entries / $MAX_ENTRIES` + 1 \` # Output the number of tables echo... (6 Replies)
Discussion started by: breezevinay
6 Replies

7. Shell Programming and Scripting

Perl failure with "main::$fn" used only once:" in error logs

Hi all, Can anyone guess why this is happening? I am new to Perl, so please help me to fix this error: - I have a static html file which calls the cgi-perl script. HTML Code: <html> <head> <title> Hey Dude! </title> </head> <body> <form method="POST"... (3 Replies)
Discussion started by: bashily
3 Replies

8. Ubuntu

What is solution for this error "tar: Exiting with failure status due to previous errors"?

Does anyone know what is solution for this error ?tar: Exiting with failure status due to previous errors from last 3 days I am trying to take backup of home/user directory getting again and again same error please anyone give me solution (8 Replies)
Discussion started by: Akshay Hegde
8 Replies

9. UNIX for Beginners Questions & Answers

Extract delta records using with "comm" and "sort" commands combination

Hi All, I have 2 pipe delimited files viz., file_old and file_new. I'm trying to compare these 2 files, and extract all the different rows between them into a new_file. comm -3 < sort file_old < sort file_new > new_file I am getting the below error: -ksh: sort: cannot open But if I do... (7 Replies)
Discussion started by: njny
7 Replies

10. UNIX for Beginners Questions & Answers

Perl: Can someone please explain this code "sort { $a <=> $b } @unsorted"

@sorted = sort { $a <=> $b } @unsorted; I am having hard time understanding how this works? I know the output but interested to know the working. Thanks in advance. (2 Replies)
Discussion started by: Tanu
2 Replies
limit(1)																  limit(1)

NAME
limit, ulimit, unlimit - set or get limitations on the system resources available to the current shell and its descendents SYNOPSIS
/usr/bin/ulimit [-f] [blocks] sh ulimit [ - [HS] [ a | cdfnstv]] ulimit [ - [HS] [ c | d | f | n | s | t | v]] limit csh limit [-h] [ resource [limit]] unlimit [-h] [resource] ksh ulimit [-HSacdfnstv] [limit] /usr/bin/ulimit The ulimit utility sets or reports the file-size writing limit imposed on files written by the shell and its child processes (files of any size may be read). Only a process with appropriate privileges can increase the limit. sh The Bourne shell built-in function, ulimit, prints or sets hard or soft resource limits. These limits are described in getrlimit(2). If limit is not present, ulimit prints the specified limits. Any number of limits may be printed at one time. The -a option prints all lim- its. If limit is present, ulimit sets the specified limit to limit. The string unlimited requests the largest valid limit. Limits may be set for only one resource at a time. Any user may set a soft limit to any value below the hard limit. Any user may lower a hard limit. Only a super-user may raise a hard limit. See su(1M). The -H option specifies a hard limit. The -S option specifies a soft limit. If neither option is specified, ulimit will set both limits and print the soft limit. The following options specify the resource whose limits are to be printed or set. If no option is specified, the file size limit is printed or set. -c maximum core file size (in 512-byte blocks) -d maximum size of data segment or heap (in kbytes) -f maximum file size (in 512-byte blocks) -n maximum file descriptor plus 1 -s maximum size of stack segment (in kbytes) -t maximum CPU time (in seconds) -v maximum size of virtual memory (in kbytes) csh The C-shell built-in function, limit, limits the consumption by the current process or any process it spawns, each not to exceed limit on the specified resource. If limit is omitted, print the current limit; if resource is omitted, display all limits. -h Use hard limits instead of the current limits. Hard limits impose a ceiling on the values of the current limits. Only the privi- leged user may raise the hard limits. resource is one of: cputime Maximum CPU seconds per process. filesize Largest single file allowed. Limited to the size of the filesystem (see df(1M)). datasize The maximum size of a process's heap in kilobytes. stacksize Maximum stack size for the process. The default stack size is 2**64. coredumpsize Maximum size of a core dump (file). This is limited to the size of the filesystem. descriptors Maximum number of file descriptors. Run the sysdef(1M) command to obtain the maximum possible limits for your system. The values reported are in hexadecimal, but can be translated into decimal numbers using the bc(1) command. memorysize Maximum size of virtual memory. limit is a number, with an optional scaling factor, as follows: nh Hours (for cputime). nk n kilobytes. This is the default for all but cputime. nm n megabytes or minutes (for cputime). mm:ss Minutes and seconds (for cputime). unlimit removes a limitation on resource. If no resource is specified, then all resource limitations are removed. See the description of the limit command for the list of resource names. -h Remove corresponding hard limits. Only the privileged user may do this. ksh The Korn shell built-in function, ulimit, sets or displays a resource limit. The available resources limits are listed below. Many systems do not contain one or more of these limits. The limit for a specified resource is set when limit is specified. The value of limit can be a number in the unit specified below with each resource, or the value unlimited. The -H and -S flags specify whether the hard limit or the soft limit for the given resource is set. A hard limit cannot be increased once it is set. A soft limit can be increased up to the value of the hard limit. If neither the -H or -S options is specified, the limit applies to both. The current resource limit is printed when limit is omitted. In this case, the soft limit is printed unless -H is specified. When more than one resource is specified, then the limit name and unit is printed before the value. -a Lists all of the current resource limits. -c The number of 512-byte blocks on the size of core dumps. -d The number of K-bytes on the size of the data area. -f The number of 512-byte blocks on files written by child processes (files of any size may be read). -n The number of file descriptors plus 1. -s The number of K-bytes on the size of the stack area. -t The number of seconds (CPU time) to be used by each process. -v The number of K-bytes for virtual memory. If no option is given, -f is assumed. Per-Shell Memory Parameters The heapsize, datasize, and stacksize parameters are not system tunables. The only controls for these are hard limits, set in a shell startup file, or system-wide soft limits, which, for the current version of the Solaris OS, is 2**64bytes. The following option is supported by ulimit: -f Sets (or reports, if no blocks operand is present), the file size limit in blocks. The -f option is also the default case. The following operand is supported by ulimit: blocks The number of 512-byte blocks to use as the new file size limit. /usr/bin/ulimit Example 1: Limiting the Stack Size The following example limits the stack size to 512 kilobytes: example% ulimit -s 512 example% ulimit -a time(seconds) unlimited file(blocks) 100 data(kbytes) 523256 stack(kbytes) 512 coredump(blocks) 200 nofiles(descriptors) 64 memory(kbytes) unlimited sh/ksh Example 2: Limiting the Number of File Descriptors The following command limits the number of file descriptors to 12: example$ ulimit -n 12 example$ ulimit -a time(seconds) unlimited file(blocks) 41943 data(kbytes) 523256 stack(kbytes) 8192 coredump(blocks) 200 nofiles(descriptors) 12 vmemory(kbytes) unlimited csh Example 3: Limiting the Core Dump File Size The following command limits the size of a core dump file size to 0 kilobytes: example% limit coredumpsize 0 example% limit cputime unlimited filesize unlimited datasize 523256 kbytes stacksize 8192 kbytes coredumpsize 0 kbytes descriptors 64 memorysize unlimited Example 4: Removing the limitation for core file size The following command removes the above limitation for the core file size: example% unlimit coredumpsize example% limit cputime unlimited filesize unlimited datasize 523256 kbytes stacksize 8192 kbytes coredumpsize unlimited descriptors 64 memorysize unlimited See environ(5) for descriptions of the following environment variables that affect the execution of ulimit: LANG, LC_ALL, LC_CTYPE, LC_MES- SAGES, and NLSPATH. The following exit values are returned by ulimit: 0 Successful completion. >0 A request for a higher limit was rejected or an error occurred. See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ |Interface Stability |Standard | +-----------------------------+-----------------------------+ bc(1), csh(1), ksh(1), sh(1), df(1M), su(1M), swap(1M), sysdef(1M), getrlimit(2), attributes(5), environ(5), standards(5) 19 Aug 2005 limit(1)
All times are GMT -4. The time now is 07:38 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy