Sponsored Content
Operating Systems Linux shmat() Failure While Using a Large Amount of Shared Memory Post 302167586 by theicarusagenda on Thursday 14th of February 2008 11:56:46 PM
Old 02-15-2008
Question shmat() Failure While Using a Large Amount of Shared Memory

Hi,

I'm developing a data processing pipeline with multiple stages, with data being moved between the stages using shared memory segments. The size of the data is typically of the order of hundreds of megabytes, and there are typically a few tens of main shared memory segments each of size around 10MB, and a few other shared memory segments of total size less than 1MB.

The problem I'm facing is that after around 600MB has been created in various shared memory segments, further shmat() fails, throwing the error 'Cannot allocate memory'. I tried various combinations of shared memory sizes, and the results are tabulated below.

Code:
----------------------------------------------------------
Size of individual segment | Total allocated shared memory
----------------------------------------------------------
      10485760 (10MB)      |      639635428 (~ 610MB)
      15728640 (15MB)      |      644877668 (~ 615MB)
      31457280 (30MB)      |      660605700 (~ 630MB)
----------------------------------------------------------

The system limits regarding shared memory as defined in the directory /proc/sys/kernel are as follows:

Code:
shmall = 2097152
shmmax = 33554432
shmmni = 4096

The contents of /proc/swaps is as follows:

Code:
Filename      Type            Size    Used    Priority
/dev/sda7     partition       3903752 34792   -1

This program used to work on a different PC which I was using earlier, but not on the current one. I'm running Kubuntu 7.10 on an Intel Core 2 Duo machine with 4GB of RAM.

Can anyone help me figure out where the problem is? Appreciate any help!

Thanks,

Jayanth

Last edited by theicarusagenda; 02-18-2008 at 07:03 AM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to get amount of memory installed.

Hi! I'm not a UNIX fanatic but I like using it for the Oracle database since it's not stable in the Windows NT environment (what is?). Problem: Is there any command to show me the amount of installed physical-memory in the machine? Is there some other way to show the processes which uses... (4 Replies)
Discussion started by: elgholm
4 Replies

2. UNIX for Dummies Questions & Answers

Amount of RAM (Memory)

How do i check/get the total amount of RAM, on a machine running UNIX? (4 Replies)
Discussion started by: samudimu
4 Replies

3. UNIX for Dummies Questions & Answers

amount of memory in my server

Hi all, is there any command that i can execute to find out the size of the harddisk and RAM on my server? Because i am not the SA, i do not have access to such infomation. (2 Replies)
Discussion started by: new2ss
2 Replies

4. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies

5. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

6. Shell Programming and Scripting

How to tar large amount of files?

Hello I have the following files VOICE_hhhh SUBSCR_llll DEL_kkkk Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files When i try to tar these files using tar -cvf backup.tar VOICE* SUBSCR* DEL* i get the error: ksh: /usr/bin/tar: arg list too long How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies

7. SuSE

shmat failed due to Cannot allocate memory

Hi, My program was running for a whole night. after 12 hours i got an error message "Cannot allocate memory" during the shmat commmand. So can you please let me know what could be the reason? is there any solution? thanks in advance. Regards, Mano (5 Replies)
Discussion started by: ManoharanMani
5 Replies

8. Programming

Shared library with acces to shared memory.

Hello. I am new to this forum and I would like to ask for advice about low level POSIX programming. I have to implement a POSIX compliant C shared library. A file will have some variables and the shared library will have some functions which need those variables. There is one special... (5 Replies)
Discussion started by: iamjag
5 Replies

9. Shell Programming and Scripting

Perl : Large amount of data put into an array

This basic code works. I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in". It... (5 Replies)
Discussion started by: sumguy
5 Replies

10. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies
shmmni(5)							File Formats Manual							 shmmni(5)

NAME
shmmni - number of System V shared memory segment identifiers in the system VALUES
Default Allowed values DESCRIPTION
Shared memory is an efficient InterProcess Communications (IPC) mechanism. One process creates a shared memory segment and attaches it to its address space. Any processes looking to communicate with this process through the shared memory segment then attach the shared memory segment to their corresponding address spaces as well. Once attached, a process can read from or write to the segment depending on the permissions specified while attaching it. This tunable effectively sets the number of unique segments creatable system wide, since each segment is assigned an identifier by the ker- nel. The identifier is simply a reference generated by the kernel such that any user process can request a particular segment for sharing with a simple integer, and let the kernel determine which segment this corresponds to. Who is Expected to Change This Tunable? Anyone. Restrictions on Changing Changes to this tunable take effect immediately. Attempting to lower below the current number of identifiers in use or below the current value of will result in an error message. Attempts to raise the value of when insufficient memory is available to the kernel to create the needed structures will result in an error message. When Should the Value of This Tunable Be Raised? should be raised if users of System V shared memory are receiving the error message on calls. What Are the Side Effects of Raising the Value? Kernel memory usage will be slightly increased, as the data structures used to track the segments are allocated based on this tunable. When Should the Value of This Tunable Be Lowered? If kernel memory is at a premium, or it is known that few segments will be needed, a slight savings can be gained from decreasing this tun- able, and thus decreasing the data structure memory usage associated with it. What Are the Side Effects of Lowering the Value? Kernel memory usage will be slightly reduced. What Other Tunable Values Should Be Changed at the Same Time? and should be considered. should be changed in the same manner as since lowering the total number of segments but raising the number available per process only makes sense if you want a few processes taking all the segments. is more complex and any changes to it really depend on the effect desired. Refer to the shmmax(5) manpage for more information before changing this tunable. WARNINGS
All HP-UX kernel tunable parameters are release specific. This parameter may be removed or have its meaning changed in future releases of HP-UX. Installation of optional kernel software, from HP or other vendors, may cause changes to tunable parameter values. After installation, some tunable parameters may no longer be at the default or recommended values. For information about the effects of installation on tun- able values, consult the documentation for the kernel software being installed. For information about optional kernel software that was factory installed on your system, see at AUTHOR
was developed by HP. SEE ALSO
shmem(5), shmmax(5), shmseg(5). Tunable Kernel Parameters shmmni(5)
All times are GMT -4. The time now is 08:30 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy