shmat() Failure While Using a Large Amount of Shared Memory

Thread Tools Search this Thread
Operating Systems Linux shmat() Failure While Using a Large Amount of Shared Memory
# 1  
Old 02-15-2008
Question shmat() Failure While Using a Large Amount of Shared Memory


I'm developing a data processing pipeline with multiple stages, with data being moved between the stages using shared memory segments. The size of the data is typically of the order of hundreds of megabytes, and there are typically a few tens of main shared memory segments each of size around 10MB, and a few other shared memory segments of total size less than 1MB.

The problem I'm facing is that after around 600MB has been created in various shared memory segments, further shmat() fails, throwing the error 'Cannot allocate memory'. I tried various combinations of shared memory sizes, and the results are tabulated below.

Size of individual segment | Total allocated shared memory
      10485760 (10MB)      |      639635428 (~ 610MB)
      15728640 (15MB)      |      644877668 (~ 615MB)
      31457280 (30MB)      |      660605700 (~ 630MB)

The system limits regarding shared memory as defined in the directory /proc/sys/kernel are as follows:

shmall = 2097152
shmmax = 33554432
shmmni = 4096

The contents of /proc/swaps is as follows:

Filename      Type            Size    Used    Priority
/dev/sda7     partition       3903752 34792   -1

This program used to work on a different PC which I was using earlier, but not on the current one. I'm running Kubuntu 7.10 on an Intel Core 2 Duo machine with 4GB of RAM.

Can anyone help me figure out where the problem is? Appreciate any help!



Last edited by theicarusagenda; 02-18-2008 at 08:03 AM..
# 2  
Old 02-21-2008
1, shmat() is deprecated SYSV legacy use shm_open()

2. shmat() is done by the kernel itself fill a bugreport for the linux kernel mailing list if you thing it is not your program that causes the problem
# 3  
Old 02-25-2008
Thanks for the suggestions, grumpf!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: - GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/ (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies

2. Shell Programming and Scripting

Perl : Large amount of data put into an array

This basic code works. I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in". It... (5 Replies)
Discussion started by: sumguy
5 Replies

3. Programming

Shared library with acces to shared memory.

Hello. I am new to this forum and I would like to ask for advice about low level POSIX programming. I have to implement a POSIX compliant C shared library. A file will have some variables and the shared library will have some functions which need those variables. There is one special... (5 Replies)
Discussion started by: iamjag
5 Replies

4. SuSE

shmat failed due to Cannot allocate memory

Hi, My program was running for a whole night. after 12 hours i got an error message "Cannot allocate memory" during the shmat commmand. So can you please let me know what could be the reason? is there any solution? thanks in advance. Regards, Mano (5 Replies)
Discussion started by: ManoharanMani
5 Replies

5. Shell Programming and Scripting

How to tar large amount of files?

Hello I have the following files VOICE_hhhh SUBSCR_llll DEL_kkkk Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files When i try to tar these files using tar -cvf backup.tar VOICE* SUBSCR* DEL* i get the error: ksh: /usr/bin/tar: arg list too long How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies

6. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

7. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies

8. UNIX for Dummies Questions & Answers

amount of memory in my server

Hi all, is there any command that i can execute to find out the size of the harddisk and RAM on my server? Because i am not the SA, i do not have access to such infomation. (2 Replies)
Discussion started by: new2ss
2 Replies

9. UNIX for Dummies Questions & Answers

Amount of RAM (Memory)

How do i check/get the total amount of RAM, on a machine running UNIX? (4 Replies)
Discussion started by: samudimu
4 Replies

10. UNIX for Dummies Questions & Answers

How to get amount of memory installed.

Hi! I'm not a UNIX fanatic but I like using it for the Oracle database since it's not stable in the Windows NT environment (what is?). Problem: Is there any command to show me the amount of installed physical-memory in the machine? Is there some other way to show the processes which uses... (4 Replies)
Discussion started by: elgholm
4 Replies
Login or Register to Ask a Question