Sponsored Content
Operating Systems Linux shmat() Failure While Using a Large Amount of Shared Memory Post 302167586 by theicarusagenda on Thursday 14th of February 2008 11:56:46 PM
Old 02-15-2008
Question shmat() Failure While Using a Large Amount of Shared Memory

Hi,

I'm developing a data processing pipeline with multiple stages, with data being moved between the stages using shared memory segments. The size of the data is typically of the order of hundreds of megabytes, and there are typically a few tens of main shared memory segments each of size around 10MB, and a few other shared memory segments of total size less than 1MB.

The problem I'm facing is that after around 600MB has been created in various shared memory segments, further shmat() fails, throwing the error 'Cannot allocate memory'. I tried various combinations of shared memory sizes, and the results are tabulated below.

Code:
----------------------------------------------------------
Size of individual segment | Total allocated shared memory
----------------------------------------------------------
      10485760 (10MB)      |      639635428 (~ 610MB)
      15728640 (15MB)      |      644877668 (~ 615MB)
      31457280 (30MB)      |      660605700 (~ 630MB)
----------------------------------------------------------

The system limits regarding shared memory as defined in the directory /proc/sys/kernel are as follows:

Code:
shmall = 2097152
shmmax = 33554432
shmmni = 4096

The contents of /proc/swaps is as follows:

Code:
Filename      Type            Size    Used    Priority
/dev/sda7     partition       3903752 34792   -1

This program used to work on a different PC which I was using earlier, but not on the current one. I'm running Kubuntu 7.10 on an Intel Core 2 Duo machine with 4GB of RAM.

Can anyone help me figure out where the problem is? Appreciate any help!

Thanks,

Jayanth

Last edited by theicarusagenda; 02-18-2008 at 07:03 AM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to get amount of memory installed.

Hi! I'm not a UNIX fanatic but I like using it for the Oracle database since it's not stable in the Windows NT environment (what is?). Problem: Is there any command to show me the amount of installed physical-memory in the machine? Is there some other way to show the processes which uses... (4 Replies)
Discussion started by: elgholm
4 Replies

2. UNIX for Dummies Questions & Answers

Amount of RAM (Memory)

How do i check/get the total amount of RAM, on a machine running UNIX? (4 Replies)
Discussion started by: samudimu
4 Replies

3. UNIX for Dummies Questions & Answers

amount of memory in my server

Hi all, is there any command that i can execute to find out the size of the harddisk and RAM on my server? Because i am not the SA, i do not have access to such infomation. (2 Replies)
Discussion started by: new2ss
2 Replies

4. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies

5. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

6. Shell Programming and Scripting

How to tar large amount of files?

Hello I have the following files VOICE_hhhh SUBSCR_llll DEL_kkkk Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files When i try to tar these files using tar -cvf backup.tar VOICE* SUBSCR* DEL* i get the error: ksh: /usr/bin/tar: arg list too long How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies

7. SuSE

shmat failed due to Cannot allocate memory

Hi, My program was running for a whole night. after 12 hours i got an error message "Cannot allocate memory" during the shmat commmand. So can you please let me know what could be the reason? is there any solution? thanks in advance. Regards, Mano (5 Replies)
Discussion started by: ManoharanMani
5 Replies

8. Programming

Shared library with acces to shared memory.

Hello. I am new to this forum and I would like to ask for advice about low level POSIX programming. I have to implement a POSIX compliant C shared library. A file will have some variables and the shared library will have some functions which need those variables. There is one special... (5 Replies)
Discussion started by: iamjag
5 Replies

9. Shell Programming and Scripting

Perl : Large amount of data put into an array

This basic code works. I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in". It... (5 Replies)
Discussion started by: sumguy
5 Replies

10. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies
SHMAT(2)						      BSD System Calls Manual							  SHMAT(2)

NAME
shmat, shmdt -- map/unmap shared memory SYNOPSIS
#include <sys/types.h> #include <sys/ipc.h> #include <sys/shm.h> void * shmat(int shmid, void *shmaddr, int shmflg); int shmdt(void *shmaddr); DESCRIPTION
shmat() maps the shared memory segment associated with the shared memory identifier shmid into the address space of the calling process. The address at which the segment is mapped is determined by the shmaddr parameter. If it is equal to 0, the system will pick an address itself. Otherwise, an attempt is made to map the shared memory segment at the address shmaddr specifies. If SHM_RND is set in shmflg, the system will round the address down to a multiple of SHMLBA bytes (SHMLBA is defined in <sys/shm.h> ). A shared memory segment can be mapped read-only by specifying the SHM_RDONLY flag in shmflg. shmdt() unmaps the shared memory segment that is currently mapped at shmaddr from the calling process' address space. shmaddr must be a value returned by a prior shmat() call. A shared memory segment will remain existant until it is removed by a call to shmctl(2) with the IPC_RMID command. RETURN VALUES
shmat() returns the address at which the shared memory segment has been mapped into the calling process' address space when successful, shmdt() returns 0 on successful completion. Otherwise, a value of -1 is returned, and the global variable errno is set to indicate the error. ERRORS
shmat() will fail if: [EACCES] The calling process has no permission to access this shared memory segment. [ENOMEM] There is not enough available data space for the calling process to map the shared memory segment. [EINVAL] shmid is not a valid shared memory identifier. shmaddr specifies an illegal address. [EMFILE] The number of shared memory segments has reached the system-wide limit. shmdt() will fail if: [EINVAL] shmaddr is not the start address of a mapped shared memory segment. SEE ALSO
shmctl(2), shmget(2), mmap(2) BSD
August 17, 1995 BSD
All times are GMT -4. The time now is 02:48 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy