Sponsored Content
Operating Systems Linux shmat() Failure While Using a Large Amount of Shared Memory Post 302167586 by theicarusagenda on Thursday 14th of February 2008 11:56:46 PM
Old 02-15-2008
Question shmat() Failure While Using a Large Amount of Shared Memory

Hi,

I'm developing a data processing pipeline with multiple stages, with data being moved between the stages using shared memory segments. The size of the data is typically of the order of hundreds of megabytes, and there are typically a few tens of main shared memory segments each of size around 10MB, and a few other shared memory segments of total size less than 1MB.

The problem I'm facing is that after around 600MB has been created in various shared memory segments, further shmat() fails, throwing the error 'Cannot allocate memory'. I tried various combinations of shared memory sizes, and the results are tabulated below.

Code:
----------------------------------------------------------
Size of individual segment | Total allocated shared memory
----------------------------------------------------------
      10485760 (10MB)      |      639635428 (~ 610MB)
      15728640 (15MB)      |      644877668 (~ 615MB)
      31457280 (30MB)      |      660605700 (~ 630MB)
----------------------------------------------------------

The system limits regarding shared memory as defined in the directory /proc/sys/kernel are as follows:

Code:
shmall = 2097152
shmmax = 33554432
shmmni = 4096

The contents of /proc/swaps is as follows:

Code:
Filename      Type            Size    Used    Priority
/dev/sda7     partition       3903752 34792   -1

This program used to work on a different PC which I was using earlier, but not on the current one. I'm running Kubuntu 7.10 on an Intel Core 2 Duo machine with 4GB of RAM.

Can anyone help me figure out where the problem is? Appreciate any help!

Thanks,

Jayanth

Last edited by theicarusagenda; 02-18-2008 at 07:03 AM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to get amount of memory installed.

Hi! I'm not a UNIX fanatic but I like using it for the Oracle database since it's not stable in the Windows NT environment (what is?). Problem: Is there any command to show me the amount of installed physical-memory in the machine? Is there some other way to show the processes which uses... (4 Replies)
Discussion started by: elgholm
4 Replies

2. UNIX for Dummies Questions & Answers

Amount of RAM (Memory)

How do i check/get the total amount of RAM, on a machine running UNIX? (4 Replies)
Discussion started by: samudimu
4 Replies

3. UNIX for Dummies Questions & Answers

amount of memory in my server

Hi all, is there any command that i can execute to find out the size of the harddisk and RAM on my server? Because i am not the SA, i do not have access to such infomation. (2 Replies)
Discussion started by: new2ss
2 Replies

4. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies

5. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

6. Shell Programming and Scripting

How to tar large amount of files?

Hello I have the following files VOICE_hhhh SUBSCR_llll DEL_kkkk Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files When i try to tar these files using tar -cvf backup.tar VOICE* SUBSCR* DEL* i get the error: ksh: /usr/bin/tar: arg list too long How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies

7. SuSE

shmat failed due to Cannot allocate memory

Hi, My program was running for a whole night. after 12 hours i got an error message "Cannot allocate memory" during the shmat commmand. So can you please let me know what could be the reason? is there any solution? thanks in advance. Regards, Mano (5 Replies)
Discussion started by: ManoharanMani
5 Replies

8. Programming

Shared library with acces to shared memory.

Hello. I am new to this forum and I would like to ask for advice about low level POSIX programming. I have to implement a POSIX compliant C shared library. A file will have some variables and the shared library will have some functions which need those variables. There is one special... (5 Replies)
Discussion started by: iamjag
5 Replies

9. Shell Programming and Scripting

Perl : Large amount of data put into an array

This basic code works. I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in". It... (5 Replies)
Discussion started by: sumguy
5 Replies

10. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies
IPCS(1) 						    BSD General Commands Manual 						   IPCS(1)

NAME
ipcs -- report System V interprocess communication facilities status SYNOPSIS
ipcs [-abcmopqstMQST] [-C system] [-N core] DESCRIPTION
The ipcs program provides information on System V interprocess communication (IPC) facilities on the system. The options are as follows: -a Show the maximum amount of information possible when displaying active semaphores, message queues, and shared memory segments. (This is shorthand for specifying the -b, -c, -o, -p, and -t options.) -b Show the maximum allowed sizes for active semaphores, message queues, and shared memory segments. The ``maximum allowed size'' is the maximum number of bytes in a message on a message queue, the size of a shared memory segment, or the number of semaphores in a set of semaphores. -c Show the creator's name and group for active semaphores, message queues, and shared memory segments. -m Display information about active shared memory segments. -o Show outstanding usage for active message queues, and shared memory segments. The ``outstanding usage'' is the number of messages in a message queue, or the number of processes attached to a shared memory segment. -p Show the process ID information for active semaphores, message queues, and shared memory segments. The ``process ID information'' is the last process to send a message to or receive a message from a message queue, the process that created a semaphore, or the last process to attach or detach a shared memory segment. -q Display information about active message queues. -s Display information about active semaphores. -t Show access times for active semaphores, message queues, and shared memory segments. The access times is the time of the last con- trol operation on an IPC object, the last send or receive of a message, the last attach or detach of a shared memory segment, or the last operation on a semaphore. -C system Extract the name list from the specified system instead of the default ``/netbsd''. -M Display system information about shared memory. -N core Extract values associated with the name list from the specified core instead of the default ``/dev/kmem''. and semaphores. -Q Display system information about messages queues. -S Display system information about semaphores. -T Display system information about shared memory, message queues and semaphores. (This is shorthand for specifying the -M, -Q, and -S options.) If none of the -M, -m, -Q, -q, -S, -s, or -T options are specified, information about all active IPC facilities is listed. RESTRICTIONS
System data structures may change while ipcs is running; the output of ipcs is not guaranteed to be consistent. FILES
/dev/kmem default kernel memory /netbsd default system name list SEE ALSO
ipcrm(1), shmat(2), shmctl(2), shmdt(2), shmget(2) AUTHORS
Thorsten Lockert <tholo@sigmasoft.com> BUGS
This manual page is woefully incomplete, because it does not at all attempt to explain the information printed by ipcs. BSD
March 21, 2004 BSD
All times are GMT -4. The time now is 09:24 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy