How to deal with lots of data in memory in order not to run out of memory
Hi,
I'm trying to learn how to manage memory when I have to deal with lots of data.
Basically I'm indexing a huge file (5GB, but it can be bigger), by creating tables that
holds offset <-> startOfSomeData information. Currently I'm mapping the whole file at
once (yep!) but of course the application quickly run outs of memory and malloc'ing
a new table fails after a bit.
My first question, which is more a request for confirmation, is the following.
Does the mapped file counts in memory usage? I'd say yes at 99,9% but I'd like
to be sure.
Second I'd like to know what syscall are available in order to retrieve memory
information of the calling process (how much memory used, how much left etc..) ?
I'm of course going to map a few pages at the time, although it'll be more tricky
to parse the file. Anyway, I'd like to know how I should deal with the tables I create and
I keep in memory. If I dump a few tables to a temporary file, and mmap it for quick access, it'd be the same thing if the answer to my first question is yes. I definitely
would not want the kernel to start swapping memory, but I'd rather have a thread
that concurrently writes those table to a file, for later retrieve.
Anyhow, my main concern is not to run out of memory (i.e. malloc has not to fail).
Any kind of suggestions for people with more expertise are very welcomed.
I'm eager to learn.
Yes memory mapping counts as part of your process. But it is possible to map a file shared, and have another process actually process the file. That does not solve the memory usage problem necessarily. If you are running out of memory see how ulimit is set.
ulimit example -
The two red lines apply to your question.
getrusage() returns resource usage for a parent process and it's children.
setrusage allows you to change ulimit soft values - you cannot go beyond the hard limits unless the sysadmin reconfigures your account/kernel.
Also you can increase virtual memory simply by adding swap space. Virtual memory (if ulimit for memory == unlimited) is the actual limit for process memory space.
I am new to AIX, I have few AIX 5.3 servers and I could see there are significant difference in paging space utilization on servers even though they are running same applications
below server is working fine which shows 2-5 % paging usage throuh out the day
cpu_scale_memp = 8... (12 Replies)
Hi Experts,
Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing.
Why memory utilization on solaris always looks high?
I have statement about memory on solaris, is this true:
Memory in solaris is used for... (4 Replies)
Hello solaris experts,
Being new to solaris containers, from Linux, feeling difficulty in understanding certain concepts. Hope somebody can help me here.
I understand that, & some questions ....
Locked memory -- memory which will not be swapped out at any cause.
is this for... (0 Replies)
Hello,
I've been reading your forums for quite a while and the great amount of information I find here always come in hand.This time however, I need some specific help...
I have a doubt with an AIX server which I'm failing to understand as I'm new to its concept of memory management...
... (8 Replies)
Is it possible to restrict physical memory in solaris zone with zone.max-locked-memory just like we can do with rcapd ? I do not want to used rcapd (1 Reply)
Hi,
Im working on Solaris 9 on SPARC-32 bit running on an Ultra-80, and I have to find out the following:-
1. Total Physical Memory in the system(total RAM).
2. Available Physical Memory(i.e. RAM Usage)
3. Total (Logical) Memory in the system
4. Available (Logical) Memory.
I know... (4 Replies)
When allocating memory for two different nodes, the resulting memory are the same. Clearly, this will lead to a mistake. This happened in a function. And the process must be in a function.
(gdb) p tree->list
$43 = (node *) 0x8be4180
(gdb) p tree->list
$44 = (node *) 0x8be4180
At the... (2 Replies)
Hi! I am new to HP-UX. :o
By using the command glance, I found the user memory usage was very high. I would like to know is there any command can show the process which consume most available memory ? (Just like the command top, but order by memory, not CPU) (1 Reply)
Hi, I try to marshal a unsigned int and a char * into a buffer, and then unmarshal them later to get them out. I need to put the char * in the front and unsigned int at the end of the buffer. However, my system always give me "BUS ERROR". I am using Sun Sparcs Sloris 2.10.
My code to marshal... (6 Replies)
I am running HP-UX B.11.11.
I'm increasing a parameter for a database engine so that it uses more memory to buffer the disk drive (to speed up performance). I have over 5GB of memory not being used.
But when I try to start the DB with the increased buffer parameter I get told.
"Not... (1 Reply)