Sponsored Content
Top Forums Programming How to deal with lots of data in memory in order not to run out of memory Post 302320520 by emitrax on Thursday 28th of May 2009 08:03:39 AM
Old 05-28-2009
How to deal with lots of data in memory in order not to run out of memory

Hi,

I'm trying to learn how to manage memory when I have to deal with lots of data.

Basically I'm indexing a huge file (5GB, but it can be bigger), by creating tables that
holds offset <-> startOfSomeData information. Currently I'm mapping the whole file at
once (yep!) but of course the application quickly run outs of memory and malloc'ing
a new table fails after a bit.

My first question, which is more a request for confirmation, is the following.
Does the mapped file counts in memory usage? I'd say yes at 99,9% but I'd like
to be sure.

Second I'd like to know what syscall are available in order to retrieve memory
information of the calling process (how much memory used, how much left etc..) ?

I'm of course going to map a few pages at the time, although it'll be more tricky
to parse the file. Anyway, I'd like to know how I should deal with the tables I create and
I keep in memory. If I dump a few tables to a temporary file, and mmap it for quick access, it'd be the same thing if the answer to my first question is yes. I definitely
would not want the kernel to start swapping memory, but I'd rather have a thread
that concurrently writes those table to a file, for later retrieve.

Anyhow, my main concern is not to run out of memory (i.e. malloc has not to fail).

Any kind of suggestions for people with more expertise are very welcomed.
I'm eager to learn.

Thanks,
S.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Shared memory shortage but lots of unused memory

I am running HP-UX B.11.11. I'm increasing a parameter for a database engine so that it uses more memory to buffer the disk drive (to speed up performance). I have over 5GB of memory not being used. But when I try to start the DB with the increased buffer parameter I get told. "Not... (1 Reply)
Discussion started by: cjcamaro
1 Replies

2. Programming

how to round up a memory address(memory alignment problem)

Hi, I try to marshal a unsigned int and a char * into a buffer, and then unmarshal them later to get them out. I need to put the char * in the front and unsigned int at the end of the buffer. However, my system always give me "BUS ERROR". I am using Sun Sparcs Sloris 2.10. My code to marshal... (6 Replies)
Discussion started by: nj302
6 Replies

3. HP-UX

Order process by consumed memory

Hi! I am new to HP-UX. :o By using the command glance, I found the user memory usage was very high. I would like to know is there any command can show the process which consume most available memory ? (Just like the command top, but order by memory, not CPU) (1 Reply)
Discussion started by: alfredo
1 Replies

4. Programming

malloc gives the same memory to two different nodes. How to deal with it?

When allocating memory for two different nodes, the resulting memory are the same. Clearly, this will lead to a mistake. This happened in a function. And the process must be in a function. (gdb) p tree->list $43 = (node *) 0x8be4180 (gdb) p tree->list $44 = (node *) 0x8be4180 At the... (2 Replies)
Discussion started by: cdbug
2 Replies

5. Solaris

How to find Total and Free Physical Memory and Logical Memory in SOLARIS 9

Hi, Im working on Solaris 9 on SPARC-32 bit running on an Ultra-80, and I have to find out the following:- 1. Total Physical Memory in the system(total RAM). 2. Available Physical Memory(i.e. RAM Usage) 3. Total (Logical) Memory in the system 4. Available (Logical) Memory. I know... (4 Replies)
Discussion started by: 0ktalmagik
4 Replies

6. Solaris

restrcit physical memory with zone.max-locked-memory

Is it possible to restrict physical memory in solaris zone with zone.max-locked-memory just like we can do with rcapd ? I do not want to used rcapd (1 Reply)
Discussion started by: fugitive
1 Replies

7. AIX

Lots of page faults and free memory

Hello, I've been reading your forums for quite a while and the great amount of information I find here always come in hand.This time however, I need some specific help... I have a doubt with an AIX server which I'm failing to understand as I'm new to its concept of memory management... ... (8 Replies)
Discussion started by: flpgdt
8 Replies

8. Solaris

relationship or difference between entitled memory and locked memory

Hello solaris experts, Being new to solaris containers, from Linux, feeling difficulty in understanding certain concepts. Hope somebody can help me here. I understand that, & some questions .... Locked memory -- memory which will not be swapped out at any cause. is this for... (0 Replies)
Discussion started by: thegeek
0 Replies

9. Solaris

[DOUBT] Memory high in idle process on Solaris 10 (Memory Utilization > 90%)

Hi Experts, Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing. Why memory utilization on solaris always looks high? I have statement about memory on solaris, is this true: Memory in solaris is used for... (4 Replies)
Discussion started by: edydsuranta
4 Replies

10. AIX

High Paging when lots of free memory AIX 5.3

I am new to AIX, I have few AIX 5.3 servers and I could see there are significant difference in paging space utilization on servers even though they are running same applications below server is working fine which shows 2-5 % paging usage throuh out the day cpu_scale_memp = 8... (12 Replies)
Discussion started by: bibish
12 Replies
KSYMS(4)                                                   BSD Kernel Interfaces Manual                                                   KSYMS(4)

NAME
ksyms -- kernel symbol table interface SYNOPSIS
device ksyms DESCRIPTION
The /dev/ksyms character device provides a read-only interface to a snapshot of the kernel symbol table. The in-kernel symbol manager is designed to be able to handle many types of symbols tables, however, only elf(5) symbol tables are supported by this device. The ELF format image contains two sections: a symbol table and a corresponding string table. Symbol Table The SYMTAB section contains the symbol table entries present in the current running kernel, including the symbol table entries of any loaded modules. The symbols are ordered by the kernel module load time starting with kernel file symbols first, followed by the first loaded module's symbols and so on. String Table The STRTAB section contains the symbol name strings from the kernel and any loaded modules that the symbol table entries refer- ence. Elf formated symbol table data read from the /dev/ksyms file represents the state of the kernel at the time when the device is opened. Since /dev/ksyms has no text or data, most of the fields are initialized to NULL. The ksyms driver does not block the loading or unloading of mod- ules into the kernel while the /dev/ksyms file is open but may contain stale data. IOCTLS
The ioctl(2) command codes below are defined in <sys/ksyms.h>. The (third) argument to the ioctl(2) should be a pointer to the type indicated. KIOCGSIZE (size_t) Returns the total size of the current symbol table. This can be used when allocating a buffer to make a copy of the kernel symbol table. KIOCGADDR (void *) Returns the address of the kernel symbol table mapped in the process memory. FILES
/dev/ksyms ERRORS
An open(2) of /dev/ksyms will fail if: [EBUSY] The device is already open. A process must close /dev/ksyms before it can be opened again. [ENOMEM] There is a resource shortage in the kernel. [ENXIO] The driver was unsuccessful in creating a snapshot of the kernel symbol table. This may occur if the kernel was in the process of loading or unloading a module. SEE ALSO
ioctl(2), nlist(3), elf(5), kldload(8) HISTORY
A ksyms device exists in many different operating systems. This implementation is similar in function to the Solaris and NetBSD ksyms driver. The ksyms driver first appeared in FreeBSD 8.0 to support lockstat(1). BUGS
Because files can be dynamically linked into the kernel at any time the symbol information can vary. When you open the /dev/ksyms file, you have access to an ELF image which represents a snapshot of the state of the kernel symbol information at that instant in time. Keeping the device open does not block the loading or unloading of kernel modules. To get a new snapshot you must close and re-open the device. A process is only allowed to open the /dev/ksyms file once at a time. The process must close the /dev/ksyms before it is allowed to open it again. The ksyms driver uses the calling process' memory address space to store the snapshot. ioctl(2) can be used to get the memory address where the symbol table is stored to save kernel memory. mmap(2) may also be used but it will map it to another address. AUTHORS
The ksyms driver was written by Stacey Son <sson@freebsd.org>. BSD April 5, 2009 BSD
All times are GMT -4. The time now is 01:27 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy