Sponsored Content
Full Discussion: Memory release latency issue
Operating Systems Linux Red Hat Memory release latency issue Post 302915967 by Corona688 on Friday 5th of September 2014 01:30:07 PM
Old 09-05-2014
But you are using massive scratch buffers, and the memory has to come from somewhere. Give it away and you might have a hard time getting it back.

How about a mid-way compromise? Let the OS decide which bits to cache. Let the OS decide how much memory can be spared against the competition of other programs. Let the OS do all the heavy lifting. You can do this all while still using what's, to your program, still a huge, contiguous chunk of memory.

Make and keep a huge scratch file. You can map that into memory with mmap() -- memory accesses effectively become file accesses. This takes advantage of the file cache, so the OS will cache any bits of it you're using. The OS will cache recently accessed chunks, and turf infrequently accessed chunks back to disk. This is how many large database applications do I/O. If you have enough memory, you'll end up dealing almost exclusively with memory, not waiting for disk.

Your application will have to wait for bits to load, but it will wait for page-sized chunks, not an entire 8-gigabyte memory alloc. And if you have enough memory, it can cache everything without you having to ask. You can also do fun things like "copy-on-write", meaning that the mapping will do just what it says above for reads, but when you write to any of the memory, it won't bother writing the changes to disk, just keep it in RAM.

You can also optimize things further, warning the OS so it doesn't have to guess. "Could you cache these nearby parts for me since the user's probably going to scroll over them soon? More important than whatever faraway parts you have cached, thanks". That's what posix_madvise can do to mmaped segments.

Last edited by Corona688; 09-05-2014 at 02:57 PM..
This User Gave Thanks to Corona688 For This Post:
 

10 More Discussions You Might Find Interesting

1. Windows & DOS: Issues & Discussions

Memory Issue

Hi There, I have upgraded the DELL poweredge 2600 server memory from 2GB to 4GB. However, the memory only showed at 2GB of utilization. How to make sure that the server is full utilize of 4GB of memory. Is there the Virtual memory need to be reconfigure as this server is run on windows 2000 and... (2 Replies)
Discussion started by: vestro
2 Replies

2. Linux

Memory issue while diff !!!

Hi All Any idea? why am I having this memory issue? perforce@ixca-pforce-bak:/home/p4-demo-root$ diff checkpoint_offline ../p4-demo-root2/checkpoint.1150 diff: memory exhausted Thanks a lot C Saha (0 Replies)
Discussion started by: csaha
0 Replies

3. AIX

Shared memory issue

Hi friends.. Help to solve this issue... Is there any parameter setting to control or limit the size of the shared memory a process can attach for the below specified environment? The man pages says it can attach upto segments of size 2GB. But when our process (which also connects to... (0 Replies)
Discussion started by: sdspawankumar
0 Replies

4. Linux

Virtual Memory issue

Hi all, I was compiling my glibc 2.6.1 source files on a new kernel 2.66.22.6 and it seems that i am running into issues with the Virtual Memory. It displays the error message: virtual memory exhausted: Cannot allocate memory‏ I saw an article on how to adjust the parameters but i can't... (5 Replies)
Discussion started by: scriptingmani
5 Replies

5. Solaris

Locked memory issue

One of our project has exceeded its assigned max-memory-locked by 3 times .. The said project is using around 9 gigs as described by rss parameter in prstat -J .. and the max-project-memory parameter has been defined as 3gigs .. is it normal or we are monitoring the project memory usage in wrong... (2 Replies)
Discussion started by: fugitive
2 Replies

6. AIX

AIX memory issue

Currently server have load while there is no heavy things running, just oracle database/ application server oracle. I don't understand why server have heavy load, 22GB is under buffer, how to clean buffer/memory in AIX load averages: 9.42, 9.43, 9.68; 05:25:08 141 processes: 125 idle, 16... (12 Replies)
Discussion started by: learnbash
12 Replies

7. Solaris

Zone memory issue

We have a zone configured in our X4600 machine with memory capped to 16GB. Most of the time zone is running with high physical memory utilization. It seems from "top" command shows that the command "kernel" is locks 15GB phy. memory and not using swap memory. Whenever we restart the application... (2 Replies)
Discussion started by: rock123
2 Replies

8. AIX

Memory issue

I have a server with 300Gb allocated to it. Some times I observed in topas Comp% 73 and Non comp 35% and client is also 35% and my paging is showing 92%. If my physical memory utilized only 70% then why paging is so high. And what is relation between Comp, noncomp and client? If the memory... (1 Reply)
Discussion started by: powerAIX
1 Replies

9. Red Hat

Memory Issue

I could not find what is consuming the memory, generated DSET reports and NO hardware wise memory issue. 64 GB RAM on a server yet all I could see is a very limited memory available. I am not sure if I am reading this correct or not. I have used free -t -m and cat /proc/meminfo (results below)... (3 Replies)
Discussion started by: rsheikh01
3 Replies

10. Linux

Swap memory issue

Hi, In our production box i can see the Swap space using the below command free total used free shared buffers cached Mem: 65963232 41041084 24922148 0 877160 35936292 -/+ buffers/cache: 4227632 61735600 Swap: 4192880 ... (6 Replies)
Discussion started by: ratheeshjulk
6 Replies
ALLOC_HUGEPAGES(2)					     Linux Programmer's Manual						ALLOC_HUGEPAGES(2)

NAME
alloc_hugepages, free_hugepages - allocate or free huge pages SYNOPSIS
void *alloc_hugepages(int key, void *addr, size_t len, int prot, int flag); int free_hugepages(void *addr); DESCRIPTION
The system calls alloc_hugepages() and free_hugepages() were introduced in Linux 2.5.36 and removed again in 2.5.54. They existed only on i386 and ia64 (when built with CONFIG_HUGETLB_PAGE). In Linux 2.4.20 the syscall numbers exist, but the calls fail with the error ENOSYS. On i386 the memory management hardware knows about ordinary pages (4 KiB) and huge pages (2 or 4 MiB). Similarly ia64 knows about huge pages of several sizes. These system calls serve to map huge pages into the process's memory or to free them again. Huge pages are locked into memory, and are not swapped. The key argument is an identifier. When zero the pages are private, and not inherited by children. When positive the pages are shared with other applications using the same key, and inherited by child processes. The addr argument of free_hugepages() tells which page is being freed: it was the return value of a call to alloc_hugepages(). (The memory is first actually freed when all users have released it.) The addr argument of alloc_hugepages() is a hint, that the kernel may or may not follow. Addresses must be properly aligned. The len argument is the length of the required segment. It must be a multiple of the huge page size. The prot argument specifies the memory protection of the segment. It is one of PROT_READ, PROT_WRITE, PROT_EXEC. The flag argument is ignored, unless key is positive. In that case, if flag is IPC_CREAT, then a new huge page segment is created when none with the given key existed. If this flag is not set, then ENOENT is returned when no segment with the given key exists. RETURN VALUE
On success, alloc_hugepages() returns the allocated virtual address, and free_hugepages() returns zero. On error, -1 is returned, and errno is set appropriately. ERRORS
ENOSYS The system call is not supported on this kernel. FILES
/proc/sys/vm/nr_hugepages Number of configured hugetlb pages. This can be read and written. /proc/meminfo Gives info on the number of configured hugetlb pages and on their size in the three variables HugePages_Total, HugePages_Free, Hugepagesize. CONFORMING TO
These calls are specific to Linux on Intel processors, and should not be used in programs intended to be portable. NOTES
These system calls are gone; they existed only in Linux 2.5.36 through to 2.5.54. Now the hugetlbfs file system can be used instead. Mem- ory backed by huge pages (if the CPU supports them) is obtained by using mmap(2) to map files in this virtual file system. The maximal number of huge pages can be specified using the hugepages= boot parameter. COLOPHON
This page is part of release 3.25 of the Linux man-pages project. A description of the project, and information about reporting bugs, can be found at http://www.kernel.org/doc/man-pages/. Linux 2007-05-31 ALLOC_HUGEPAGES(2)
All times are GMT -4. The time now is 01:45 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy