About efficiency of parallel memory allocation


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users About efficiency of parallel memory allocation
# 1  
Old 11-01-2017
Power About efficiency of parallel memory allocation

Hello, there.
I'm a new beginner to Linux kernel and curious about its memory management.
When multiple applications apply for memory space at the same time, how Linux kernel solve the resource contending problem for high performance?
I have known that there is a buddy system for allocating and recalling memory for big page and there is a slab allocator for little buffers.
Is the performance guaranteed by multi-layer memory management architecture?
Wish for some experienced help. Smilie
This User Gave Thanks to blackwall For This Post:
# 2  
Old 11-01-2017
Do you understand virtual memory versus physical memory? When a process asks for memory, especially huge chunks of it, one common strategy is to allocate a small part of it and then mark virtual memory (which is on disk usually) as allocated for the rest of the request.

See
Code:
free
swapon -s

# 3  
Old 11-01-2017
Quote:
Originally Posted by jim mcnamara
Do you understand virtual memory versus physical memory? When a process asks for memory, especially huge chunks of it, one common strategy is to allocate a small part of it and then mark virtual memory (which is on disk usually) as allocated for the rest of the request.

See
Code:
free
swapon -s

Thanks for replying.
I have learned about different mapping methods of virtual memory to physical memory.

But whole physical memory is divided into two parts: one for Kernel, the other for User and usually 1:3.
For user part, user threads or processes share the physical memory by virtual memory mapping, so it means that each thread gets it memory without contention in the allocation even though they actually share the same physical memory? (I know the conflict can happen when they load their virtual memory segment to the physical memory area)
For kernel part, there is part of memory(For 4GB memory, it's 896MB) that can be used directly without mapping from virtual memory to physical memory. If parallel allocation happens in this area(such as parallel system call), will contention exist? If yes, how does it solve the performance problem?
Thanks Smilie
# 4  
Old 11-02-2017
First off, virtual memory mapping does not necessarily mean "contention free". Kernel memory is virtual too. There's really no difference between kernel and user memory except who gets to use it.

The two ways a process asks the kernel for address space -- brk() and mmap() -- are unavoidably high-overhead. Every kernel call is a task-switch, after all, even mundane things like read()! brk() and mmap() aren't generally bottlenecks, though, because they're called so seldom -- a handful of times to set up libraries, then a call to brk() to assign a large block of undifferentiated memory that the process can use as it pleases without bothering the kernel. You may know this block as the heap segment.

The library function malloc() carves out chunks of the heap segment on demand. It keeps keeps a list of available chunks at the base of the heap. brk() expands the heap in large jumps if it runs out, otherwise the kernel isn't involved at all. Whether there's any contention depends on how malloc() is written.

Most Linux systems use GNU libc, which avoids malloc() contention by using several independent "arenas". I think this amounts to independent heaps.

Now, inside the kernel, there's two basic options for allocating memory depending on your needs - kmalloc() and vmalloc().

kmalloc() is the kernel's cousin of userspace malloc() - more or less a heap kept inside kernel space. It's fast, doesn't change the page table, and lets you avoid contention if absolutely necessary. But you usually can't allocate more than 4 megs at a time(architecture dependent), and the total limit on size might also be as small as 128 megs.

The other is vmalloc(), which alters the page table and forces a synchronization but doesn't have a lot of limits otherwise, since it's not restricted to a chunk of preallocated kernel memory.

Last edited by Corona688; 11-02-2017 at 05:48 PM..
These 2 Users Gave Thanks to Corona688 For This Post:
# 5  
Old 11-03-2017
Quote:
Originally Posted by Corona688
First off, virtual memory mapping does not necessarily mean "contention free". Kernel memory is virtual too. There's really no difference between kernel and user memory except who gets to use it.

The two ways a process asks the kernel for address space -- brk() and mmap() -- are unavoidably high-overhead. Every kernel call is a task-switch, after all, even mundane things like read()! brk() and mmap() aren't generally bottlenecks, though, because they're called so seldom -- a handful of times to set up libraries, then a call to brk() to assign a large block of undifferentiated memory that the process can use as it pleases without bothering the kernel. You may know this block as the heap segment.

The library function malloc() carves out chunks of the heap segment on demand. It keeps keeps a list of available chunks at the base of the heap. brk() expands the heap in large jumps if it runs out, otherwise the kernel isn't involved at all. Whether there's any contention depends on how malloc() is written.

Most Linux systems use GNU libc, which avoids malloc() contention by using several independent "arenas". I think this amounts to independent heaps.

Now, inside the kernel, there's two basic options for allocating memory depending on your needs - kmalloc() and vmalloc().

kmalloc() is the kernel's cousin of userspace malloc() - more or less a heap kept inside kernel space. It's fast, doesn't change the page table, and lets you avoid contention if absolutely necessary. But you usually can't allocate more than 4 megs at a time(architecture dependent), and the total limit on size might also be as small as 128 megs.

The other is vmalloc(), which alters the page table and forces a synchronization but doesn't have a lot of limits otherwise, since it's not restricted to a chunk of preallocated kernel memory.
Very thanks for the detailed explanation!
Now I know the main idea of system's memory management. Costly operations are completed mostly in creating new processes or allocating more memory, but general cheap operations are completed in each process. It ensures that kernel will not be frequently bothered.
Interesting, I have to read more about the kernel.
Thanks again!
This User Gave Thanks to blackwall For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Memory allocation problem

I am using ubuntu. I have written a program to calculate prime factors. it works perfectly fine till entered number is less than 9989 (or so ) but when one enters a number higher than that, for example 15000, it does not work. Can anyone guide me whats the problem ? although new codes are welcome,... (2 Replies)
Discussion started by: Abhishek_kumar
2 Replies

2. Programming

memory allocation for string in C

hi in the following code, how the memory is allocated for a1 which holds the values of a2 after cpy function call. #include <stdio.h> #include <string.h> void cpy(char* d, const char* s){ while(*d++=*s++); } main(){ char* a1; char* a2="done"; cpy(a1,a2); ... (3 Replies)
Discussion started by: mprakasheee
3 Replies

3. Programming

Dynamic Memory Allocation

Hello Guys I have a small confusion in the dynamic memory allocation concept. If we declare a pointer say a char pointer, we need to allocate adequate memory space. char* str = (char*)malloc(20*sizeof(char)); str = "This is a string"; But this will also work. char* str = "This... (2 Replies)
Discussion started by: tene
2 Replies

4. Programming

Memory allocation in C

Hi Experts I need some help in static memory allocation in C. I have a program in which I declared 2 variables, one char array and one integer. I was little surprised to see the addresses of the variables. First: int x; char a; printf("%u %u\n', &x, a); I got the addresses displayed... (2 Replies)
Discussion started by: unx_freak
2 Replies

5. Programming

Is there a problem with the memory allocation???

I have a scenario like the client has to search for the active server.There will be many servers.But not all server are active.And at a time not more than one server will be active. The client will be in active state always i.e, it should always search for an active server until it gets one.I... (1 Reply)
Discussion started by: vigneshinbox
1 Replies

6. Programming

Memory allocation problem

I have a program that will fetch some particular lines and store it in a buffer for further operations.The code which is given below works but with some errors.I couldn't trace out the error.Can anybody help on this plz?? #include <stdio.h> #include <stdlib.h> #include<string.h> #define... (1 Reply)
Discussion started by: vigneshinbox
1 Replies

7. Programming

Dynamic memory allocation

Hi, I am trying to process line by line of a file. But I should not be allocating static allocation for reading the contents of the file. The memory should be dynamically allocated. The confusion here is how do I determine the size of each line, put it into a buffer with the memory allocated... (11 Replies)
Discussion started by: naan
11 Replies

8. UNIX for Dummies Questions & Answers

HP-UX memory usage allocation

Hi all, I have a HP-UX Server with 4 gigabytes of physical RAM. When I use the 'Glance' utility to see what my memory utilization is, my memory usage shows up maxed out at 99%. I shut off all the known processes that I'm running on that box and the memory utilization is still at 78% (with Swap... (1 Reply)
Discussion started by: dehuang83
1 Replies

9. UNIX for Advanced & Expert Users

threads and memory allocation

Hello! First of all, forgive me for bad English. When I starts new thread (pthread_create), system allocates some memory for it (for example, for thread's stack). I wonder when does it deallocate this memory? The problem is that I have a program which sometimes creates new threads and sometimes... (3 Replies)
Discussion started by: prankster
3 Replies

10. UNIX for Dummies Questions & Answers

memory allocation

I would like to know how I could allocate some more memory to a process. Please note that I am not the root user. (1 Reply)
Discussion started by: sagar
1 Replies
Login or Register to Ask a Question