Memory fragmentation in a Linux settop box


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Memory fragmentation in a Linux settop box
# 1  
Old 07-06-2014
Memory fragmentation in a Linux settop box

Being a moderator at openATV, a forum for Linux settup boxes, I have seen reports, and sometimes am experiencing myself, artefacts during video playback or timeshift.
As the artefacts are non repetetive (rewinding and watching again does not show artefacts), I can exclude a corrupted video source.

We found that each artefact (up to one per minute in average) 100% correlates with an entry in /var/log/messages like

Code:
Jun 29 14:54:54 ventonhdx user.warn kernel: enigma2: page allocation failure: order:5, mode:0xd0
Jun 29 14:54:54 ventonhdx user.warn kernel: Call Trace:
Jun 29 14:54:54 ventonhdx user.warn kernel: [<805ff9d0>] dump_stack+0x8/0x34
Jun 29 14:54:54 ventonhdx user.warn kernel: [<80091f0c>] warn_alloc_failed+0xe4/0x124
Jun 29 14:54:54 ventonhdx user.warn kernel: [<80094690>] __alloc_pages_nodemask+0x434/0x6e8
Jun 29 14:54:54 ventonhdx user.warn kernel: [<800cb6c8>] cache_alloc_refill+0x318/0x8c0
Jun 29 14:54:54 ventonhdx user.warn kernel: [<800cbdc4>] __kmalloc+0x154/0x19c
Jun 29 14:54:54 ventonhdx user.warn kernel: [<800a8900>] memdup_user+0x24/0x94
Jun 29 14:54:54 ventonhdx user.warn kernel: [<8045e198>] dvbdmx_write+0x48/0xd0
Jun 29 14:54:54 ventonhdx user.warn kernel: [<800cf8b0>] vfs_write+0x9c/0x184
Jun 29 14:54:54 ventonhdx user.warn kernel: [<800cfcd8>] sys_write+0x50/0xb0
Jun 29 14:54:54 ventonhdx user.warn kernel: [<8000e928>] stack_done+0x20/0x44
Jun 29 14:54:54 ventonhdx user.warn kernel: Mem-Info:
Jun 29 14:54:54 ventonhdx user.warn kernel: Normal per-cpu:
Jun 29 14:54:54 ventonhdx user.warn kernel: CPU    0: hi:  186, btch:  31 usd:   0
Jun 29 14:54:54 ventonhdx user.warn kernel: CPU    1: hi:  186, btch:  31 usd: 173
Jun 29 14:54:54 ventonhdx user.warn kernel: active_anon:11037 inactive_anon:11111 isolated_anon:0
Jun 29 14:54:54 ventonhdx user.warn kernel:  active_file:4120 inactive_file:24772 isolated_file:0
Jun 29 14:54:54 ventonhdx user.warn kernel:  unevictable:0 dirty:6121 writeback:1050 unstable:0
Jun 29 14:54:54 ventonhdx user.warn kernel:  free:12176 slab_reclaimable:1301 slab_unreclaimable:1821
Jun 29 14:54:54 ventonhdx user.warn kernel:  mapped:997 shmem:69 pagetables:129 bounce:0
Jun 29 14:54:54 ventonhdx user.warn kernel: Normal free:62716kB min:2876kB low:3592kB high:4312kB active_anon:44148kB inactive_anon:44444kB active_file:16480kB inactive_file:85132kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:518144kB mlocked:0kB dirty:
Jun 29 14:54:54 ventonhdx user.warn kernel: lowmem_reserve[]: 0 0
Jun 29 14:54:54 ventonhdx user.warn kernel: Normal: 3575*4kB 4431*8kB 2239*16kB 78*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 88260kB
Jun 29 14:54:54 ventonhdx user.warn kernel: 21111 total pagecache pages
Jun 29 14:54:54 ventonhdx user.warn kernel: 3465 pages in swap cache
Jun 29 14:54:54 ventonhdx user.warn kernel: Swap cache stats: add 4601, delete 1136, find 7/9
Jun 29 14:54:54 ventonhdx user.warn kernel: Free swap  = 14448kB
Jun 29 14:54:54 ventonhdx user.warn kernel: Total swap = 32764kB
Jun 29 14:54:54 ventonhdx user.warn kernel: 131072 pages RAM
Jun 29 14:54:54 ventonhdx user.warn kernel: 58359 pages reserved
Jun 29 14:54:54 ventonhdx user.warn kernel: 14100 pages shared
Jun 29 14:54:54 ventonhdx user.warn kernel: 34321 pages non-shared
Jun 29 14:54:54 ventonhdx user.warn kernel: SLAB: Unable to allocate memory on node 0 (gfp=0xd0)
Jun 29 14:54:54 ventonhdx user.warn kernel:   cache: size-131072, object size: 131072, order: 5
Jun 29 14:54:54 ventonhdx user.warn kernel:   node 0: slabs: 4/4, objs: 4/4, free: 0

This seems to indicate severe memory fragmentation. Although enough total memory is available, the supply of 128k blocks is low.
This is not always the case. After starting the box, or after "echo 3 > /proc/sys/vm/drop_caches", there is a lot of memeory available:
Code:
root@gbquad:~# cat /proc/buddyinfo
Node 0, zone   Normal    133    314    249    919   1558    655    178     23      0      0      1

Within the next few minutes, the caches fill up until approx. 6MB RAM are left. In the "good state", fragmentation is low, note the 4MB segment:
Code:
root@gbquad:~# cat /proc/buddyinfo
Node 0, zone   Normal    232    160      0      0      0      0      0      0      0      0      1

In the "bad state", memory is severely fragmented, resulting in allocation failures and playback artefacts:
Code:
root@gbquad:~# cat /proc/buddyinfo
Node 0, zone   Normal   1409    350     16      1      0      0      0      0      0      0      0

Unfortunately, we have not yet found out what is causing the "bad state". I have so far only seen it after configuring the timeshift buffer to be on a USB stick and then moving it back to the HDD.

We have tried some approaches trying to cure symptoms (not addressing the root cause):

Clearing caches
"echo 3 > /proc/sys/vm/drop_caches" is freeing up memory, and executing this every 3 minutes in a cron job seems to be helpful.
Many Linux users may say that dropping caches is a bad idea. And yes, dropping them and allowing them to fill again in a cyclic manner is definitely a waste of performance, so avoiding or reducing caching from the start would probably be better. In contrast to a Linux PC executing the OS and programs from HDD, these settop boxes never execute code from HDD but from built-in Flash memory, so caching of CPU code is not required. The cache used for video data may be required for "reading ahead" and thus guaranteeing a continuous stream, but data actually may be dropped after playing it. Having said this, this may actually be critical, there is a risk that data is dropped that is just about to be played. I know very little about this and cannot say whether this is an issue. I'm also not sure what other data is being cached, I can see the cache fill up (much more slowly) with timeshift disabled.

Memory compaction
"echo 1 > /proc/sys/vm/compact_memory", (with CONFIG_COMPACTION=y), executed regularly in a cron job may be helpful as well, though I have not been able to test yet whether in the "bad state" the fragmentation is actually improving (on my box, the "bad state" is rare).

Swap
Some users reported an improvement after installing swap on a USB stick. Other experiments show that swap, though installed, is hardly being used, and I'm also a bit concerned about the access time of swap on a USB stick.

I would be grateful for thoughts and hints, espacially about strategies for finding the root cause of the memory fragmentation, knowing that this may be very difficult without detailed knowledge of the settop box internals.
# 2  
Old 07-07-2014
I have made the observation that the Linux kernel has performance hick-ups if there is continuous I/O.
This has to do with buffering all and everything, according to the dogma "every unused byte of memory is wasted memory". If allocation happens too fast, freeing buffers can take a long time.
But the term "memory fragmentation" does not fit here.
All commercial Unix kernels have configurable limits for buffers and caches; and behave smoothly in such a situation (and little slower in others).
In the very past I had a SuSE PAE kernel that even invoked OOM-killer at continuous I/O. The following program, invoked every 5 minutes, helped:
Code:
#!/bin/bash
dropcaches=/proc/sys/vm/drop_caches
meminfo=/proc/meminfo
[ -e $dropcaches ] || exit
lowfree=`
while read key val kb
do
 if [ "$key" = "LowFree:" ]; then
  [ $val -lt 102400 ] && echo $val
  break
 fi
done < $meminfo
`
[ -z "$lowfree" ] && exit
#enable shortly
echo 0 > $dropcaches
sleep 1
#disable buffercache, leave only dentry cache
sync
echo 1 > $dropcaches

---------- Post updated at 06:56 AM ---------- Previous update was at 03:14 AM ----------

In the case of a 64-bit kernel only the following should be called every 5 or 10 minutes:
Code:
#!/bin/bash
dropcaches=/proc/sys/vm/drop_caches
echo 1 > $dropcaches

In addition try to write smaller portions of data to the disk, in /etc/sysctl.conf:
Code:
vm.dirty_background_ratio = 5
vm.dirty_ratio = 15


Last edited by MadeInGermany; 09-14-2014 at 12:28 PM..
# 3  
Old 07-07-2014
I don't think that dropping caches will help much at all. They are already considered "Free", and the non-free stuff remains non-free, so how can dropping caches defragment anything?

What is your kernel?

Do you know if your application uses hugepages?
# 4  
Old 07-07-2014
What's using all that cache?

IO? For streaming large files, caching is irrelevant because you're almost never going to need to replay any one block. Use direct IO and bypass the page cache, and quit thrashing memory.

Last edited by achenle; 07-07-2014 at 04:22 PM..
# 5  
Old 07-07-2014
Quote:
Originally Posted by Corona688
I don't think that dropping caches will help much at all. They are already considered "Free", and the non-free stuff remains non-free, so how can dropping caches defragment anything?

What is your kernel?

Do you know if your application uses hugepages?
The cache cannot be considered free if the mentioned drop_caches takes over 1 minute... please measure yourself!
# 6  
Old 07-08-2014
Thanks for your replies.

@MadeInGermany, such script is certainly better than always dropping caches every few minutes. I may adjust it so that it drops caches based on fragmentation. Still it's a (better) way of trying to fix symptoms only.

@achenle: Yes, it's IO cache, and I share your view that for video streams data should not be cached. Can you give me more details on how a cache can be bypassed, or is this too specific to the application?

@Corona688: It's a 3.3.8 kernel. How would I detect whether hugepages are used? By looking at a defconfig file? Can you give me a keyword?

In the meantime I found that in the "bad state" executing "echo 1 > /proc/sys/vm/compact_memory" only improves fragmentation a little bit. A colleague thinks this is due to many small memory pages having been blocked.

Is there any command for finding out which memory pages have been allocated by which process?
# 7  
Old 07-08-2014
Does your app request physically contiguous pages of heap memory for cache? - Oracle does that. This triggers the onset of heap (memory) fragmentation especially quickly in a NUMA environment - i.e., most modern commodity cpus. Multicore x86, for example. It also causes system performance problems, by requiring the core to access memory increasingly inefficiently, all the way across the bridge via another core.

This article discusses ways to invoke memory compaction, and among other things, deals with memory fragmentation. It may not help if your app does something verging on the unreasonable like the oracle db engine does. Also note: java apps that use huge heap sizes may force the jvm to reallocate memory and cause memory fragmentation for the same reason oracle does. Usually this is controllable with java parameter settings for max jvm heap size.

Memory compaction [LWN.net]
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. How to Post in the The UNIX and Linux Forums

Copying , renaming the file from windox box and ftp to Linux box

Hello my dear friends, Two file are auto generated from mon - fri at different directories on same windows box.Every day i have to copy the file, rename it (specific name)and ftp it to linux box specified directory. is it possible to automate this process,If yes this has to be done from windows... (1 Reply)
Discussion started by: umesh yadav
1 Replies

2. Red Hat

How to access redhat Linux box graphically from windows box?

Hi I have a linux box and need to access from windows graphically # uname -a Linux pc-l416116 2.6.18-155.el5 #1 SMP Fri Jun 19 17:06:47 EDT 2009 i686 i686 i386 GNU/Linux What components do I need to install on Linux and windows to do that? TIA (6 Replies)
Discussion started by: magnus29
6 Replies

3. UNIX for Dummies Questions & Answers

Mounting Linux box to Linux box

Hi, I've been able to mount my linux box to a windows machine, but I can't seem to mount my linux box to another linux box I have. (I know I could scp, but for other reasons I need to do it this way) Samba is installed. Here is an example where I mount to a Win machine.--> works fine mount... (12 Replies)
Discussion started by: jdilts
12 Replies

4. Solaris

Memory issue on solaris10 box

my system has 128G of installed memory. top, vmstat shows the system has just over 10G of free memory on the system. but as per prstat o/p the usage is just 50-55G is there anyway i can find which process/zone is using more memory ? System has 3 zones and all running application servers. ... (1 Reply)
Discussion started by: fugitive
1 Replies

5. Solaris

Memory of box

I have certain questions. 1) How can i see the memory of the unix box. 2) How can i see the size of the database on the box 3)can anyone suggest an article or tutorial that explains the concept of file systems and mount point in UNIX. 4)How can i see the dblink on the server I... (5 Replies)
Discussion started by: asalman.qazi
5 Replies

6. Linux

my box can't see full memory

Hi I'be recently installed Virtouzzo on Centos 5 on 16GB box , but the system could only see 4 GB of RAM, I installed the package kernel-PAE, but the virtuozzo kernel still can't see the full memory. even the kernel system can see 16GB of RAM is there any idea bout that ? Thanks (2 Replies)
Discussion started by: Raied
2 Replies

7. HP-UX

help me decipher how much memory on my box

hi, if I do top, I get Memory: 19277012K (5868296K) real, 33860312K (11294208K) virtual, 795392K free If I do swapinfo -tm I get: % swapinfo -tm Mb Mb Mb PCT TYPE AVAIL USED FREE USED dev 16384 0 16383 0% dev ... (3 Replies)
Discussion started by: JamesByars
3 Replies

8. Shell Programming and Scripting

Script to Reboot a linux box from a windows box

HI All, I need a script to reboot a linux box from a windows box. The script needs to run automatically whenever a sitescope alerts with an error message. Have searched for this in the forums, but could not get something relative. Pls. let me know the various alternatives we have to do... (2 Replies)
Discussion started by: Crazy_murli
2 Replies

9. UNIX for Dummies Questions & Answers

Fragmentation command in linux?

Hi, Please let me know more details on fragmentation in redhat linux and command to check fragmented files? Thanks, Bache Gowda (2 Replies)
Discussion started by: bache_gowda
2 Replies

10. Filesystems, Disks and Memory

Memory usage in the box

Hello: Environment is: Oracle 817 on IBM RS/6000 AIX 433 I have 4GB RAM on the box and Page/Swap is about the same. Presently I am using close to 1GB of RAM towards 5 instances of ORACLE production environments. How can I know, how much of memory/RAM is used for : Oracle Processes , I... (2 Replies)
Discussion started by: ST2000
2 Replies
Login or Register to Ask a Question