Being a moderator at openATV, a forum for Linux settup boxes, I have seen reports, and sometimes am experiencing myself, artefacts during video playback or timeshift.
As the artefacts are non repetetive (rewinding and watching again does not show artefacts), I can exclude a corrupted video source.
We found that each artefact (up to one per minute in average) 100% correlates with an entry in /var/log/messages like
This seems to indicate severe memory fragmentation. Although enough total memory is available, the supply of 128k blocks is low.
This is not always the case. After starting the box, or after "echo 3 > /proc/sys/vm/drop_caches", there is a lot of memeory available:
Within the next few minutes, the caches fill up until approx. 6MB RAM are left. In the "good state", fragmentation is low, note the 4MB segment:
In the "bad state", memory is severely fragmented, resulting in allocation failures and playback artefacts:
Unfortunately, we have not yet found out what is causing the "bad state". I have so far only seen it after configuring the timeshift buffer to be on a USB stick and then moving it back to the HDD.
We have tried some approaches trying to cure symptoms (not addressing the root cause):
Clearing caches
"echo 3 > /proc/sys/vm/drop_caches" is freeing up memory, and executing this every 3 minutes in a cron job seems to be helpful.
Many Linux users may say that dropping caches is a bad idea. And yes, dropping them and allowing them to fill again in a cyclic manner is definitely a waste of performance, so avoiding or reducing caching from the start would probably be better. In contrast to a Linux PC executing the OS and programs from HDD, these settop boxes never execute code from HDD but from built-in Flash memory, so caching of CPU code is not required. The cache used for video data may be required for "reading ahead" and thus guaranteeing a continuous stream, but data actually may be dropped after playing it. Having said this, this may actually be critical, there is a risk that data is dropped that is just about to be played. I know very little about this and cannot say whether this is an issue. I'm also not sure what other data is being cached, I can see the cache fill up (much more slowly) with timeshift disabled.
Memory compaction
"echo 1 > /proc/sys/vm/compact_memory", (with CONFIG_COMPACTION=y), executed regularly in a cron job may be helpful as well, though I have not been able to test yet whether in the "bad state" the fragmentation is actually improving (on my box, the "bad state" is rare).
Swap
Some users reported an improvement after installing swap on a USB stick. Other experiments show that swap, though installed, is hardly being used, and I'm also a bit concerned about the access time of swap on a USB stick.
I would be grateful for thoughts and hints, espacially about strategies for finding the root cause of the memory fragmentation, knowing that this may be very difficult without detailed knowledge of the settop box internals.
I have made the observation that the Linux kernel has performance hick-ups if there is continuous I/O.
This has to do with buffering all and everything, according to the dogma "every unused byte of memory is wasted memory". If allocation happens too fast, freeing buffers can take a long time.
But the term "memory fragmentation" does not fit here.
All commercial Unix kernels have configurable limits for buffers and caches; and behave smoothly in such a situation (and little slower in others).
In the very past I had a SuSE PAE kernel that even invoked OOM-killer at continuous I/O. The following program, invoked every 5 minutes, helped: ---------- Post updated at 06:56 AM ---------- Previous update was at 03:14 AM ----------
In the case of a 64-bit kernel only the following should be called every 5 or 10 minutes:
In addition try to write smaller portions of data to the disk, in /etc/sysctl.conf:
Last edited by MadeInGermany; 09-14-2014 at 12:28 PM..
I don't think that dropping caches will help much at all. They are already considered "Free", and the non-free stuff remains non-free, so how can dropping caches defragment anything?
IO? For streaming large files, caching is irrelevant because you're almost never going to need to replay any one block. Use direct IO and bypass the page cache, and quit thrashing memory.
I don't think that dropping caches will help much at all. They are already considered "Free", and the non-free stuff remains non-free, so how can dropping caches defragment anything?
What is your kernel?
Do you know if your application uses hugepages?
The cache cannot be considered free if the mentioned drop_caches takes over 1 minute... please measure yourself!
@MadeInGermany, such script is certainly better than always dropping caches every few minutes. I may adjust it so that it drops caches based on fragmentation. Still it's a (better) way of trying to fix symptoms only.
@achenle: Yes, it's IO cache, and I share your view that for video streams data should not be cached. Can you give me more details on how a cache can be bypassed, or is this too specific to the application?
@Corona688: It's a 3.3.8 kernel. How would I detect whether hugepages are used? By looking at a defconfig file? Can you give me a keyword?
In the meantime I found that in the "bad state" executing "echo 1 > /proc/sys/vm/compact_memory" only improves fragmentation a little bit. A colleague thinks this is due to many small memory pages having been blocked.
Is there any command for finding out which memory pages have been allocated by which process?
Does your app request physically contiguous pages of heap memory for cache? - Oracle does that. This triggers the onset of heap (memory) fragmentation especially quickly in a NUMA environment - i.e., most modern commodity cpus. Multicore x86, for example. It also causes system performance problems, by requiring the core to access memory increasingly inefficiently, all the way across the bridge via another core.
This article discusses ways to invoke memory compaction, and among other things, deals with memory fragmentation. It may not help if your app does something verging on the unreasonable like the oracle db engine does. Also note: java apps that use huge heap sizes may force the jvm to reallocate memory and cause memory fragmentation for the same reason oracle does. Usually this is controllable with java parameter settings for max jvm heap size.
Hello my dear friends,
Two file are auto generated from mon - fri at different directories on same windows box.Every day i have to copy the file, rename it (specific name)and ftp it to linux box specified directory.
is it possible to automate this process,If yes this has to be done from windows... (1 Reply)
Hi
I have a linux box and need to access from windows graphically
# uname -a
Linux pc-l416116 2.6.18-155.el5 #1 SMP Fri Jun 19 17:06:47 EDT 2009 i686 i686 i386 GNU/Linux
What components do I need to install on Linux and windows to do that?
TIA (6 Replies)
Hi,
I've been able to mount my linux box to a windows machine, but I can't seem to mount my linux box to another linux box I have. (I know I could scp, but for other reasons I need to do it this way) Samba is installed.
Here is an example where I mount to a Win machine.--> works fine
mount... (12 Replies)
my system has 128G of installed memory. top, vmstat shows the system has just over 10G of free memory on the system. but as per prstat o/p the usage is just 50-55G is there anyway i can find which process/zone is using more memory ?
System has 3 zones and all running application servers.
... (1 Reply)
I have certain questions.
1) How can i see the memory of the unix box.
2) How can i see the size of the database on the box
3)can anyone suggest an article or tutorial that explains the concept of
file systems and mount point in UNIX.
4)How can i see the dblink on the server
I... (5 Replies)
Hi
I'be recently installed Virtouzzo on Centos 5 on 16GB box , but the system could only see 4 GB of RAM, I installed the package kernel-PAE, but the virtuozzo kernel still can't see the full memory.
even the kernel system can see 16GB of RAM
is there any idea bout that ?
Thanks (2 Replies)
hi,
if I do top, I get
Memory: 19277012K (5868296K) real, 33860312K (11294208K) virtual, 795392K free
If I do swapinfo -tm I get:
% swapinfo -tm
Mb Mb Mb PCT
TYPE AVAIL USED FREE USED
dev 16384 0 16383 0%
dev ... (3 Replies)
HI All,
I need a script to reboot a linux box from a windows box.
The script needs to run automatically whenever a sitescope alerts with an error message.
Have searched for this in the forums, but could not get something relative.
Pls. let me know the various alternatives we have to do... (2 Replies)
Hello:
Environment is: Oracle 817 on IBM RS/6000 AIX 433
I have 4GB RAM on the box and Page/Swap is about the same.
Presently I am using close to 1GB of RAM towards 5 instances of ORACLE production environments.
How can I know, how much of memory/RAM is used for :
Oracle Processes , I... (2 Replies)