Do you know where the source code for those portions can be found? I've looked through the Enigma 2 source and it's pretty much all Python - as expected. Where is the source code for the core C++ elements?

I'm interested in seeing what it's doing.

Because I don't think the Python code would fragment physical memory like that unless you were swapping, which I don't think you are. What I strongly suspect is happening is akin to the Oracle/ZFS ARC wars on Solairs I described earlier in this thread where Oracle would request large pages, release them, the ZFS ARC would fragment the pages...

In your case, I suspect the C++ core is probably mmap() anonymous memory, requesting large pages, and using them for video processing buffers. And, unfortunately, then releasing them, allowing the page cache to fragment the large pages. Eventually you get to the point where there are no large pages available because of fragmentation.

Unfortunately, Linux now doesn't provide any way to limit page cache size, so all you can do is try to limit its growth by either not using it (preferred since you know your IO pattern), or through kernel parameter tuning.

You can try upping the value of vm.vfs_cache_pressure and see if that helps your issues.

Or you can figure out how to do the video processing with direct IO. Even if you take a few milliseconds or even tenths of a second hit in performance when searching or stepping through a video, that won't matter much because the person doing the searching or stepping likely won't notice that. They do notice the dropped frames from memory fragmentation, though.

In your opening post you said you would drop cache and improve performance. Don't try to keep it around now.

You can use an LD_PRELOAD library to intercept all open() calls and turn on direct IO for, say, *.mpg files, for example:

Code:

#include <strings.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>

/* can't include fcntl.h since that will conflict with the
   redefinition of open() (open() really isn't varargs - it's
   three args and uses the old C calling conventions and I
   am just too lazy to play around with varargs in this...) */
#ifdef __linux
/* weak symbol in libc that's another name for open() */
int __open( const char *file, int flags, mode_t mode );
/* we need O_DIRECT */
#define __USE_GNU
#define _FCNTL_H
#include <bits/fcntl.h>

/* Solaris version */
#else
int _open( const char *file, int flags, mode_t mode );
int directio( int fd, int flag );
#define DIRECTIO_ON (1)
#endif

static int useDirectIO( const char *file )
{
    char *pp;

    /* look for last "." char in filename */
    pp = strrchr( file, '.' );
    if ( NULL == pp )
    {
        return( 0 );
    }

    /* if there is a "." char, go to the next char */
    pp++;

    /* does it end with "mpg" (any case)? */
    if ( 0 == strcasecmp( pp, "mpg" ) )
    {
        return( 1 );
    }

    /* not an "mpg" file */
    return( 0 );
}

int open( const char *file, int flags, mode_t mode )
{
    int fd;

    /* check if direct IO needs to be used */
    int setFlag = useDirectIO( file );

#ifdef __linux
    if ( setFlag )
    {
        flags |= O_DIRECT;
    }
    fd = __open( file, flags, mode );

#else /* Solaris version */
    fd = _open( file, flags, mode );
    if ( setFlag )
    {
        directio( fd, DIRECTIO_ON );
    }
#endif

    return( fd );
}

gcc [-m64] -fPIC -shared -lc open.c -o libopen.so

When you start your process, just add

Code:

LD_PRELOAD=/path/to/lib/libopen.so

and every open() library call will be intercepted.

Then - if your application does IO in ways compatible with direct IO, you'll bypass the page cache. If your application does IO in ways that are not compatible with direct IO, it'll work very badly indeed.