Programming

I wanted to use the pthread read/write locks in an application, unfortunately my task requires that readers perform very well, but writers are never starved. I have found that the AIX implementation, while having excellent reader performance, heavily starves writers.

To test this, I wrote an application that spawned about twenty read threads and a single writer thread. Letting it run I can see that the poor writer never gets the mutex; not even once! Worse, even if I throttle the readers back, the writer still gets the mutex in an untimely manner.

Unfortunately, I have tried two alternate strategies which, while more fair, have drastically worse reader performance than the pthreads implementation. This is why I'm here. I am hoping someone can either a) point me to something that may perform better, b) tell me the drastic performance change should be expected and move on with my life, lol, or c) give me some ideas how to tweak the pthreads implementation into submission.

If anyone is interested, the two alternate stragegies were:

1) I "rolled my own" read write lock, modelled after sources I found online. This uses a mutex, two condition variables, and counters for current readers and blocked writers/readers. After profiling, it seems the performance impediment here is high contention for the mutex, maybe because it is causing lots of context switches. All I can tell is that pthread_mutex_lock is killing me.

2) I grabbed a source which made use of the atomic operation fetch_and_add to implement the lock. Here, it seems fetch_and_add is expensive; which I suppose it would be.

Because the final application is mostly concerned with reader performance, I tested these with the driver spawning only reader threads and no writers to contend with them. Configured this way, the reader performance was as follows:

pthread_rw_lock: .02 milisecond average acquisition time over 11 million acquisitions (sum of all threads).
my home grown lock: .09 miliseconds and 2.18 million acquisitions.
fetch_and_add lock: .08 miliseconds and 2.5 million acquisitions.

Any ideas, questions, comments? Thanks!

Can you share your test application with us?

How much data are you keeping in memory?

If you can keep multiple copies in RAM, you can do away with the locking almost entirely.

Code:

data_t *old;
volatile data_t *current;
volatile data_t *new;
   .
   .
   .
/* data update thread */
while ( 1 )
{
    new = generateUpdatedData();
    cleanUpOldData( old );
    old = current;
    current = new;
}
   .
   .
   .
/* data reader */
data_t *myCopy = current;
extractDataTypeOne( myCopy );
extractDataTypeTwo( myCopy );

As long as the time it takes to generate a new copy of data and clean up the old copy of data is guaranteed to be longer than the time it takes for any thread to access the current copy of the data, you won't have any problems. If you can't get that guaranteed time, you can track the number of reader threads using a simple integral counter using atomic increments and decrements and not clean up the old copy until the counter goes to zero, because once a copy of the data is moved from "current" to "old", no new threads should be able to access it.

Yes, I'm assuming the compiled binary will take care of visibiity issues across multiple processors when the pointers are swapped around. I'm not sure what AIX offers for memory barriers, but Solaris and (IIRC) BSD provide membar_producer() and membar_consumer(). See "man membar_ops" or read "sys/atomic.h" on Solaris.

fpmurphy: I can not share the driver right now, it is locked up in a Citrix environment and it will not allow file transfer in/out nor internet access. I'll see what I can do to get it over here.

achenle: As far as doing in-memory swaps, I would still need a mutex to control access to the reference count, or use the atomic add operations. Both seem to suffer from a performance problem.

I know that the OS/400 implementation of pthread read/write locks does not favor writers.

I wonder does the same apply on AIX?

fpmurphy, I will attempt to make a change to the application to use pthread_rwlock_tryrdlock and see if it helps my situation and allows the writers in more often. Thanks for the suggestion, I'll let you know how it turns out.

I've used a single semaphore for reader/writer locks on occasion.

It's used somewhat like it usually would -- readers call sem_wait, do their business, then call sem_post -- but the semaphore starts with a value higher than one so multiple threads can read without contention. To get a write lock, the writer waits n times, does its business, then posts n times. This does make writing somewhat of a bigger operation than reading but if readers don't hog, writers never starve.