Noting your help on the Programming forum too!

Are you suggesting then, that if I used such an instruction relatively frequently (say once in a loop of maybe a 100 execution statements, per core), I shouldn't notice a significant drop in throughput of the application?

You'd expect that each core accessing the XCHG variable though would have to get the value from memory though as soon as it accessed it, otherwise what use would CMPXCHG be? Not sure about this area to be honest, (but I read that these atomic operations do create a memory barrier so a core cannot execute instructions either side of said barrier out of order).

What I'm building is a thread-pool with n pthreads equal to the number of cores (so I am using pthreads). The pthreads continually execute a list of "lightweight tasks". The user can create tasks and send "messages" between them. (If you've ever used Erlang, something similar to the abstraction provided there but in my case using C).

The pthreads occasionally check the "value" of a task "state", when they reach that task in the queue, therefore if the "state" isn't "ready" they simply move on to the next task (hence the pthread has more work to do and isn't polling continuously). You see what this means, as long as a pthread "eventually" discovers a task is "ready" that's okay, even if it's not asap. It seems like a lock would be unnecessary here then, but a pthread shouldn't detect that the task state is "ready" before its other data members have been updated (hence the need for a memory barrier).

If using these atomic operations isn't going to impact throughput, then great they solve the problem, but even that seems like overkill when I only need to ensure that a handful of statements are executed in a certain order.