UNIX for Advanced & Expert Users

The policy SCHED_FIFO or SCHED_RR seems to be for multiple threads within a LWP, and the nice of that LWP would affect them all. If such threads have different numerical priorities like the nice, I wonder how you could set the LWP niceness. I would expect it must reflect the highest thread's priority, as the lower need to not take long else the priority on the higher cannot be honored/recognized.

This is my understanding how things are supposed to work . Warning: the last time I dealt intensively with such questions was 6 years ago, so I can't guarantee to get everything right.

POSIX differentiates between threads subject to process contention (PCS) and system contention scope (SCS). PCS threads corresponds to the user level threads that are scheduled by the thread library, whereas SCS threads to kernel level threads that are scheduled by the OS (I think, Solaris calls SCS threads LWP).

We may set different scheduling policy and priority to these PCS; the scheduling shall be always relative to the other PCS threads within the process. We may for instance have 3 PCS threads; thread 1 with SCHED_FIFO policy and priority 1, thread 2 with priority 2, and thread 3 with the default time sharing policy (SCHED_OTHER). Assume further that these 3 PCS threads are mapped onto 1 SCS thread. When this SCS thread gets scheduled by the OS, the thread library shall look which PCS thread should run: thread 2 shall be scheduled if runnable, otherwise thread 1, otherwise thread 3.

Now, how does nice() affects the whole thing? First nice() only operates at the process level, and doesn't really make sense for real-time policy. We expect therefore nice() to affect only SCS threads that are not subject to real-time policy. Indeed, POSIX states:
Back to our previous example. If I renice my process, and if my process is not subject to real-time policy, the corresponding SCS shall be scheduled less frequently (assuming I increased the nice level) compared to other SCS running on the system. So will the 3 PCS threads. From a system perspective, we can say that all 3 PCS have been impacted simultaneously by this renice operation. From the perspective of our PCS threads however, nothing has changed since the scheduling is always relative to other PCS threads (except perhaps that it feels like running on a slower CPU).

HTH, Loïc

Yes, that is my model, too.

If you stack threads within a lwp, then the library dispatcher switches between them, but if it has a new lwp, then it gets what the O/S kernel dispatcher gives, and can be truly concurrent. Solaris and perhaps others only allow you 512 lwp, so if you want more, you must either multiprocess or share the lwp.

If any thread in a lwp blocks, the other threads in there are not running. So, if your threading is to segregate blocking I/O, you need lwp's, but if you are just servicing/polling minority activities in neat, separate threads, then having them share a lwp is appropriate. Often, threading can be used to do asynchronous I/O by letting threads block, but then if you need bandwidth, you need a lwp per thread/device. If you want each thread to exploit a different SMP CPU, you need a lwp per thread.

Finally, the yield functions are for the threads sharing a lwp, to give the CPU to their brothers, but in either case, how do you get rid of the CPU when you have no use for it? Does some flavor of yield know that all threads on the lwp are very recently satisfied, and the CPU should be handed off? Do you have to sleep or poll(0,0,1) or the like?

My understanding is that a good M:N scheduler should avoid the situation that when an user level thread blocks, all the user level threads mapped to the lwp block (because the lwp itself blocks). One possibility is to use scheduler activation, see this article if you're interested.

You mean how to give the CPU away for the entire process? The only way I know to achieve this would be to raise SIGSTOP, but then there is no mean to get again the CPU, unless an external process sends SIGCONT ;-) The question is: why would you want to do this?

The POSIX way to inform a thread to relinquish the CPU is sched_yield(). This is appropriate for switching between user level thread; otherwise such a thread would run until it blocks or completes. Usually sched_yield() causes the calling thread to be moved at the end of some scheduling queues. So if your thread is the only one in that queue, it still continues to run after sched_yield()...

And even if a 1:1 thread model is used, what does the OS scheduler do when all runnable threads call sched_yield()? I guess that most schedulers would schedule the threads (more or less in turns), until the process time slice has been exhausted. work-around like sleep or poll(0,0,1) or the like would exhibit a similar behaviour, I am afraid.

Cheers, Loïc

I give away the cpu because I am done with it and have other processes on the host that might be cpu bound. My best scenario is to have a blocking thread on each lwp thread, so the CPU is released and returned. Similar issues occur with poll/select and no-blocking and asynch i/o -- OK, now that you have nothing to write/send or no buffer space left on this host, and no incoming data in your buffers, how do you pass off the CPU like a good UNIX citizen if you want really low latency, less than poll()'s nominal millisecond? How can you become interrupt driven, and avoid wasting CPU on polling? Windows loop detection was caca!

Which OSes are you targeting?

Loïc

?!?!?

This code:

Code:

#include <pthread.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>

void *run( void *arg )
{
    int id;
    id = ( int ) arg;
    sleep( 1 );
    return( NULL );
}

int main( int argc, char **argv )
{
    int ii;
    int rc;
    int num_thr;
    num_thr = strtol( argv[ 1 ], NULL, 0 );
    pthread_t *tids = calloc( num_thr, sizeof( *tids ) );
    for ( ii = 0; ii < num_thr; ii++ )
    {
        rc = pthread_create( &( tids[ ii ] ), NULL, run, ( void * ) ii );
        if ( 0 != rc )
        {
            fprintf( stderr, "Failed on thread %d\n", ii );
            break;
        }
    }

    fprintf( stderr, "started %d threads\n", ii );

    for ( ii = 0; ii < num_thr; ii++ )
    {
        pthread_join( tids[ ii ], NULL );
    }

    return( 0 );
}

produces this on Solaris 10:

Code:

-bash-3.00$ ./thr 32000
started 32000 threads
-bash-3.00$ ./thr 100000
started 100000 threads
-bash-3.00$ ./thr 1000000
started 1000000 threads

Yes, 1,000,000 threads. I didn't check to see if they were all concurrent at that point, though, since it took about 100 seconds to run. The 32,000 thread example ran in a second or two.