Basic multithreaded program


 
Thread Tools Search this Thread
Top Forums Programming Basic multithreaded program
# 1  
Old 06-14-2009
Basic multithreaded program

I'd like to write a program (I'm flexible on language; C/C++ was my original idea but a scripting language would probably be better) that runs hundreds of programs, but only N = 4 (say) at a time. The idea is to keep all the cores on a multicore machine busy.

How can I do this? In particular, I'd like a library call I can make in some appropriate language that can
  • Start a new thread
  • Start a command-line process in the thread with arbitrary arguments
  • Recognize when the process is complete, return information to the main thread and terminate the helper/slave thread
  • Ideally, send and receive information on standard in/out
This seems like a very basic thing to ask; I'm just looking for something that would make this simple. I'm probably going to write many programs like this for various tasks, and I thought it would be good to ask around before diving into something that's not quite appropriate. I started reading about the (new) Python threading earlier before I thought to ask for advice/help.
# 2  
Old 06-14-2009
Could you give us some more information?
  • How do you get the list of programs to be run?
  • Are there any dependencies between the programs? If yes, how would you describe them?
  • What should happen to stdout/stderr? Output to the console or saved somewhere?
  • Do the programs require some kind of input once they're running?
# 3  
Old 06-14-2009
What OS are you planning to use this program on? Can you control processor affinity on this OS?
# 4  
Old 06-14-2009
Quote:
Originally Posted by pludi
How do you get the list of programs to be run?
The program will read in a text file with a bunch of numbers, do some processing on them, then create an array based on those numbers. Each element of the array will be passed through a function which will create an appropriate data set for that entry. The data will be used to create a temporary file and a command line argument using that file.

I didn't think this was relevant before so I didn't mention it. Basically, the program does some work and comes up with a list of commands to run.

Quote:
Originally Posted by pludi
Are there any dependencies between the programs? If yes, how would you describe them?
Ah, that's a rather important question I forgot to address! The programs are independent. Dependencies make for complicated programs; mine is just a basic one.

Quote:
Originally Posted by pludi
What should happen to stdout/stderr? Output to the console or saved somewhere?
I'd like it to be passed back to the program as a string, if possible. It should not be displayed.

Quote:
Originally Posted by pludi
Do the programs require some kind of input once they're running?
This one I'm working on does not, but I'd like a method that could send input because other similar programs I'll write will probably need to do this.

-----Post Update-----

Quote:
Originally Posted by fpmurphy
What OS are you planning to use this program on? Can you control processor affinity on this OS?
I just changed my OS to 64-bit Ubuntu 9.04. The programs are processor-intensive 64-bit programs with small-medium memory footprints.
# 5  
Old 06-14-2009
Quote:
Originally Posted by CRGreathouse
The program will read in a text file with a bunch of numbers, do some processing on them, then create an array based on those numbers. Each element of the array will be passed through a function which will create an appropriate data set for that entry. The data will be used to create a temporary file and a command line argument using that file.
So you've got a bunch of data, transpose it to another form & then process that further, right? If you've got access to the source for the last part of the processing, might it be possible to rewrite it using OpenMP (Wikipedia)? That way it'd be portable across different OS', processors and number of cores, and you'd eliminate the need for a central control program.
# 6  
Old 06-14-2009
Quote:
Originally Posted by pludi
So you've got a bunch of data, transpose it to another form & then process that further, right? If you've got access to the source for the last part of the processing, might it be possible to rewrite it using OpenMP (Wikipedia)? That way it'd be portable across different OS', processors and number of cores, and you'd eliminate the need for a central control program.
The individual programs aren't really parallelizable. (In CS jargon, they're conjectured to be P-complete, outside NC.) They're doing many sequential operations to a single piece of data. That's why I want to run them individually.

So instead I write a program to find the most efficient solution for each piece, calculate its likely runtime, solve an approximate bin-packing problem, and schedule each across a user-tunable number of processors.
# 7  
Old 06-14-2009
OK, from what you've posted 'till now, the basic structure would be something like this (pseudo-code)
Code:
Read numbers from file
Process them
Create output files
Create array of commands to run
Total processes=0
While there are commands left
    pop a command from the stack
    fork() a subprocess
    In the child
        system() the command
    In the parent
        Total processes++
    If total processes >= 4
        wait() until any process returns

Input isn't really a problem here, since fork()ed processes inherit the parents file descriptors. Output is, since as far as I know it's hard to return data from the child to the parent without shared memory or something similar. You could save the output to a file in each child my changing stdout just before the system().
Alternatively, you could run exec() the program yourself, to reduce the fork-rate, but I'm not sure how well this would work.
Plus, with shed_setaffinity you can set the affinity of the processes (direct it which CPU to use), but you'd have to track which CPU the last process used.

Anyone with more experience with algorithms has a better idea?
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Homework & Coursework Questions

Log file analyzer, super basic sh program

Hello! I'd like some help with this assignment. 1. The problem statement, all variables and given/known data: 1)Write a shell script that can uses two types of files as inputs, apache.log and apache.error.log 2)Make it so that you can switch between the two file types 3)Make it so that the... (5 Replies)
Discussion started by: malfiory
5 Replies

2. Homework & Coursework Questions

Need help with Basic Unix Program

I am a newbie to UNIX. I'm learning UNIX on my own, just trying to get the jerk of how things work in UNIX environment. I am familiar with Windows environment. Can anyone pls write simple 'envprint' programs to : 1) List all the environment Information (using the -l or --l options) 2) ... (1 Reply)
Discussion started by: agup17
1 Replies

3. HP-UX

Program monitor on BT-Basic

Hi, The "program monitor" command in BT-Basic prompt you for a user name and a password. How can I grant access only to certain users ? Thank you. (0 Replies)
Discussion started by: fosiceanu
0 Replies

4. UNIX for Advanced & Expert Users

forking in multithreaded program

hi all, i am using pthreads. What will happen if a thread does a fork? will all the threads are duplicated for the new process or only the called thread is duplicated? are the resources shared across the processes? thank you (0 Replies)
Discussion started by: skyrulz
0 Replies

5. Shell Programming and Scripting

execute command multithreaded util without programming

Hello all is there any way in unix to execute command in multithreaded way without doing it in java or cpp can one of the scripts handle multithread execution ? i need to test server requests ( corba ) in multithread Thanks (0 Replies)
Discussion started by: umen
0 Replies

6. Linux

getting problem in my code:::: plz help...(multithreaded appn with serial comm.)

hello, here I am copying my code... I am using two threads for reading and writing at com ports....one for reading and one for writing...in read thread I am using select() api....and polling again and again if there is some data to be read....but select is not returning any positive value so... (0 Replies)
Discussion started by: arunchaudhary19
0 Replies

7. Shell Programming and Scripting

Having trouble writing a basic shell program

Hello. I'm trying to write a shell script that will take files that have .tar, .tar.gz, .tar.Z, .gz, .Z and .zip file extensions and uncompress and unarchive them. The script should be able to take multiple arguments. So far I can write a script using the case command that will do this but it will... (3 Replies)
Discussion started by: SeanWuzHere
3 Replies

8. Programming

basic math program 4 child

Iam writing a script for my sisters friends little brother i want the program to say hello lets say his name is Joe and ask him how he is and he can write a reply back saying fine and then it replys "Iam happy you are (his response) today and then it goes into a basic math where he can put in a... (9 Replies)
Discussion started by: perleo
9 Replies

9. Programming

Getting errno in a Multithreaded program

In Tru64 Unix, the 'errno' variable is not thread safe. Could anybody help me about how to make it thread safe or how to check 'errno' in a Multithreaded program ???? The Programming process is like this. There are some definite number of threads having their own task. There is one... (2 Replies)
Discussion started by: S.Vishwanath
2 Replies
Login or Register to Ask a Question

Featured Tech Videos