01-28-2009
In need of multi threaded perl assistance
I need to write a perl script to execute external programs and grab the output and return code. Each program should be killed if it has not completed within X seconds.
Imagine that the script goes something like this :
@commands = &get_commands();
foreach $cmd (@commands) {
$pid = execute($cmd, $timeout);
push (@pids, $pid);
}
# pid included above in case it's needed
&wait_for_completion_or_timeout;
foreach (@commands) {
if ($output{$cmd}) {
print "Output of $cmd was $output{$cmd}, it comleted in $duration{$cmd} seconds\n";
} else {
print "$cmd did not complete in the allowed time and was killed";
}
... what could the execute function look like ?
Currently I have something roughly like this
sub execute {
my $cmd = shift;
my $pid;
unless ($pid = fork) {
exec("$cmd > /tmp/out.$$");
}
return $pid; # returning this pid, just in case :-)
}
This works, but I don't feel it's very elegant. I have to keep track of all pids and kill processes that take more than X seconds using 'ps' to track processes and 'kill' to kill them. Ideally I'd like something platform independent. I've looked into multi threaded programming but I'm unfamiliar with this and I'm stomped.
Example of my attempted code to achieve the same as above:
sub execute {
my $cmd = shift;
system("$cmd >/tmp/out.$$");
}
foreach $cmd (@commands) {
my $thread[$i++] = threads->create(\&execute,$cmd);
}
If I run this code I get an error like
A thread exited while 5 threads were running.
... where do I go from here ? How do I wait until each thread has completed, and how would I kill slow threads ?
Should I be looking at a completely different approach ?
Any assistance appreciated !
10 More Discussions You Might Find Interesting
1. Programming
I am trying to writa a multi-client & multi-threaded TCP server.
There is a thread pool. Each thread in the pool will handle
requests of multiple clients.
But here I have a problem. I find a solution but it is not how
it must be... i think. When threads working without sleep(1)
I can't... (0 Replies)
Discussion started by: Parahat Melayev
0 Replies
2. Programming
I was wondering if anyone could give me a good idea how to calculate how balanced the threading is on a multi-threaded application. I want a percentage, such as "threads are 80% balanced."
This is the way I am currently going about it, maybe it is good, maybe not.
First, whenever a thread... (2 Replies)
Discussion started by: DreamWarrior
2 Replies
3. Shell Programming and Scripting
I need to query a http site and then parse the xml results, this works well if I use the string in IE but I require an automated solution.
I have tried using the following as well as HTTP::Request, nothing seems to work any suggestions would be appreciated, I have tried diffrnt things I found on... (7 Replies)
Discussion started by: bryanthomas
7 Replies
4. AIX
I have a Multithreaded program which is hanging on AIX.
OS Version: AIX 5.2 and thread library version : 5.2.0.75
We Initiate the process with 50 threads..when we are disconnecting from the process it hangs.There is lots of other stuff involved here.I am just sending the piece of the problem with... (0 Replies)
Discussion started by: hikrishn
0 Replies
5. UNIX for Advanced & Expert Users
Hello, are any of the encryption programs capable of true multi-threading ? Friend of mine tells me that he's been running some testing on Fedora 11 and that the kernel doesn't support multi-threading at that level.
I've been looking into TrueCrypt, encfs and both calm to support... (0 Replies)
Discussion started by: TehOne
0 Replies
6. Linux
Hello, are any of the encryption programs capable of true multi-threading ? Friend of mine tells me that he's been running some testing on Fedora 11 and that the kernel doesn't support multi-threading at that level.
I've been looking into TrueCrypt, encfs and both calm to support... (1 Reply)
Discussion started by: TehOne
1 Replies
7. Programming
Hello All :
I write a .c program to test the exactually resource the memory leak
as follows:
1 #include <stdio.h>
2 #define NUM 100000
3 void *Thread_Run(void * arg){
4 //TODO
5 //pthread_datch(pthread_self());
6 int socket= (int)arg;
7 ... (1 Reply)
Discussion started by: aobai
1 Replies
8. UNIX for Dummies Questions & Answers
Good morning. I have a piece of code that is currently taking multiple files and using the CAT.exe command to combine into one file that is then sorted in reverse order based on the 3rd field of the file, then displayed on screen. I am trying to change this so that the files are being combined into... (4 Replies)
Discussion started by: jaacmmason
4 Replies
9. Programming
I'm having a hard time figuring out how to manage deallocation of memory in multithreaded environments. Specifically what I'm having a hard time with is using a lock to protect a structure, but when it's time to free the structure, you have to unlock the lock to destroy the lock itself. Which will... (5 Replies)
Discussion started by: gngrwzrd
5 Replies
10. Shell Programming and Scripting
hey everyone,
I'm having some trouble breaking down some code. It's simple a control script that takes machines meant to be backed up from a list. Then according to that will run multi-threaded processes up until the specified thread limit.
for example if there are 4 machines to be backed up,... (2 Replies)
Discussion started by: terrell
2 Replies
LEARN ABOUT MOJAVE
ipc::open25.18
IPC::Open2(3pm) Perl Programmers Reference Guide IPC::Open2(3pm)
NAME
IPC::Open2 - open a process for both reading and writing using open2()
SYNOPSIS
use IPC::Open2;
$pid = open2(*CHLD_OUT, *CHLD_IN, 'some cmd and args');
# or without using the shell
$pid = open2(*CHLD_OUT, *CHLD_IN, 'some', 'cmd', 'and', 'args');
# or with handle autovivification
my($chld_out, $chld_in);
$pid = open2($chld_out, $chld_in, 'some cmd and args');
# or without using the shell
$pid = open2($chld_out, $chld_in, 'some', 'cmd', 'and', 'args');
waitpid( $pid, 0 );
my $child_exit_status = $? >> 8;
DESCRIPTION
The open2() function runs the given $cmd and connects $chld_out for reading and $chld_in for writing. It's what you think should work when
you try
$pid = open(HANDLE, "|cmd args|");
The write filehandle will have autoflush turned on.
If $chld_out is a string (that is, a bareword filehandle rather than a glob or a reference) and it begins with ">&", then the child will
send output directly to that file handle. If $chld_in is a string that begins with "<&", then $chld_in will be closed in the parent, and
the child will read from it directly. In both cases, there will be a dup(2) instead of a pipe(2) made.
If either reader or writer is the null string, this will be replaced by an autogenerated filehandle. If so, you must pass a valid lvalue
in the parameter slot so it can be overwritten in the caller, or an exception will be raised.
open2() returns the process ID of the child process. It doesn't return on failure: it just raises an exception matching "/^open2:/".
However, "exec" failures in the child are not detected. You'll have to trap SIGPIPE yourself.
open2() does not wait for and reap the child process after it exits. Except for short programs where it's acceptable to let the operating
system take care of this, you need to do this yourself. This is normally as simple as calling "waitpid $pid, 0" when you're done with the
process. Failing to do this can result in an accumulation of defunct or "zombie" processes. See "waitpid" in perlfunc for more
information.
This whole affair is quite dangerous, as you may block forever. It assumes it's going to talk to something like bc, both writing to it and
reading from it. This is presumably safe because you "know" that commands like bc will read a line at a time and output a line at a time.
Programs like sort that read their entire input stream first, however, are quite apt to cause deadlock.
The big problem with this approach is that if you don't have control over source code being run in the child process, you can't control
what it does with pipe buffering. Thus you can't just open a pipe to "cat -v" and continually read and write a line from it.
The IO::Pty and Expect modules from CPAN can help with this, as they provide a real tty (well, a pseudo-tty, actually), which gets you back
to line buffering in the invoked command again.
WARNING
The order of arguments differs from that of open3().
SEE ALSO
See IPC::Open3 for an alternative that handles STDERR as well. This function is really just a wrapper around open3().
perl v5.18.2 2013-11-04 IPC::Open2(3pm)