Sponsored Content
Top Forums Programming Capture stdout from multiple processes Post 302547060 by agama on Saturday 13th of August 2011 11:20:22 AM
Old 08-13-2011
You are most welcome.

The sample actually executes the commands in parallel, but reads the output serially to demonstrate that the output was not intermixed.

Here's a small example of a main that starts a thread for each filename (up to 10) given on the command line. Each thread uses a popen() to cat the file (please no flames about useless uses of cat!!) and reads the data counting the number of times the character 'e' appears in the file. It's a silly (um, useless) programme, because the thread could open and read the file directly, but rather than invoking a ps or netstat command, I wanted something that I could repeat to ensure that the proper number of e's from each file were being counted (to prove no mixing of output from the commands).

The function invoked as the thread sleeps to demonstrate that the threads are indeed created in parallel -- all "create" messages from the main should print before any thread actually announces that it's running the command.

Hope this helps to get you going

Code:
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include <pthread.h>
#include <string.h>



/*
    started as a thread. creates a child process via popen() and reads
    the child's output counting the number of target characters (global)
    that were in the output. When done, the count is printed on stdout.
    A delay of 2s is imposed to demonstrate that the main has started
    all threads in parallel, otherwise it is unnecessary.
*/
void *reader( void *data )
{
    char    target = 'e';       /* count these from the command's output */
    char    *cmd;               /* command to execute */
    FILE    *rpipe;
    char    buf[4096];          /* read bufffer */
    int     tcount = 0;         /* count of target */
    char    *cp;                /* pointer as we walk the buffer looking for target */

    cmd = (char *) data;

    sleep( 2);
    fprintf( stderr, "starting the command: %s\n", cmd );

    rpipe = popen( cmd, "r" );                      /* start the child */
    while( fgets( buf, sizeof( buf ), rpipe ) )     /* read all of the child's stdout */
    {
        cp = buf;
        while( (cp = strchr( cp, target )) )        /* count number of target characters in this buffer */
        {
            tcount++;
            cp++;
        }
    }

    pclose( rpipe );

    printf( "output from `%s` contained %d '%c' characters\n", cmd, tcount, target );

    free( data );
    pthread_exit( 0 );

    return NULL;            /* shouldn't matter, but keeps compilers happy */
}

int main( int argc, char **argv )
{
    pthread_t tids[10];
    char    cbuf[1024];
    int     i;

    if( argc < 2 || argc > 11 )
    {
        fprintf( stderr, "usage: %s filename1 [filename2... filename10]\n", argv[0] );
        exit( 1 );
    }


    for( i = 1; i < argc; i++ )         /* start all threads to process all files in parallel */
    {
        snprintf( cbuf, sizeof( cbuf ), "cat %s", argv[i] );
        fprintf( stderr, "starting thread: %s\n", cbuf );
        pthread_create( &tids[i-1], NULL, reader, (void *) strdup( cbuf ) );        
                                                  /* thread must free dup to avoid leak */
    }

    fprintf( stderr, "all threads started, main thread waits....\n" );

    for( i = 0; i < argc-1; i++ )                       /* wait for all threads to finish */
        pthread_join( tids[i], NULL );

    fprintf( stderr, "all threads finished\n" );

    return 0;
 }

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

not able to capture STDOUT

I am using a third party API to get some real time feed. When I run the command it shows the results properly: etd@mhs-apps5009 $ mamalistenc -m lbm -tport mamaqa -S MLALERTS -s KANA wFinancialStatus Type CTRL-C to exit. (null).MLALERTS.KANA Type: INITIAL Status OK wFinancialStatus |... (5 Replies)
Discussion started by: aks__
5 Replies

2. UNIX for Advanced & Expert Users

How to capture STDOut of script in a CGI script?

Hi Perl Experts, I am invoking a shell script thru a perl script and the perl script is cgi script.I need to capture the STDOUT of the shell script in the html page where I am invoking the script .?The shell script takes couple of mintutes to complete its execution .Mean while my html page does... (1 Reply)
Discussion started by: kittu1979
1 Replies

3. Shell Programming and Scripting

How do I capture multiple lines of the status output of a command?

I need to know what the upload speed of an Internet connection. I thought the easiest way to do this would be to transfer a file via FTP to my server using the command: sh-3.2$ ftp -u ftp://username:password@computerdomain/directory/ file_to_be_uploaded Note: My environment allows me to issue... (2 Replies)
Discussion started by: zzz1528
2 Replies

4. UNIX for Advanced & Expert Users

Capture child processes and change return values question

Thanks in advance. My environment is Ubuntu 9.04 desktop customized to be a high school classroom server for teaching code development. I have a unique "fake" jail called "lshell" which is very easy to setup and restricts users to commands that I dictate DISALLOWING ANYTHING ELSE. These... (6 Replies)
Discussion started by: tuxhats
6 Replies

5. UNIX for Dummies Questions & Answers

Redirect stdin stdout to multiple files

Hi, i know how to a) redirect stdout and stderr to one file, b) and write to two files concurrently with same output using tee command Now, i want to do both the above together. I have a script and it should write both stdout and stderr in one file and also write the same content to... (8 Replies)
Discussion started by: ysrini
8 Replies

6. Shell Programming and Scripting

Capture values using multiple regex patterns

I have to read the file, in each line of file i need to get 2 values using more than one search pattern. ex: <0112 02:12:20 def > /some string/some string||some string||124 i donot have same delimiter in the line, I have to read '0112 02:12:20' which is timestamp, and last field '124' which is... (4 Replies)
Discussion started by: adars1
4 Replies

7. UNIX for Dummies Questions & Answers

multiple variables assignement (stdout/stderr outputs)

Hi all, I've been looking around for this for a while and can't seem to find a satifactory way to do what I want: I would like to assign the output of stdout to a variable and that of stderr to another one, and this without using temporary files/named pipes. In other words be able to assign... (4 Replies)
Discussion started by: anthalamus
4 Replies

8. Shell Programming and Scripting

grep - Extracting multiple key words from stdout

Hello. From command line, the command zypper info nxclient return a bloc of data : linux local # zypper info nxclient Loading repository data... Reading installed packages... Information for package nxclient: Repository: zypper_local Name: nxclient Version: 3.5.0-7 Arch: x86_64... (7 Replies)
Discussion started by: jcdole
7 Replies

9. UNIX for Dummies Questions & Answers

Capture Multiple Lines Into Variable As Of Standard Output

Hello All, I have the below script and output. cat test.sh #!/bin/bash -x logit() { echo " - ${*}" > ${LOG_FILE} } LOG_FILE=/home/infrmtca/bin/findtest.log VAR=`find . -type f -name "*sql"` logit $VAR Output: cat /home/infrmtca/bin/findtest.log -... (9 Replies)
Discussion started by: Ariean
9 Replies

10. Shell Programming and Scripting

Execute Multiple Scripts and Capture Log Details

Hi All, I have a requirement to execute multiple scripts (say 4) one after the other in one script and capture log details and error messages in a log file below LOG_FILE= FILE.`date ++"%Y%m%d%H:%M:%S"` Script 1 : File_Checkr.sh Script 2 : Pre_Validation.sh Script 3 : Testing.sh Script... (12 Replies)
Discussion started by: Deena1984
12 Replies
Capture::Tiny(3)					User Contributed Perl Documentation					  Capture::Tiny(3)

NAME
Capture::Tiny - Capture STDOUT and STDERR from Perl, XS or external programs VERSION
version 0.24 SYNOPSIS
use Capture::Tiny ':all'; # capture from external command ($stdout, $stderr, $exit) = capture { system( $cmd, @args ); }; # capture from arbitrary code (Perl or external) ($stdout, $stderr, @result) = capture { # your code here }; # capture partial or merged output $stdout = capture_stdout { ... }; $stderr = capture_stderr { ... }; $merged = capture_merged { ... }; # tee output ($stdout, $stderr) = tee { # your code here }; $stdout = tee_stdout { ... }; $stderr = tee_stderr { ... }; $merged = tee_merged { ... }; DESCRIPTION
Capture::Tiny provides a simple, portable way to capture almost anything sent to STDOUT or STDERR, regardless of whether it comes from Perl, from XS code or from an external program. Optionally, output can be teed so that it is captured while being passed through to the original filehandles. Yes, it even works on Windows (usually). Stop guessing which of a dozen capturing modules to use in any particular situation and just use this one. USAGE
The following functions are available. None are exported by default. capture ($stdout, $stderr, @result) = capture &code; $stdout = capture &code; The "capture" function takes a code reference and returns what is sent to STDOUT and STDERR as well as any return values from the code reference. In scalar context, it returns only STDOUT. If no output was received for a filehandle, it returns an empty string for that filehandle. Regardless of calling context, all output is captured -- nothing is passed to the existing filehandles. It is prototyped to take a subroutine reference as an argument. Thus, it can be called in block form: ($stdout, $stderr) = capture { # your code here ... }; Note that the coderef is evaluated in list context. If you wish to force scalar context on the return value, you must use the "scalar" keyword. ($stdout, $stderr, $count) = capture { my @list = qw/one two three/; return scalar @list; # $count will be 3 }; Also note that within the coderef, the @_ variable will be empty. So don't use arguments from a surrounding subroutine without copying them to an array first: sub wont_work { my ($stdout, $stderr) = capture { do_stuff( @_ ) }; # WRONG ... } sub will_work { my @args = @_; my ($stdout, $stderr) = capture { do_stuff( @args ) }; # RIGHT ... } Captures are normally done to an anonymous temporary filehandle. To capture via a named file (e.g. to externally monitor a long-running capture), provide custom filehandles as a trailing list of option pairs: my $out_fh = IO::File->new("out.txt", "w+"); my $err_fh = IO::File->new("out.txt", "w+"); capture { ... } stdout => $out_fh, stderr => $err_fh; The filehandles must be read/write and seekable. Modifying the files or filehandles during a capture operation will give unpredictable results. Existing IO layers on them may be changed by the capture. When called in void context, "capture" saves memory and time by not reading back from the capture handles. capture_stdout ($stdout, @result) = capture_stdout &code; $stdout = capture_stdout &code; The "capture_stdout" function works just like "capture" except only STDOUT is captured. STDERR is not captured. capture_stderr ($stderr, @result) = capture_stderr &code; $stderr = capture_stderr &code; The "capture_stderr" function works just like "capture" except only STDERR is captured. STDOUT is not captured. capture_merged ($merged, @result) = capture_merged &code; $merged = capture_merged &code; The "capture_merged" function works just like "capture" except STDOUT and STDERR are merged. (Technically, STDERR is redirected to the same capturing handle as STDOUT before executing the function.) Caution: STDOUT and STDERR output in the merged result are not guaranteed to be properly ordered due to buffering. tee ($stdout, $stderr, @result) = tee &code; $stdout = tee &code; The "tee" function works just like "capture", except that output is captured as well as passed on to the original STDOUT and STDERR. When called in void context, "tee" saves memory and time by not reading back from the capture handles, except when the original STDOUT OR STDERR were tied or opened to a scalar handle. tee_stdout ($stdout, @result) = tee_stdout &code; $stdout = tee_stdout &code; The "tee_stdout" function works just like "tee" except only STDOUT is teed. STDERR is not teed (output goes to STDERR as usual). tee_stderr ($stderr, @result) = tee_stderr &code; $stderr = tee_stderr &code; The "tee_stderr" function works just like "tee" except only STDERR is teed. STDOUT is not teed (output goes to STDOUT as usual). tee_merged ($merged, @result) = tee_merged &code; $merged = tee_merged &code; The "tee_merged" function works just like "capture_merged" except that output is captured as well as passed on to STDOUT. Caution: STDOUT and STDERR output in the merged result are not guaranteed to be properly ordered due to buffering. LIMITATIONS
Portability Portability is a goal, not a guarantee. "tee" requires fork, except on Windows where "system(1, @cmd)" is used instead. Not tested on any particularly esoteric platforms yet. See the CPAN Testers Matrix <http://matrix.cpantesters.org/?dist=Capture-Tiny> for test result by platform. PerlIO layers Capture::Tiny does it's best to preserve PerlIO layers such as ':utf8' or ':crlf' when capturing (only for Perl 5.8.1+) . Layers should be applied to STDOUT or STDERR before the call to "capture" or "tee". This may not work for tied filehandles (see below). Modifying filehandles before capturing Generally speaking, you should do little or no manipulation of the standard IO filehandles prior to using Capture::Tiny. In particular, closing, reopening, localizing or tying standard filehandles prior to capture may cause a variety of unexpected, undesirable and/or unreliable behaviors, as described below. Capture::Tiny does its best to compensate for these situations, but the results may not be what you desire. Closed filehandles Capture::Tiny will work even if STDIN, STDOUT or STDERR have been previously closed. However, since they will be reopened to capture or tee output, any code within the captured block that depends on finding them closed will, of course, not find them to be closed. If they started closed, Capture::Tiny will close them again when the capture block finishes. Note that this reopening will happen even for STDIN or a filehandle not being captured to ensure that the filehandle used for capture is not opened to file descriptor 0, as this causes problems on various platforms. Prior to Perl 5.12, closed STDIN combined with PERL_UNICODE=D leaks filehandles and also breaks tee() for undiagnosed reasons. So don't do that. Localized filehandles If code localizes any of Perl's standard filehandles before capturing, the capture will affect the localized filehandles and not the original ones. External system calls are not affected by localizing a filehandle in Perl and will continue to send output to the original filehandles (which will thus not be captured). Scalar filehandles If STDOUT or STDERR are reopened to scalar filehandles prior to the call to "capture" or "tee", then Capture::Tiny will override the output filehandle for the duration of the "capture" or "tee" call and then, for "tee", send captured output to the output filehandle after the capture is complete. (Requires Perl 5.8) Capture::Tiny attempts to preserve the semantics of STDIN opened to a scalar reference, but note that external processes will not be able to read from such a handle. Capture::Tiny tries to ensure that external processes will read from the null device instead, but this is not guaranteed. Tied output filehandles If STDOUT or STDERR are tied prior to the call to "capture" or "tee", then Capture::Tiny will attempt to override the tie for the duration of the "capture" or "tee" call and then send captured output to the tied filehandle after the capture is complete. (Requires Perl 5.8) Capture::Tiny may not succeed resending UTF-8 encoded data to a tied STDOUT or STDERR filehandle. Characters may appear as bytes. If the tied filehandle is based on Tie::StdHandle, then Capture::Tiny will attempt to determine appropriate layers like ":utf8" from the underlying filehandle and do the right thing. Tied input filehandle Capture::Tiny attempts to preserve the semantics of tied STDIN, but this requires Perl 5.8 and is not entirely predictable. External processes will not be able to read from such a handle. Unless having STDIN tied is crucial, it may be safest to localize STDIN when capturing: my ($out, $err) = do { local *STDIN; capture { ... } }; Modifying filehandles during a capture Attempting to modify STDIN, STDOUT or STDERR during "capture" or "tee" is almost certainly going to cause problems. Don't do that. No support for Perl 5.8.0 It's just too buggy when it comes to layers and UTF-8. Perl 5.8.1 or later is recommended. Limited support for Perl 5.6 Perl 5.6 predates PerlIO. UTF-8 data may not be captured correctly. ENVIRONMENT
PERL_CAPTURE_TINY_TIMEOUT Capture::Tiny uses subprocesses for "tee". By default, Capture::Tiny will timeout with an error if the subprocesses are not ready to receive data within 30 seconds (or whatever is the value of $Capture::Tiny::TIMEOUT). An alternate timeout may be specified by setting the "PERL_CAPTURE_TINY_TIMEOUT" environment variable. Setting it to zero will disable timeouts. SEE ALSO
This module was, inspired by IO::CaptureOutput, which provides similar functionality without the ability to tee output and with more complicated code and API. IO::CaptureOutput does not handle layers or most of the unusual cases described in the "Limitations" section and I no longer recommend it. There are many other CPAN modules that provide some sort of output capture, albeit with various limitations that make them appropriate only in particular circumstances. I'm probably missing some. The long list is provided to show why I felt Capture::Tiny was necessary. o IO::Capture o IO::Capture::Extended o IO::CaptureOutput o IPC::Capture o IPC::Cmd o IPC::Open2 o IPC::Open3 o IPC::Open3::Simple o IPC::Open3::Utils o IPC::Run o IPC::Run::SafeHandles o IPC::Run::Simple o IPC::Run3 o IPC::System::Simple o Tee o IO::Tee o File::Tee o Filter::Handle o Tie::STDERR o Tie::STDOUT o Test::Output SUPPORT
Bugs / Feature Requests Please report any bugs or feature requests through the issue tracker at <https://github.com/dagolden/Capture-Tiny/issues>. You will be notified automatically of any progress on your issue. Source Code This is open source software. The code repository is available for public review and contribution under the terms of the license. <https://github.com/dagolden/Capture-Tiny> git clone https://github.com/dagolden/Capture-Tiny.git AUTHOR
David Golden <dagolden@cpan.org> CONTRIBUTOR
Dagfinn Ilmari Mannsaaker <ilmari@ilmari.org> COPYRIGHT AND LICENSE
This software is Copyright (c) 2009 by David Golden. This is free software, licensed under: The Apache License, Version 2.0, January 2004 perl v5.16.3 2014-02-06 Capture::Tiny(3)
All times are GMT -4. The time now is 05:02 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy