Sponsored Content
Top Forums Shell Programming and Scripting awk reading from named pipe (fifo) Post 302551404 by hfreyer on Tuesday 30th of August 2011 09:27:20 AM
Old 08-30-2011
The problem seems to be, what happens if the program reading from the pipe, does not get data every time it would like to have them. I am not sure how to come around this in awk, but using perl you could try the following. Create a perl script "namedpipe.pl":
Code:
#!/usr/bin/perl
$tmax=5; # timeout after 5 seconds without data from pipe
open(IN,"< my_fifo") || die("cannot open my_fifo"); # blocks until first data available
$tstart=time();
LOOP: while(1){
  $_=<IN>; # non-blocking read, undef if no data available
  if (length) {
    # line contains something, process it
    chomp;
    @f=split;
    $a{@f[0]} += @f[1];
    # reset timestamp
    $tstart=time();
  }
  # leave loop on timeout
  last LOOP if (time()-$tstart>$tmax);
}
# final print with sorting
foreach $key (sort(keys(%a))) {
  print "$key $a{$key}\n"
}

and then use it like:
Code:
mkfifo my_fifo
perl namedpipe.pl > sorted_output &
grep -v '^@' massive_file | parallel --max-procs 16 --pipe -N 2500 a_program -h my_fifo > stdout 2> stderr; wait

 

10 More Discussions You Might Find Interesting

1. Programming

Pipe & fifo....

Could someone Help me with this code please? #include <stdio.h> #include <unistd.h> #include <sys/types.h> #include <sys/stat.h> #include <string.h> #include <fcntl.h> #define SIZE_B 256 /*buffer's size */ #define NUM_ARG 20 /* max number of args for any command */ int... (4 Replies)
Discussion started by: M3xican
4 Replies

2. Filesystems, Disks and Memory

PIPEs and Named PIPEs (FIFO) Buffer size

Hello! How I can increase or decrease predefined pipe buffer size? System FreeBSD 4.9 and RedHat Linux 9.0 Thanks! (1 Reply)
Discussion started by: Jus
1 Replies

3. UNIX for Advanced & Expert Users

PIPE and FIFO buffer size

Hello! How I can increase (or decrease) the predefined pipe buffer size? Thanks! (1 Reply)
Discussion started by: Jus
1 Replies

4. Shell Programming and Scripting

FIFO named pipes

Hi...Can anyone please guide me on FIFO Pipes in UNIX.I have lerant things like creating fifo pipes,using them for reads and writes etc.I want to know what is the maximum amount of memory that such a pipe may have? Also can anyone guide me on where to get info on this topic from? (1 Reply)
Discussion started by: tej.buch
1 Replies

5. UNIX for Dummies Questions & Answers

Named PIPE

Gurus, I've a File Transaction Server, which communicates with other servers and performs some processing.It uses many Named PIPE's. By mistake i copied a named PIPE into a text file. I heard that PIPE files shouldn't be copied.Isn't it? Since it's a production box, i'm afraid on... (2 Replies)
Discussion started by: Tamil
2 Replies

6. Shell Programming and Scripting

Reading from blocking fifo pipe in shell script

Hi!! I have a problem reading from a fifo pipe in shell script. The idea is simple, I have a C program with two pipe files: An input pipe I use to send commands in shell script to the C program (echo "command" > input.pipe) An output pipe that I read the result of the command also in... (4 Replies)
Discussion started by: victorin
4 Replies

7. UNIX for Dummies Questions & Answers

fifo or named pipe working?

Can someone explain to me the working of fifo() system call using simple C programs so that I can implement them in the UNIX environement? (1 Reply)
Discussion started by: lvkchaitanya
1 Replies

8. UNIX for Advanced & Expert Users

Why not SIGPIPE for readers of pipe/FIFO?

Hi This is a exercise question from Unix network programming vol2. Why the SIGPIPE signal is generated only for writers when readers disappear. why not it is generated for readers when writer disappears. I guess, if the writer didn't get any response like the reader gets EOF, it will... (4 Replies)
Discussion started by: kumaran_5555
4 Replies

9. Programming

Pipe & fifo size limit

Hi guys. 1. how much is the size of pipe?(i mean the buffer size) 2. is this size different in various UNIX derivations? 3. what happens if we write to a full pipe? does it block until get some free space(the other side receive data) or returns an error? 3. FIFO s are physical files on the... (2 Replies)
Discussion started by: majid.merkava
2 Replies

10. Shell Programming and Scripting

UNIX fifo concurrent read from a named pipe

I have created a fifo named pipe in solaris, which writes the content of a file, line by line, into pipe as below: $ mkfifo namepipe $ cat books.txt "how to write unix code" "how to write oracle code" $ cat books.txt >> namepipe & I have a readpipe.sh script which reads the named... (2 Replies)
Discussion started by: naveen mani
2 Replies
IO::Async::ChildManager(3pm)				User Contributed Perl Documentation			      IO::Async::ChildManager(3pm)

NAME
"IO::Async::ChildManager" - facilitates the execution of child processes SYNOPSIS
This object is used indirectly via an "IO::Async::Loop": use IO::Async::Loop; use POSIX qw( WEXITSTATUS ); my $loop = IO::Async::Loop->new; ... $loop->run_child( command => "/bin/ps", on_finish => sub { my ( $pid, $exitcode, $stdout, $stderr ) = @_; my $status = WEXITSTATUS( $exitcode ); print "ps [PID $pid] exited with status $status "; }, ); $loop->open_child( command => [ "/bin/ping", "-c4", "some.host" ], stdout => { on_read => sub { my ( $stream, $buffref, $eof ) = @_; while( $$buffref =~ s/^(.*) // ) { print "PING wrote: $1 "; } return 0; }, }, on_finish => sub { my ( $pid, $exitcode ) = @_; my $status = WEXITSTATUS( $exitcode ); ... }, ); my ( $pipeRd, $pipeWr ) = IO::Async::OS->pipepair; $loop->spawn_child( command => "/usr/bin/my-command", setup => [ stdin => [ "open", "<", "/dev/null" ], stdout => $pipeWr, stderr => [ "open", ">>", "/var/log/mycmd.log" ], chdir => "/", ] on_exit => sub { my ( $pid, $exitcode ) = @_; my $status = WEXITSTATUS( $exitcode ); print "Command exited with status $status "; }, ); $loop->spawn_child( code => sub { do_something; # executes in a child process return 1; }, on_exit => sub { my ( $pid, $exitcode, $dollarbang, $dollarat ) = @_; my $status = WEXITSTATUS( $exitcode ); print "Child process exited with status $status "; print " OS error was $dollarbang, exception was $dollarat "; }, ); DESCRIPTION
This module extends the functionality of the containing "IO::Async::Loop" to manage the execution of child processes. It acts as a central point to store PID values of currently-running children, and to call the appropriate continuation handler code when the process terminates. It provides useful wrapper methods that set up filehandles and other child process details, and to capture the child process's STDOUT and STDERR streams. METHODS
When active, the following methods are available on the containing "Loop" object. $pid = $loop->spawn_child( %params ) This method creates a new child process to run a given code block or command. The %params hash takes the following keys: command => ARRAY or STRING Either a reference to an array containing the command and its arguments, or a plain string containing the command. This value is passed into perl's "exec" function. code => CODE A block of code to execute in the child process. It will be called in scalar context inside an "eval" block. setup => ARRAY A reference to an array which gives file descriptors to set up in the child process before running the code or command. See below. on_exit => CODE A continuation to be called when the child processes exits. It will be invoked in the following way: $on_exit->( $pid, $exitcode, $dollarbang, $dollarat ) The second argument is passed the plain perl $? value. To use that usefully, see "WEXITSTATUS" and others from "POSIX". Exactly one of the "command" or "code" keys must be specified. If the "command" key is used, the given array or string is executed using the "exec" function. If the "code" key is used, the return value will be used as the exit(2) code from the child if it returns (or 255 if it returned "undef" or thows an exception). Case | WEXITSTATUS($exitcode) | $dollarbang | $dollarat --------------+------------------------+-------------+---------- exec succeeds | exit code from program | 0 | "" exec fails | 255 | $! | "" $code returns | return value | $! | "" $code dies | 255 | $! | $@ It is usually more convenient to use the "open_child" method in simple cases where an external program is being started in order to interact with it via file IO, or even "run_child" when only the final result is required, rather than interaction while it is running. "setup" array This array gives a list of file descriptor operations to perform in the child process after it has been fork(2)ed from the parent, before running the code or command. It consists of name/value pairs which are ordered; the operations are performed in the order given. fdn => ARRAY Gives an operation on file descriptor n. The first element of the array defines the operation to be performed: [ 'close' ] The file descriptor will be closed. [ 'dup', $io ] The file descriptor will be dup2(2)ed from the given IO handle. [ 'open', $mode, $file ] The file descriptor will be opened from the named file in the given mode. The $mode string should be in the form usually given to the "open" function; such as '<' or '>>'. [ 'keep' ] The file descriptor will not be closed; it will be left as-is. A non-reference value may be passed as a shortcut, where it would contain the name of the operation with no arguments (i.e. for the "close" and "keep" operations). IO => ARRAY Shortcut for passing "fdn", where n is the fileno of the IO reference. In this case, the key must be a reference that implements the "fileno" method. This is mostly useful for $handle => 'keep' fdn => IO A shortcut for the "dup" case given above. stdin => ... stdout => ... stderr => ... Shortcuts for "fd0", "fd1" and "fd2" respectively. env => HASH A reference to a hash to set as the child process's environment. nice => INT Change the child process's scheduling priority using "POSIX::nice". chdir => STRING Change the child process's working directory using "chdir". setuid => INT setgid => INT Change the child process's effective UID or GID. setgroups => ARRAY Change the child process's groups list, to those groups whose numbers are given in the ARRAY reference. On most systems, only the privileged superuser change user or group IDs. "IO::Async" will NOT check before detaching the child process whether this is the case. If setting both the primary GID and the supplementary groups list, it is suggested to set the primary GID first. Moreover, some operating systems may require that the supplementary groups list contains the primary GID. If no directions for what to do with "stdin", "stdout" and "stderr" are given, a default of "keep" is implied. All other file descriptors will be closed, unless a "keep" operation is given for them. If "setuid" is used, be sure to place it after any other operations that might require superuser privileges, such as "setgid" or opening special files. AUTHOR
Paul Evans <leonerd@leonerd.org.uk> perl v5.14.2 2012-10-24 IO::Async::ChildManager(3pm)
All times are GMT -4. The time now is 03:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy