awk reading from named pipe (fifo) Post: 302551562

Sponsored Content

Top Forums Shell Programming and Scripting awk reading from named pipe (fifo) Post 302551562 by nathanhaigh on Tuesday 30th of August 2011 11:02:17 PM

08-31-2011

Registered User

Quote:

Originally Posted by tange

If 'a_program' cannout output to stdout, you should be able to do this:

Code:

grep -v '^@' massive_file | parallel --max-procs 16 --pipe -N 2500 'mkfifo out_{#}; a_program -h out_{#} >stdout_{#} 2> stderr{#} & cat out_{#}' |
  awk '{ a[$1] = a[$1] + $2 } END { for (i in a) print i, a[i] }' | sort -nk1 > sorted_output

That will create a fifo for each job, save the output to the fifo while cat'ting it out from the fifo. GNU Parallel will then catch the output and send it to awk when the job is done.

I have the feeling we are talking a lot of data coming into and out from 'a_program' and that you would prefer not having temporary files (which GNU Parallel will use for buffering the output). In that case consider putting the awk script into parallel with a_program.

Hi Ole,

You are correct, we are talking 10's millions of lines going into 'a_program' so there are going to be 5,000-10,000 intermediary files created. How do I go about putting the awk script in parallel with 'a_program'?

Quote:

Originally Posted by tange

The awk script seems to count the frequency of a given input and it should not be too hard to merge several outputs from the awk script.

You are correct, the file written by 'a_program' is a 2 column file, for each value in the first column I increment the count by the corresponding value in the send column. I then simply do a numerical sort of the lines by the first column.

Cheers!
Nathan

nathanhaigh

View Public Profile for nathanhaigh

Find all posts by nathanhaigh

10 More Discussions You Might Find Interesting

1. Programming

Pipe & fifo....

Could someone Help me with this code please? #include <stdio.h> #include <unistd.h> #include <sys/types.h> #include <sys/stat.h> #include <string.h> #include <fcntl.h> #define SIZE_B 256 /*buffer's size */ #define NUM_ARG 20 /* max number of args for any command */ int...

2. Filesystems, Disks and Memory

PIPEs and Named PIPEs (FIFO) Buffer size

Hello! How I can increase or decrease predefined pipe buffer size? System FreeBSD 4.9 and RedHat Linux 9.0 Thanks!

3. UNIX for Advanced & Expert Users

PIPE and FIFO buffer size

Hello! How I can increase (or decrease) the predefined pipe buffer size? Thanks!

4. Shell Programming and Scripting

FIFO named pipes

Hi...Can anyone please guide me on FIFO Pipes in UNIX.I have lerant things like creating fifo pipes,using them for reads and writes etc.I want to know what is the maximum amount of memory that such a pipe may have? Also can anyone guide me on where to get info on this topic from?

5. UNIX for Dummies Questions & Answers

Named PIPE

Gurus, I've a File Transaction Server, which communicates with other servers and performs some processing.It uses many Named PIPE's. By mistake i copied a named PIPE into a text file. I heard that PIPE files shouldn't be copied.Isn't it? Since it's a production box, i'm afraid on...

6. Shell Programming and Scripting

Reading from blocking fifo pipe in shell script

Hi!! I have a problem reading from a fifo pipe in shell script. The idea is simple, I have a C program with two pipe files: An input pipe I use to send commands in shell script to the C program (echo "command" > input.pipe) An output pipe that I read the result of the command also in...

7. UNIX for Dummies Questions & Answers

fifo or named pipe working?

Can someone explain to me the working of fifo() system call using simple C programs so that I can implement them in the UNIX environement?

8. UNIX for Advanced & Expert Users

Why not SIGPIPE for readers of pipe/FIFO?

Hi This is a exercise question from Unix network programming vol2. Why the SIGPIPE signal is generated only for writers when readers disappear. why not it is generated for readers when writer disappears. I guess, if the writer didn't get any response like the reader gets EOF, it will...

9. Programming

Pipe & fifo size limit

Hi guys. 1. how much is the size of pipe?(i mean the buffer size) 2. is this size different in various UNIX derivations? 3. what happens if we write to a full pipe? does it block until get some free space(the other side receive data) or returns an error? 3. FIFO s are physical files on the...

10. Shell Programming and Scripting

UNIX fifo concurrent read from a named pipe

I have created a fifo named pipe in solaris, which writes the content of a file, line by line, into pipe as below: $ mkfifo namepipe $ cat books.txt "how to write unix code" "how to write oracle code" $ cat books.txt >> namepipe & I have a readpipe.sh script which reads the named...

LEARN ABOUT REDHAT

igawk

IGAWK(1)							 Utility Commands							  IGAWK(1)

NAME

       igawk - gawk with include files

SYNOPSIS

       igawk [ all gawk options ] -f program-file [ -- ] file ...
       igawk [ all gawk options ] [ -- ] program-text file ...

DESCRIPTION

       Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1).

       AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like

	      @include getopt.awk

       in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path.

OPTIONS

       See gawk(1) for a full description of the AWK language and the options that gawk supports.

EXAMPLES

       cat << EOF > test.awk
       @include getopt.awk

       BEGIN {
	    while (getopt(ARGC, ARGV, "am:q") != -1)
		 ...
       }
       EOF

       igawk -f test.awk

SEE ALSO

       gawk(1)

       Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995.

AUTHOR

       Arnold Robbins (arnold@skeeve.com).

Free Software Foundation					    Nov 3 1999								  IGAWK(1)