Visit Our UNIX and Linux User Community


How Unix tee to send pipeline output to 2 pipes ?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How Unix tee to send pipeline output to 2 pipes ?
# 1  
Old 03-13-2009
How Unix tee to send pipeline output to 2 pipes ?

Hi,

I would like to process, filter the same ASCII asynchronous live data stream in more than one pipe pipeline.

So the one pipeline should filter out some records using grep key word
and more than one pipes pipelines
each should grep for another key words, each set seperately for each pipe.

There is a number of good examples from the net, how to output pipe result to terminal as well as to save to a file,
but I need to process the same live data stream by a number of pipes pipelines in parallel and output the results to seperate files on-the-fly.

It would work as fork.
How can I define a numer of parallel processes, reading the same
data stream in parallel , as in the example below ?

Jack

--------

Since tee can read the standard input, and write to multiple files, we may leverage this feature so that it writes to multiple processes (instead of files).
tee >(process1) >(process2) >(process3) | process4

Here's a simple example of how to do this. Run the following command to get a directory listing on your terminal, while also redirecting the output to a file named poop.out:
ls -al | tee poop.out

echo “hello world” | tee test.txt
# 2  
Old 03-15-2009
follow-up

thanks to visitors,
any chance to discuss the issue ?

Jack
# 3  
Old 03-15-2009
Hi.

The bash shell (and possibly others) can set up processes like that.

Or are you telling us that this is a technique that you are now using? ... cheers, drl
# 4  
Old 03-16-2009
Quote:
Originally Posted by drl
Hi.

The bash shell (and possibly others) can set up processes like that.

Or are you telling us that this is a technique that you are now using? ... cheers, drl
Hi,

I am looking for any technique to process data streams on-the-fly
not creating cache file or not saving data stream into a file for a postprocessing.
Frankly speaking I could save each record as a string and process it by another set of instructions.
As the same data stream is used for 2-way asynchronous transmission
I need to learn how to process live data streams made of a number of 2-way substreams.

First flow charting, than algorithm and code finally (selected shell script code as to make it easy to share for discussion with my friends).

Jack

Last edited by jack2; 03-16-2009 at 07:44 PM..
# 5  
Old 03-16-2009
Hi.

Here is an example using things you mentioned:
Code:
#!/usr/bin/env bash

# @(#) s1       Demonstrate bash connecting output to processes.

echo
set +o nounset
LC_ALL=C ; LANG=C ; export LC_ALL LANG
echo "Environment: LC_ALL = $LC_ALL, LANG = $LANG"
echo "(Versions displayed with local utility \"version\")"
version >/dev/null 2>&1 && version "=o" $(_eat $0 $1) tee
set -o nounset
echo

FILE=${1-data1}

echo " Data file $FILE:"
cat $FILE

# Remove files from previous runs.

rm -f vowels four

echo
echo " Results:"
cat $FILE |
tee >( grep '^[aeiouy]' > vowels ) >( grep '^....$' > four )

sleep 1
echo
echo " Auxiliary files created:"
wc -l vowels four

exit 0

Producing:
Code:
% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.11-x1, i686
Distribution        : Xandros Desktop 3.0.3 Business
GNU bash 2.05b.0
tee (coreutils) 5.2.1

 Data file data1:
red
orange
yellow
green
blue
indigo
violet

 Results:
red
orange
yellow
green
blue
indigo
violet

 Auxiliary files created:
 3 vowels
 1 four
 4 total

The main data file is run through tee, the output of which bash connects to a few grep processes. The count at the end verifies the grep - one looks for lines beginning with a vowel, the other looks for lines that have 4 characters total. You can replace the wc to look at the content rather than simply counting.

The sleep is there because the processes that tee was writing to were not quite finished. The first few times I ran it there were no data in the files.

See man pages for details ... cheers, drl
# 6  
Old 03-16-2009
Quote:
Originally Posted by drl
Hi.

Here is an example using things you mentioned:
Code:
#!/usr/bin/env bash

# @(#) s1       Demonstrate bash connecting output to processes.

echo
set +o nounset
LC_ALL=C ; LANG=C ; export LC_ALL LANG
echo "Environment: LC_ALL = $LC_ALL, LANG = $LANG"
echo "(Versions displayed with local utility \"version\")"
version >/dev/null 2>&1 && version "=o" $(_eat $0 $1) tee
set -o nounset
echo

FILE=${1-data1}

echo " Data file $FILE:"
cat $FILE

# Remove files from previous runs.

rm -f vowels four

echo
echo " Results:"
cat $FILE |
tee >( grep '^[aeiouy]' > vowels ) >( grep '^....$' > four )

sleep 1
echo
echo " Auxiliary files created:"
wc -l vowels four

exit 0

Producing:
Code:
% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.11-x1, i686
Distribution        : Xandros Desktop 3.0.3 Business
GNU bash 2.05b.0
tee (coreutils) 5.2.1

 Data file data1:
red
orange
yellow
green
blue
indigo
violet

 Results:
red
orange
yellow
green
blue
indigo
violet

 Auxiliary files created:
 3 vowels
 1 four
 4 total

The main data file is run through tee, the output of which bash connects to a few grep processes. The count at the end verifies the grep - one looks for lines beginning with a vowel, the other looks for lines that have 4 characters total. You can replace the wc to look at the content rather than simply counting.

The sleep is there because the processes that tee was writing to were not quite finished. The first few times I ran it there were no data in the files.

See man pages for details ... cheers, drl
thanks, really excellent solution and it really works
now I have to replace data files by data streams (virtual devices or alike)
and build pipelines.
I don't know how to create output pipeline.
Redirecting file output >> to append to a file I risk generating oversized file.
So it should work as output device , like monitor, line printer, serial device
or TCP/IP port
Does shell support direct writing to a TCP/IP port or serial, virtual port, created and defined by shell self ?

Jack

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Automatically send stdout and stderror to a file as well as to the screen, but without using tee

Hi, I've been using the following commands in my automated scripts, to ensure that all text output is sent to a log file instead of to the screen: exec 1>>$SCRIPT_LOG_FILE exec 2>>$SCRIPT_LOG_FILE However, I've now discovered that the system used for automating the script executions... (4 Replies)
Discussion started by: confusedAdmin
4 Replies

2. Homework & Coursework Questions

program to send messages to parent using pipes and select system call

Write a program using select, which will create some number of child processes that continuously send text messages to the parent process using pipes. Each child has its own pipe that it uses to communicate with the parent. The parent uses select () to decide what pipes should be processed to... (1 Reply)
Discussion started by: ripssingh
1 Replies

3. Shell Programming and Scripting

how to run a script using cron job and send the output as attachment via e-mail using unix

how to run a script using cron job and send the output as attachment via e-mail using unix. please help me. how my cron job entry should be? As of now my cron job entry is to run the script at specific time, 15 03 * * * /path/sample.sh | mail -s "Logs" email_id In the above entry, what... (8 Replies)
Discussion started by: vidhyaS
8 Replies

4. Shell Programming and Scripting

problem with suppressed output to file using echo and tee command

Hi, When I run the following command in terminal it works. The string TEST is appended to a file silently. echo TEST | tee -a file.txt &>/dev/null However, when I paste this same line to a file, say shell1.sh, and use bourne shell . I run this file in terminal, ./shell1.sh. However I... (1 Reply)
Discussion started by: shahanali
1 Replies

5. Shell Programming and Scripting

Why cannot have multiple pipes from tee?

why I cannot do this? prog_name | tee logfile | awk /regexp/ | awk /regexp/ I now this is not elegant code, but am intrigued as to why multiple pipes from tee not allowed. (2 Replies)
Discussion started by: euval
2 Replies

6. UNIX for Dummies Questions & Answers

tee vs output redirection confusion

ok, suppose i have a file called f1 $ cat f1 this is file1 the quick brown fox jumped over the lazy dog this is file1 who let the dogs out this is unix this is file1 and i have another file f2 $ cat f2 this is file2 the task is to eliminate the repeated lines in f1 and add the... (11 Replies)
Discussion started by: c_d
11 Replies

7. Shell Programming and Scripting

Duplicate output without tee

Hi, Is there anyway to duplicate output without using tee? Let me explain the problem. We are use ssh to login to remote server and save output to a file using tee commands for auditing purposes. When we use vi editor in ssh session, letters get garbled and cant really use vi. Without tee it... (7 Replies)
Discussion started by: eagles1
7 Replies

8. UNIX for Advanced & Expert Users

tee into 2 named pipes

The following code does not work (zsh, Solaris), but works without the first line (files instead of pipes) :confused: mkfifo p1 p2 echo "Hello" | tee p1 > p2 & paste p1 p2 I would high appreciate any help to fix it. (9 Replies)
Discussion started by: zzol
9 Replies

9. Shell Programming and Scripting

log script input and output using tee ?

hi, new to to forum... i've been trying to create a script in tcsh but i'm having a problem with one thing... the script has to keep log of it's input and output so i'm using tee -a log | script | tee -a log this keeps the logs as asked, but it gives me an extra empty prompt (not in the... (0 Replies)
Discussion started by: moseschrist
0 Replies

10. UNIX for Dummies Questions & Answers

Unix Pipeline help

Does anyone know how to answer this? I have tried many different commands, I just cant get it right..... Search the file 'data' for all of the lines that contain the pattern 'unx122' and put those lines in the file 'matches'. (2 Replies)
Discussion started by: netmaster
2 Replies

Featured Tech Videos