tee into 2 named pipes


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users tee into 2 named pipes
# 8  
Old 12-12-2006
I specified zsh in the first line of thread Smilie but if a code works in bash or ksh it usually works in zsh.
File A could be ~1e7 - 1e8 lines (in reality it's a set of files), B ~ e6 lines; really instead of
echo "Hello" | tee p1 > p2 &
it could be something as
uncompress -c A_2006*.Z | tee p1 > p2 &
or cycle as
for f in A_2006*.Z
{uncompress -c $f | tee p1 > p2 &
...
# 9  
Old 12-12-2006
I would write a lot of this in C. If that is not an option, then perl. This program would read A style records and would write them if they passed the test. To do that, it would first preprocess file B. I would read file B and build an array. 1,000,000 elements, each of which is 11 bytes is a very large array but not prohibitive. I would explore reducing the 11 bytes though. Then I would sort the array and have a binary search function for it. To validate each A record, the program then just does one binary search on the arrray. Ideally, enough physical memory should be available that the program fits into core without paging.

A shell script would unzip the data files and feed them to the program. Ideally multiple cpu's would be available. This would allow the unzip and the c program to settle into dedicated cpu's.
# 10  
Old 12-12-2006
Thanks, it's a better way than that I tryed to do, because for use join I have to sort file A originally and it's ~ N log(N) operations, so using join is good if A is sorted.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Why cannot have multiple pipes from tee?

why I cannot do this? prog_name | tee logfile | awk /regexp/ | awk /regexp/ I now this is not elegant code, but am intrigued as to why multiple pipes from tee not allowed. (2 Replies)
Discussion started by: euval
2 Replies

2. UNIX for Advanced & Expert Users

Named pipes using MKS Toolkit

I'm not sure whether or not this question really belongs in this forum and will accept rebuke should I have mistakenly put it in the wrong place (hopefully the rebuke will be accompanied by an answer, though) I wish to implement named pipe communication between two process using MKS Toolkit. I... (2 Replies)
Discussion started by: ArndW
2 Replies

3. Shell Programming and Scripting

Temporary named pipes in Hpux Kornshell

Tried the following on Hpux 11.11, using both ksh, and dtksh $diff <(sort file1) <(sort file2) $ ksh: syntax error: `(' unexpected Strange thing is I tried the same command under RHEL5 using ksh 93 and it works fine. Does anyone know if this is possible on HPUX without the use of... (0 Replies)
Discussion started by: fire!
0 Replies

4. Shell Programming and Scripting

How Unix tee to send pipeline output to 2 pipes ?

Hi, I would like to process, filter the same ASCII asynchronous live data stream in more than one pipe pipeline. So the one pipeline should filter out some records using grep key word and more than one pipes pipelines each should grep for another key words, each set seperately for each... (5 Replies)
Discussion started by: jack2
5 Replies

5. UNIX for Dummies Questions & Answers

Named Pipes

hi, i am working on a script for oracle export, m using a parameter file... i want to compress the dump file that is generated.. in my script following is the code i have written. i am not able to generata .gz file mknod /tmp/exp_tesd1_pipe p gzip -cNf... (4 Replies)
Discussion started by: saharookiedba
4 Replies

6. HP-UX

remove named pipes

Hi, Please help me on this. I am creating a named pipe in a kshell script. I am using mkfifo pipe_name command to create the pipe. I want to remove the named pipe after my work is completed. How can i do that. (8 Replies)
Discussion started by: chintapalli001
8 Replies

7. Shell Programming and Scripting

named pipes

How to have a conversation between 2 processes using named pipes? (5 Replies)
Discussion started by: kanchan_agr
5 Replies

8. Shell Programming and Scripting

FIFO named pipes

Hi...Can anyone please guide me on FIFO Pipes in UNIX.I have lerant things like creating fifo pipes,using them for reads and writes etc.I want to know what is the maximum amount of memory that such a pipe may have? Also can anyone guide me on where to get info on this topic from? (1 Reply)
Discussion started by: tej.buch
1 Replies

9. Filesystems, Disks and Memory

PIPEs and Named PIPEs (FIFO) Buffer size

Hello! How I can increase or decrease predefined pipe buffer size? System FreeBSD 4.9 and RedHat Linux 9.0 Thanks! (1 Reply)
Discussion started by: Jus
1 Replies

10. Shell Programming and Scripting

named pipes

Hi I am having trouble with a script to export individual schemas to tape from an oracle database. Basicaly I need to export each shema through a pipe with compression and store each shema name in a file with the relevant tape marker. (4 Replies)
Discussion started by: truma1
4 Replies
Login or Register to Ask a Question