How does pipe work?


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers How does pipe work?
# 1  
Old 12-16-2008
Question How does pipe work?

I am confused over piping. Smilie

A | B

Will A and B run at the same time? or must A finish running before B starts to run?



Suppose I want to do the following:

Code:
sqlplus ... | split -1000 - filename_

sqlplus will return 1million rows, I want write the output into files of 1000 records each.

Does sqlplus finish running before split starts to run?
I am concern with the huge amount of data in the pipe before split can even run...
# 2  
Old 12-16-2008
Here A must finish before B starts. Because The output of A is the input for B.
# 3  
Old 12-16-2008
Quote:
Originally Posted by siba.s.nayak
Here A must finish before B starts. Because The output of A is the input for B.
Thank you.
# 4  
Old 12-16-2008
Quote:
Originally Posted by siba.s.nayak
Here A must finish before B starts. Because The output of A is the input for B.
That is not correct. It is undefined whether A or B starts first. They might start at the exactly the same time if there are multiple cpu's. A pipe can hold an undefined but finite amount of data.

If B tries to read from the pipe, but no data is available, B will wait until the data arrives. If B was reading from a disk, B might have the same problem and need to wait until a disk read finishes. A closer analogy would be reading from a keyboard. There, B would need to wait for a user to type. But in all of these cases, B has started a "read" operation and must wait until it finishes.

If A tries to write to the pipe, and the pipe is full, A must wait for some room in the pipe to become free. A could have the same problem if A was writing to a terminal. A terminal has flow control and can moderate the pace of data. In any event, to A, it has started a "write" operation and will wait until the write operation finishes.

A and B are behaving as co-processes, although not all co-processes will be communicating with a pipe. Neither is in full control of the other.

In a case, like:
A | sort
The sort command cannot output anything until it reads all of the data. So the sort command will do that, just as it would if it was reading from a file. Many other programs strive to read and write data if they can. This allows them to be used in long pipelines with data continuously flowing though the entire pipeline.
This User Gave Thanks to Perderabo For This Post:
# 5  
Old 12-16-2008
Quote:
Originally Posted by Perderabo
That is not correct. It is undefined whether A or B starts first. They might start at the exactly the same time if there are multiple cpu's. A pipe can hold an undefined but finite amount of data.

If B tries to read from the pipe, but no data is available, B will wait until the data arrives. If B was reading from a disk, B might have the same problem and need to wait until a disk read finishes. A closer analogy would be reading from a keyboard. There, B would need to wait for a user to type. But in all of these cases, B has started a "read" operation and must wait until it finishes.

If A tries to write to the pipe, and the pipe is full, A must wait for some room in the pipe to become free. A could have the same problem if A was writing to a terminal. A terminal has flow control and can moderate the pace of data. In any event, to A, it has started a "write" operation and will wait until the write operation finishes.

A and B are behaving as co-processes, although not all co-processes while be communicating with a pipe. Neither is in full control of the other.

In a case, like:
A | sort
The sort command cannot output anything until it reads all of the data. So the sort command will do that, just as it would if it was reading from a file. Many other programs strive to read and write data if they can. This allows them to be used in long pipelines with data continuously flowing though the entire pipeline.
Thanks a lot. Though I knew these things, I could not map the same things with the question. This is purely lack of understanding on IPC. Thanks a lot again for refreshing up me.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. IP Networking

Discussion at work, would a router work pluging a cable in wan1 and lan1?

hi all. and sorry for the random question, but this sparkled a raging flame-war at work and i want more points of view situation a router, with linux of some sort, dhcp client requesting for ip in wan1 (as usual with wan ports) dhcp server listening in lan1, and assigning ip (as usual... (9 Replies)
Discussion started by: broli
9 Replies

2. Shell Programming and Scripting

Would pipe work better with this command

Hi again, have a script that I would like run, but before I can run it I need to strip out the windows \r end of lines. I have put the command into a text file and set the command to run every 10 seconds the coomand I use to do this is while sleep 10; do... (15 Replies)
Discussion started by: Paul Walker
15 Replies

3. Shell Programming and Scripting

Why does bc work with 'here string' and not via pipe in this example?

Hi! I actually got it running, but I still would like to understand, why and how, since I am a beginner in bash scripting. I Need floating numbers and thus use bc in my bash script. Here it is: #!/bin/bash num1="10^-15" | bc -l #power function piped to bc - DOES NOT WORK echo $num1... (4 Replies)
Discussion started by: McHale
4 Replies

4. Shell Programming and Scripting

How to ignore Pipe in Pipe delimited file?

Hi guys, I need to know how i can ignore Pipe '|' if Pipe is coming as a column in Pipe delimited file for eg: file 1: xx|yy|"xyz|zzz"|zzz|12... using below awk command awk 'BEGIN {FS=OFS="|" } print $3 i would get xyz But i want as : xyz|zzz to consider as whole column... (13 Replies)
Discussion started by: rohit_shinez
13 Replies

5. Shell Programming and Scripting

pipe to grep doesn't work in bash script

Hi, I'm trying to write a script that checks gvfs to see if a mount exists so I can run it from network-manager's status hooks. I thought I'd pipe the output of gvfs-mount -l to grep for the particular mounts I care about. When I do this in a bash script: cmnd="gvfs-mount -l | grep -i... (4 Replies)
Discussion started by: kcstrom
4 Replies

6. Shell Programming and Scripting

Replace pipe with Broken Pipe

Hi All , Is there any way to replace the pipe ( | ) with the broken pipe (0xA6) in unix (1 Reply)
Discussion started by: saj
1 Replies

7. Shell Programming and Scripting

awk's getline < "-" seems not work for pipe

Hi all, I have an gawk script to get user's input, So I use getline name < "-" (or getline name < "/dev/stdin") in my script They both work fine when my script deals with files. But it is broken for pipes. When I try "some command | my awk script", the variable name just gets an empty... (17 Replies)
Discussion started by: qiulang
17 Replies

8. Shell Programming and Scripting

How to Avoid intermediate files when pipe does nt work

problem with piping one output to another.Would like to avoid the intermediate file creation.The piping does nt work on places where files have been created and goes in an endless loop. sed -e "s/^\.\///g" $LINE1| sed -e "s/_\(\)/kkk\1/g" > $file1 tr -s '_' ' ' < $file1| \ sort -n -k... (1 Reply)
Discussion started by: w020637
1 Replies

9. UNIX for Dummies Questions & Answers

Script doesn't work, but commands inside work

Howdie everyone... I have a shell script RemoveFiles.sh Inside this file, it only has two commands as below: rm -f ../../reportToday/temp/* rm -f ../../report/* My problem is that when i execute this script, nothing happened. Files remained unremoved. I don't see any error message as it... (2 Replies)
Discussion started by: cheongww
2 Replies

10. Linux

By angle-brackets/"pipe" button doesn't work?

How can I configure it? I have a swedish keyboard with swedish keyboard setting. Everything works perfectly (едц) except that button. What can be wrong? /Richard ++ NOTE: It seems like the computer notices the input but that the button isn't assigned to anything (the keyboard-cursor stops).... (1 Reply)
Discussion started by: riwa
1 Replies
Login or Register to Ask a Question