Unix/Linux Go Back    


UNIX for Dummies Questions & Answers This forum is closed for new posts. Please post beginner questions to learn unix and learn linux in this forum UNIX for Beginners Questions & Answers

learn unix and linux commands

How does pipe work?

UNIX for Dummies Questions & Answers


 
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 12-16-2008   -   Original Discussion by Leion
Leion's Unix or Linux Image
Leion Leion is offline
Registered User
 
Join Date: Dec 2008
Last Activity: 15 December 2015, 11:57 AM EST
Posts: 67
Thanks: 3
Thanked 12 Times in 9 Posts
Unix or Linux Question How does pipe work?

I am confused over piping. Linux

A | B

Will A and B run at the same time? or must A finish running before B starts to run?



Suppose I want to do the following:



Code:
sqlplus ... | split -1000 - filename_

sqlplus will return 1million rows, I want write the output into files of 1000 records each.

Does sqlplus finish running before split starts to run?
I am concern with the huge amount of data in the pipe before split can even run...
Sponsored Links
    #2  
Old Unix and Linux 12-16-2008   -   Original Discussion by Leion
siba.s.nayak's Unix or Linux Image
siba.s.nayak siba.s.nayak is offline
Registered User
 
Join Date: Jun 2007
Last Activity: 6 December 2011, 3:39 AM EST
Posts: 207
Thanks: 3
Thanked 0 Times in 0 Posts
Here A must finish before B starts. Because The output of A is the input for B.
Sponsored Links
    #3  
Old Unix and Linux 12-16-2008   -   Original Discussion by Leion
Leion's Unix or Linux Image
Leion Leion is offline
Registered User
 
Join Date: Dec 2008
Last Activity: 15 December 2015, 11:57 AM EST
Posts: 67
Thanks: 3
Thanked 12 Times in 9 Posts
Quote:
Originally Posted by siba.s.nayak View Post
Here A must finish before B starts. Because The output of A is the input for B.
Thank you.
    #4  
Old Unix and Linux 12-16-2008   -   Original Discussion by Leion
Perderabo's Unix or Linux Image
Perderabo Perderabo is offline Forum Advisor  
Unix Daemon (Administrator Emeritus)
 
Join Date: Aug 2001
Last Activity: 26 February 2016, 12:31 PM EST
Location: Ashburn, Virginia
Posts: 9,930
Thanks: 63
Thanked 471 Times in 271 Posts
Quote:
Originally Posted by siba.s.nayak View Post
Here A must finish before B starts. Because The output of A is the input for B.
That is not correct. It is undefined whether A or B starts first. They might start at the exactly the same time if there are multiple cpu's. A pipe can hold an undefined but finite amount of data.

If B tries to read from the pipe, but no data is available, B will wait until the data arrives. If B was reading from a disk, B might have the same problem and need to wait until a disk read finishes. A closer analogy would be reading from a keyboard. There, B would need to wait for a user to type. But in all of these cases, B has started a "read" operation and must wait until it finishes.

If A tries to write to the pipe, and the pipe is full, A must wait for some room in the pipe to become free. A could have the same problem if A was writing to a terminal. A terminal has flow control and can moderate the pace of data. In any event, to A, it has started a "write" operation and will wait until the write operation finishes.

A and B are behaving as co-processes, although not all co-processes will be communicating with a pipe. Neither is in full control of the other.

In a case, like:
A | sort
The sort command cannot output anything until it reads all of the data. So the sort command will do that, just as it would if it was reading from a file. Many other programs strive to read and write data if they can. This allows them to be used in long pipelines with data continuously flowing though the entire pipeline.
The Following User Says Thank You to Perderabo For This Useful Post:
jawsnnn (05-31-2012)
Sponsored Links
    #5  
Old Unix and Linux 12-16-2008   -   Original Discussion by Leion
siba.s.nayak's Unix or Linux Image
siba.s.nayak siba.s.nayak is offline
Registered User
 
Join Date: Jun 2007
Last Activity: 6 December 2011, 3:39 AM EST
Posts: 207
Thanks: 3
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by Perderabo View Post
That is not correct. It is undefined whether A or B starts first. They might start at the exactly the same time if there are multiple cpu's. A pipe can hold an undefined but finite amount of data.

If B tries to read from the pipe, but no data is available, B will wait until the data arrives. If B was reading from a disk, B might have the same problem and need to wait until a disk read finishes. A closer analogy would be reading from a keyboard. There, B would need to wait for a user to type. But in all of these cases, B has started a "read" operation and must wait until it finishes.

If A tries to write to the pipe, and the pipe is full, A must wait for some room in the pipe to become free. A could have the same problem if A was writing to a terminal. A terminal has flow control and can moderate the pace of data. In any event, to A, it has started a "write" operation and will wait until the write operation finishes.

A and B are behaving as co-processes, although not all co-processes while be communicating with a pipe. Neither is in full control of the other.

In a case, like:
A | sort
The sort command cannot output anything until it reads all of the data. So the sort command will do that, just as it would if it was reading from a file. Many other programs strive to read and write data if they can. This allows them to be used in long pipelines with data continuously flowing though the entire pipeline.
Thanks a lot. Though I knew these things, I could not map the same things with the question. This is purely lack of understanding on IPC. Thanks a lot again for refreshing up me.
Sponsored Links
 


Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
pipe to grep doesn't work in bash script kcstrom Shell Programming and Scripting 4 02-11-2012 09:18 PM
Replace pipe with Broken Pipe saj Shell Programming and Scripting 1 04-08-2011 10:18 AM
awk's getline < "-" seems not work for pipe qiulang Shell Programming and Scripting 17 04-09-2009 06:41 AM
How to Avoid intermediate files when pipe does nt work w020637 Shell Programming and Scripting 1 02-04-2009 07:19 PM
By angle-brackets/"pipe" button doesn't work? riwa Linux 1 04-02-2006 07:43 PM



All times are GMT -4. The time now is 03:51 PM.