UNIX for Dummies Questions & Answers

Hi,
I was wondering if pipes ("|"), or rather the command that follow them, know when they're supposed to wait for all the data?

For instance, if you take this:

cat my_file | sort | uniq

for uniq to work well, it needs to have rows sorted, but for lines to be sorted properly, it needs my_file in its entirety. So, will "uniq" wait until it has the entire file before starting to process?

I've tried a couple of manual test and it looks like it does, but I suspect it's just that my input file is too small and that therefore all results arrive so quickly that uniq has everything it needs already.

Any hindsight would be welcome!
Thanks,

Anthony

In general terms, yes. If there is too much data stuck because the other process does not take it out again quickly enough, the original process automatically blocks to guarantee that all data is processed:

More importantly, "sort" will wait until it has all the records before passing them sorted to "uniq".
The "uniq" program does not work on the whole file, it just looks at consecutive records. This is why it needs sorted input.

Thank you both.

So what I understand from what you say, is that doing this:

cat my_file | sort > tmp
uniq <tmp

and

cat my_file | sort | uniq

is equivalent in terms of the result it produces (not in terms of processing time of course). Can I safely assume this is always the case?

One never wants to assume, but it's pretty safe to bank on it. Take for example, piping sqlplus output through tr or sed to assist in formatting: the end-point in the redirection may receive content in spurts as the pages are released, but the process and/or resource itself is always gated and not released until completion.

Thanks a lot guys, that's very helpful