Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to use an input pipe ?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to use an input pipe ?
# 1  
Old 08-16-2011
How to use an input pipe ?
Hi all,

I would like to use properly an input pipe, like this :
Code:
cat myFile.txt | myCommand.sh

I always find this solution :
Code:
while read line; do ...; done

but I have a great lost of performance !

On a big file, with a simple grep, I can spend 2400 times more time ! oO
(from 0,023sec to 1m)

So, what did I miss ? ^^'
Thx a lot !
Last edited by Scott; 08-17-2011 at 05:14 AM.. Reason: Code tags
# 2  
Old 08-16-2011
Quote:
Originally Posted by LeNouveau
I would like to use properly an input pipe, like this :
cat myFile.txt | myCommand.sh
That is a useless use of cat, which may account for a little (but probably not most) of the lost performance.
Quote:
On a big file, with a simple grep, I can spend 2400 times more time ! oO
(from 0,023sec to 1m)

So, what did I miss ? ^^'
I don't know -- I can't see the contents of this myCommand.sh from here. There may well be things that can be improved if you show us what you actually did. Frequent mistakes include the use of cut/awk/sed and so forth many times per loop instead of shell builtins, burdensomely long pipe chains, etc, etc, etc.

In any case, grep is a purpose-built utility made in raw C -- of course it's faster.
# 3  
Old 08-17-2011
the command line "cat myFile.txt | ..." was only for an example and was present in my both tests, for an equality. ^^

For example, here the test 1 :
Code:
cat myFile.txt | grep -o "toto"

And here the second test :
Code:
cat myFyle.txt | ./test2.sh

with in test2.sh :
Code:
while read line
  do echo $line | grep -o "toto"
done



So, between those tests, there is the "while" and the "echo" in difference.
Sure, the "echo" imply a lost of performances, but I don't think it's very important. So, it would come from the "while".

Of course, grep is faster because it is written in C, but it is in both case. So it should not change so much...

So, what did I miss ? ^^

---------- Post updated at 02:25 AM ---------- Previous update was at 02:07 AM ----------

So, i made some other tests :

Test 1 :
Code:
while read line
  do echo $line; grep "coucou"
done


This is so long !

Test2 :
Code:
while read line
  do echo $line; grep "coucou"
  echo $line; grep "hello"
done

 Twice longer !

Test 3 :
Code:
while read line
  do grep "coucou"
done

Almost as quick as a normal grep. Smilie

So, my problem would not come from my pipe, but how I use it.
Two solutions :
- "echo" slows my script.
- the use of the var $line slows my script.

I think it's the second solution. And you ? ^^'
Last edited by Scott; 08-17-2011 at 05:16 AM.. Reason: Please use code tags
# 4  
Old 08-17-2011
Quote:
Originally Posted by LeNouveau
the command line "cat myFile.txt | ..." was only for an example and was present in my both tests, for an equality. ^^
All of your examples could have been done without cat. you didn't need to use cat to make them "equal".
Quote:
For example, here the test 1 :
Code:
cat myFile.txt | grep -o "toto"

And here the second test :
Code:
cat myFyle.txt | ./test2.sh

with in test2.sh :
Code:
while read line
  do echo $line | grep -o "toto"
done

Of course running grep once is faster than running the same program once for each individual line Smilie

Quote:
So, between those tests, there is the "while" and the "echo" in difference.
Sure, the "echo" imply a lost of performances, but I don't think it's very important. So, it would come from the "while".
No. Running grep for each individual line has a ton of overhead. You're not supposed to do that.
Quote:
Code:
while read line
  do echo $line; grep "coucou"
done

[/I]
This is so long !
It doesn't feed the first line into grep at all. You ran grep without a pipe or parameter, so it tries to read from standard input, running grep once, slurping up all the data, causing the loop to break.
Quote:
Code:
while read line
  do grep "coucou"
done

Almost as quick as a normal grep. Smilie
Well, it would be. It only runs grep once. Without the pipe, where do you think it's reading from? If not the echo, it's reading from standard input, slurping in all the data in one go, hitting EOF, causing read to fail the next loop, and quitting.

It also ignores the first line because the while loop got to it first.
Quote:
So, my problem would not come from my pipe, but how I use it.
Please don't take offense, but you have no clue what you're doing. Substituting semicolons for pipes doesn't just make it "faster", it totally alters the meaning of your program. You should learn how to use a shell before complaining that they're slow.

There is a high cost to running and quitting an external shell utility. It has to load and map quite a few files, and for all that work you only feed it a handful of bytes before starting over. This is why constructs like echo single-line | grep | sed | awk | cut are wasteful -- they create and destroy tons of processes per loop, wasting tons of time loading files and writing between pipes that could've been used to do actual work. Imagine only being allowed to say one word per telephone call and calling over and over.

---------- Post updated at 09:49 AM ---------- Previous update was at 09:04 AM ----------

An example of everything that happens when you pipe "echo asdf" into grep. Overhead is red, useful work is green.

Code:
execve("/bin/grep", ["grep", "asdf"], [/* 27 vars */]) = 0
brk(0)                                  = 0x871f000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb78bf000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=36795, ...}) = 0
mmap2(NULL, 36795, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb78b6000
close(3)                                = 0
open("/lib/libpcre.so.0", O_RDONLY)     = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260\22\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=201952, ...}) = 0
mmap2(NULL, 204928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7883000
mmap2(0xb78b4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x30) = 0xb78b4000
close(3)                                = 0
open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20m\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1339676, ...}) = 0
mmap2(NULL, 1345832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb773a000
mmap2(0xb787d000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x143) = 0xb787d000
mmap2(0xb7880000, 10536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7880000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7739000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb77396c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb787d000, 8192, PROT_READ)   = 0
mprotect(0xb78b4000, 4096, PROT_READ)   = 0
mprotect(0x805b000, 4096, PROT_READ)    = 0
mprotect(0xb78de000, 4096, PROT_READ)   = 0
munmap(0xb78b6000, 36795)               = 0
brk(0)                                  = 0x871f000
brk(0x8740000)                          = 0x8740000
read(0, "asdf\n", 32768)                = 5
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb78be000
write(1, "asdf\n", 5)                   = 5
read(0, "", 32768)                      = 0
close(1)                                = 0
munmap(0xb78be000, 4096)                = 0
exit_group(0)                           = ?

It can work really fast once it's loaded, but it takes a bit of work to get there. So it makes sense to run external programs only when you have a reasonable amount of work for them to do.
Last edited by Corona688; 08-17-2011 at 12:44 PM..
These 2 Users Gave Thanks to Corona688 For This Post:
LeNouveau RavinderSingh13
# 5  
Old 08-17-2011
I understood where was my mistake on the third example.
I did not understand what really happened. Thx.

And I did not realise there were so many actions for a simple pipe (which is not so simple ^^').
Now, I understand why I slowed my script.
Thx a lot !
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Help on passing an input variable to the voronota program (after third pipe)

Dear UNIX forum members, I am using macbook pro 13 (2015 edition) with MAC OS Mojave and am trying to write the shell script where when it is run through terminal it asks for an input (in the code below an input variable is domains) and then that input becomes capital letter or letters which... (3 Replies)
Discussion started by: Aurimas
3 Replies

2. Shell Programming and Scripting

Space in input filename with pipe

Hello, Normally below script works, could you please comment out what could be the reason of failure if there are spaces in input filename: script.sh #!/bin/bash cd /home/hts/.hts/tvh/ file="$1 $2 $3 $4" read -d $'\x04' name < "$file" /usr/bin/ffmpeg -i ""$name"" -vcodec copy -preset... (1 Reply)
Discussion started by: baris35
1 Replies

3. Shell Programming and Scripting

While read pipe input issue

Hello, I have an ffmpeg bash script which is working nice and I need to do the same for other sources. To create new scripts and to deal with multiple bash files sounds not logical. It is a bit hard to manage for me.. I wondered if it was possible to make my input file as variable. Then I... (1 Reply)
Discussion started by: baris35
1 Replies

4. Shell Programming and Scripting

Read input files and merge them in given order and write them to input one param or one file

Dear Friends, I am looking for a shell script to merge input files into one file .. here is my idea: 1st paramter would be outfile file (all input files content) read all input files and merge them to input param 1 ex: if I pass 6 file names to the script then 1st file name as output file... (4 Replies)
Discussion started by: hyd1234
4 Replies

5. Shell Programming and Scripting

How to ignore Pipe in Pipe delimited file?

Hi guys, I need to know how i can ignore Pipe '|' if Pipe is coming as a column in Pipe delimited file for eg: file 1: xx|yy|"xyz|zzz"|zzz|12... using below awk command awk 'BEGIN {FS=OFS="|" } print $3 i would get xyz But i want as : xyz|zzz to consider as whole column... (13 Replies)
Discussion started by: rohit_shinez
13 Replies

6. Shell Programming and Scripting

Script to delete files with an input for directories and an input for path/file

Hello, I'm trying to figure out how best to approach this script, and I have very little experience, so I could use all the help I can get. :wall: I regularly need to delete files from many directories. A file with the same name may exist any number of times in different subdirectories.... (3 Replies)
Discussion started by: *ShadowCat*
3 Replies

7. Shell Programming and Scripting

Replace pipe with Broken Pipe

Hi All , Is there any way to replace the pipe ( | ) with the broken pipe (0xA6) in unix (1 Reply)
Discussion started by: saj
1 Replies

8. Shell Programming and Scripting

pass more than one input with pipe ("|")

Hi everyone, Is there any way to use the pipe "|" operator to pass more than one input? For instance do: (command1) (command2) | command3 with command 3 taking 2 parameters? or maybe using ``s? Thanks! Anthony (4 Replies)
Discussion started by: anthony.cros
4 Replies

9. UNIX for Dummies Questions & Answers

Standard error output to Pipe input - solved

Hi, I want to check a particular word is in standard error output or not. Can I acheive it in single command? For example, Delete file_name 2>error.log cat error.log Output: XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXX Successfully deleted XXXXXXXXXXXXXXXXX where delete is... (2 Replies)
Discussion started by: poova
2 Replies

10. Shell Programming and Scripting

Reading specific contents from 1 input files and appending it to another input file

Hi guys, I am new to AWK and unix scripting. Please see below my problem and let me know if anyone you can help. I have 2 input files (example given below) Input file 2 is a standard file (it will not change) and we have to get the name (second column after comma) from it and append it... (5 Replies)
Discussion started by: sksahu
5 Replies
Login or Register to Ask a Question