How to Avoid intermediate files when pipe does nt work


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to Avoid intermediate files when pipe does nt work
# 1  
Old 02-04-2009
How to Avoid intermediate files when pipe does nt work

problem with piping one output to another.Would like to avoid the intermediate file creation.The piping does nt work on places where files have been created and goes in an endless loop.


sed -e "s/^\.\///g" $LINE1| sed -e "s/_\([a-zA-Z]\)/kkk\1/g" > $file1
tr -s '_' ' ' < $file1| \
sort -n -k 1.1,1.1 -k 2.1,2.8 -k 3.1,3.6 | \
awk '{ arr[$1]=$0 } END {for (i in arr) { print arr[i] } }' | \
tr '_' ' ' | sort > $resultsfile
sed -e "s/ [ ]*/_/g" $resultsfile |sed -e "s/^_//g"|sed -e "s/_$//g"| \
grep -v "^$"|sed -e "s/kkk/_/g"| while read LINE4
do
process
done
# 2  
Old 02-04-2009
Quote:
Originally Posted by w020637
problem with piping one output to another.Would like to avoid the intermediate file creation.The piping does nt work on places where files have been created and goes in an endless loop.

Where is the "endless loop"?

When posting code, please put it inside [code] tags, and format it so that is it easily readable. I.e., put each command on a new line with indentation to display the structure of the script and to make commenting on individual commands easier.

I've reformatted it here:

Code:
sed -e "s/^\.\///g" $LINE1 |
 sed -e "s/_\([a-zA-Z]\)/kkk\1/g" > $file1

Why are you using a file? Why not pipe it directly to tr?

Code:
tr -s '_' ' ' < $file1 |
 sort -n -k 1.1,1.1 -k 2.1,2.8 -k 3.1,3.6 |
  awk '   { arr[$1]=$0 }
      END {for (i in arr) { print arr[i] } }
      ' |

If the purpose of the awk code is to remove duplicates, see below for a more efficient way to do it,
Code:
   tr '_' ' ' | sort > $resultsfile

Why are you using a file? Why not pipe it directly to sed?

Code:
sed -e "s/ [ ]*/_/g" $resultsfile |
 sed -e "s/^_//g"|
  sed -e "s/_$//g"|
   grep -v "^$"|
    sed -e "s/kkk/_/g"|

Why are you using multiple calls to sed and grep?

Code:
sed -e "s/ [ ]*/_/g" \
    -e "s/^_//g" \
    -e "s/_$//g" \
    -e "/^$/d" \
    -e "s/kkk/_/g" $resultsfile |
while read LINE4
do
  process
done

What's wrong with this:

Code:
sed -e "s/^\.\///g" \
    -e "s/_\([a-zA-Z]\)/kkk\1/g" "$LINE1" |
  sort -t_ -n -k 1,1 -k 2.1,2.8 -k 3.1,3.6 |
    awk ' { !arr[$1]++ { print } } ' |
      sort |
       sed -e 's/ [ ]*/_/g' \
           -e  '/^$/d' \
           -e 's/kkk/_/g'|
while read LINE4
do
  process
done

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Would pipe work better with this command

Hi again, have a script that I would like run, but before I can run it I need to strip out the windows \r end of lines. I have put the command into a text file and set the command to run every 10 seconds the coomand I use to do this is while sleep 10; do... (15 Replies)
Discussion started by: Paul Walker
15 Replies

2. Shell Programming and Scripting

Why does bc work with 'here string' and not via pipe in this example?

Hi! I actually got it running, but I still would like to understand, why and how, since I am a beginner in bash scripting. I Need floating numbers and thus use bc in my bash script. Here it is: #!/bin/bash num1="10^-15" | bc -l #power function piped to bc - DOES NOT WORK echo $num1... (4 Replies)
Discussion started by: McHale
4 Replies

3. Shell Programming and Scripting

How to avoid ssh :Write failed: Broken pipe?

Hello, I am trying to run some code on Matlab over ssh . The code takes around 5-6 hours to complete. so after giving the command to run it , I locked my machine and then went off to sleep at night, only to discover in the morning that I get this message : ...Code running, partial results... (1 Reply)
Discussion started by: ajayram
1 Replies

4. Shell Programming and Scripting

piping from grep to awk without intermediate files

I am trying to extract the file names alone, for example "TVLI_STATS_NRT_XLSTWS03_20120215_132629.csv", from below output which was given by the grep. sam:/data/log: grep "C10_Subscribe.000|subscribe|newfile|" PDEWG511_TVLI_JOB_STATS.ksh.201202* Output: ... (6 Replies)
Discussion started by: siteregsam
6 Replies

5. Shell Programming and Scripting

pipe to grep doesn't work in bash script

Hi, I'm trying to write a script that checks gvfs to see if a mount exists so I can run it from network-manager's status hooks. I thought I'd pipe the output of gvfs-mount -l to grep for the particular mounts I care about. When I do this in a bash script: cmnd="gvfs-mount -l | grep -i... (4 Replies)
Discussion started by: kcstrom
4 Replies

6. Shell Programming and Scripting

initializing loop to delete intermediate output files

Hi, I am running a script which produces a number of intermediate output files for each time step. is there a way to remove these intermediate files and just retain the final output at every end of the loop, like sort of an initialization process? this the inefficient way i do it. for i in... (3 Replies)
Discussion started by: ida1215
3 Replies

7. Shell Programming and Scripting

How to avoid duplication within 2 files?

Hi all, Actually 2 files are there - file1, file2. file1 contains ---> london mosco america russia mosco file2 contains --> europe india japan mosco england london Question is I want to print all the city names without duplication cities in those... (10 Replies)
Discussion started by: balan_mca
10 Replies

8. UNIX for Dummies Questions & Answers

How does pipe work?

I am confused over piping. :confused: A | B Will A and B run at the same time? or must A finish running before B starts to run? Suppose I want to do the following: sqlplus ... | split -1000 - filename_ sqlplus will return 1million rows, I want write the output into files of 1000... (4 Replies)
Discussion started by: Leion
4 Replies

9. Shell Programming and Scripting

Avoid files being archived

hi all, i want to write a shell script which can automatically touch my all files within a folder in an interval of 90 days ...so that i can avoid them being archived. I don't want to manually touch the all files instead i want an automated shell script to do this. Thanks in advance, Om (3 Replies)
Discussion started by: koti
3 Replies

10. Shell Programming and Scripting

Help needed in removing intermediate segments from a pipe delimited segment file

Hi, I just stuckup in doing some regular expressions on a file. I have data which has multiple FHS and BTS segments like: FHS|12121|LOCAL|2323 MSH|10101|POTAMAS|2323 PID|121221|THOMAS|DAVID|23432 OBX|2342|H1211|3232 BTS|0000|MERSTO|LIABLE FHS|12121|LOCAL|2323 MSH|10101|POTAMAS|2323... (3 Replies)
Discussion started by: naren_0101bits
3 Replies
Login or Register to Ask a Question