initializing loop to delete intermediate output files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting initializing loop to delete intermediate output files
# 1  
Old 12-30-2011
initializing loop to delete intermediate output files

Hi, I am running a script which produces a number of intermediate output files for each time step. is there a way to remove these intermediate files and just retain the final output at every end of the loop, like sort of an initialization process? this the inefficient way i do it.
Code:
for i in $list; do
    command 1 > inter1.out
    command 2 > inter2.out
    command 3 > final.out
done
rm inter1.out inter2.out

thanks a lot.
# 2  
Old 12-30-2011
I believe it will be easier if you try to setup and post a sample test code that anyone could run and an example of the expected outcome.
# 3  
Old 12-30-2011
Hi, this is a sample code I'm running. the command options are from another program and I just do sort of an automation to a series of this program built-in functions using bash.
Code:
#!/bin/bash
afiles="$(ls *.nc)"
for i in $afiles; do
  ./ncgrddump $i 0001 | awk 'NR >= 13{print $2,$1,$4}' > "${i%.*}.u"
  ./ncgrddump $i 0002 | awk 'NR >= 13{print $2,$1,$4}' > "${i%.*}.v"
  nearneighbor "${i%.*}.u"  -G"${i%.*}.ugrd" -Rd -I0.25 -N8 -S35k -F  
  nearneighbor "${i%.*}.v" -G"${i%.*}.vgrd" -Rd -I0.25 -N8 -S35k -F
  grdsample "${i%.*}.ugrd" -G"${i%.*}.ures" $range2 
  grdsample "${i%.*}.vgrd" -G"${i%.*}.vres" $range2
  grdmath "${i%.*}.ures" SQR = "${i%.*}.usqr" 
  grdmath "${i%.*}.vres" SQR = "${i%.*}.vsqr"
  grdmath "${i%.*}.usqr" "${i%.*}.vsqr" ADD = "${i%.*}.sum"
  grdmath "${i%.*}.sum" 0.5 MUL = "${i%.*}.eke" 
done
rm *.ugrd *.vgrd *.usqr *.vsqr  *.sum

is there a way to incorporate the rm option within the loop? thanks much.
# 4  
Old 12-30-2011
Well,
are those programs able to read and write from and to stdin/stdout?
If yes, you could try to use variables (or pipes), instead of temporary files.
Something like this (I don't know these programs, so this is just a pseudo-code):

Code:
ures=$(
  ./ncgrddump "$i" 0001 | 
    awk 'NR >= 13 { print $2, $1, $4 }' |
      nearneighbor -Rd -I0.25 -N8 -S35k -F |
        grdsample "$range2"
      )    
grdmath <<< "$ures"  SQR = "${i%.*}.usqr"

The here-string <<< is not standard, but most modern shells support it.

Or:

Code:
  ./ncgrddump "$i" 0001 | 
    awk 'NR >= 13 { print $2, $1, $4 }' |
      nearneighbor -Rd -I0.25 -N8 -S35k -F |
        grdsample "$range2" |
          grdmath SQR = "${i%.*}.usqr"

And of course, you don't need to execute ls in order to process the files:

Code:
for f in *.nc; do ...


Last edited by radoulov; 12-30-2011 at 05:55 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

For loop to accept params and delete folder/files

Hi Folks - I'm trying to build a simple for loop to accept params and then delete the folder & files on the path older than 6 days. Here is what I have: Purge () { for _DIR in "$1" do find "${_DIR}"/* -mtime +0 -exec rm {} \; done } I would be passing... (4 Replies)
Discussion started by: SIMMS7400
4 Replies

2. Shell Programming and Scripting

For loop in bash - Direct output to two files

Hello all, i have a code in which when doing a for loop, i need to direct the output to two files, one just a single output, the other to always append (historical reasons). So far i managed to do the following, which is working, but am still considering it as "dirty". ... (4 Replies)
Discussion started by: nms
4 Replies

3. Shell Programming and Scripting

Output Multiple Files in the For Loop

while IFS= read -r line do # sV for version detection nmap -T4 -Pn -v -sS "$line" > "text/$line" done < <(grep '' $file) Hi, where line represents the IP. I am using NMAP to do scanning. How can I set to execute that command in the loop several concurrently at a time instead of one... (5 Replies)
Discussion started by: alvinoo
5 Replies

4. Shell Programming and Scripting

awk: too many output files created from while loop

I am using awk to read lines from a CSV file then put data into other files. These other files are named using the value of a certain column. Column 7 is a name such as "att" or "charter" . I want to end up with file names with the value of column 7 appended to them, like this: ... (5 Replies)
Discussion started by: dodgerfan78
5 Replies

5. Shell Programming and Scripting

piping from grep to awk without intermediate files

I am trying to extract the file names alone, for example "TVLI_STATS_NRT_XLSTWS03_20120215_132629.csv", from below output which was given by the grep. sam:/data/log: grep "C10_Subscribe.000|subscribe|newfile|" PDEWG511_TVLI_JOB_STATS.ksh.201202* Output: ... (6 Replies)
Discussion started by: siteregsam
6 Replies

6. Shell Programming and Scripting

Loop folders, delete files, copy new ones

Folks, I am hopeful that you may be able to help me out with writing a script that can be run nightly (as cron?) to loop through all subfolders within the "/media" directory, delete all of the files in each of them, and then copy in all of the files from the "/home//sansa" directory to each of... (6 Replies)
Discussion started by: acraig
6 Replies

7. Shell Programming and Scripting

How to Avoid intermediate files when pipe does nt work

problem with piping one output to another.Would like to avoid the intermediate file creation.The piping does nt work on places where files have been created and goes in an endless loop. sed -e "s/^\.\///g" $LINE1| sed -e "s/_\(\)/kkk\1/g" > $file1 tr -s '_' ' ' < $file1| \ sort -n -k... (1 Reply)
Discussion started by: w020637
1 Replies

8. UNIX for Dummies Questions & Answers

Initializing files to empty in korn shell

hello, i want to know how to initialize a file to an empty one in korn shell scripting? i'm using a file name and building it during a while loop using >>. The problem occurs when the file is not empty before reaching the while loop. therefore, i want to initialize it before the loop to get... (6 Replies)
Discussion started by: alrinno
6 Replies

9. UNIX for Dummies Questions & Answers

Re-initializing startup files without rebooting

Sorry for the newbie question. I'm using OSX BSD by remotely logging in and need to re-initialize the startup sequence but don't want to reboot the machine. How can I do it? Thanks for any help. (3 Replies)
Discussion started by: DrScar
3 Replies
Login or Register to Ask a Question