Hi,

I have over 5 gb of data in a files structure in which month folders are in year folders, day folders are in month folders, and individual climate stations are in each day. I am trying to extract precipitation measured at 5 minute intervals for a duration of 15 years, but the script never gets beyond the fifth year. I have run it many many times in Snow Leopard Terminal 2.1 (272). It stops at:
+ echo STIL 89 570 85 13.5 0.7 0.7 137 5.0 0.2 1.0 0.00 -999.00 1 14.4 0.0 15.6 18.2 15.4 16.5 15.8

Here is the script:
#!/bin/ksh -x
Myoutput=/MyOutput/MyOutput_$$.txt
cd ../wine
# takes output of ls command and pass to next line

ls |\

while read nextyear

do
Year=${nextyear}
cd ${nextyear}
ls |\
while read nextmonth
do
Month=${nextmonth}
cd ${nextmonth}
ls |\
while read nextday
do
Day=${nextday}
cd ${nextday}
ls | grep stil | xargs grep -i stil |\
while read nextline
do



Stid=`echo ${nextline} | awk '{print $1}'`
Time=`echo ${nextline} | awk '{print $3}'`
Rain=`echo ${nextline} | awk '{print $12}'`
echo ${Stid} ${Year} ${Month} ${Day} ${Time} ${Rain} >> $Myoutput

done


cd ..
done
cd ..
done
cd ..
done

This is how the output is supposed to look:
STIL 1994 01 01 0 0.00
STIL 1994 01 01 5 0.00
STIL 1994 01 01 10 0.00
STIL 1994 01 01 15 0.00
STIL 1994 01 01 20 0.00
STIL 1994 01 01 25 0.00
STIL 1994 01 01 30 0.00
STIL 1994 01 01 35 0.00
STIL 1994 01 01 40 0.00
STIL 1994 01 01 45 0.00
STIL 1994 01 01 50 0.00
STIL 1994 01 01 55 0.00
STIL 1994 01 01 60 0.00
STIL 1994 01 01 65 0.00
STIL 1994 01 01 70 0.00
STIL 1994 01 01 75 0.00
STIL 1994 01 01 80 0.00
STIL 1994 01 01 85 0.00
STIL 1994 01 01 90 0.00
STIL 1994 01 01 95 0.00
STIL 1994 01 01 100 0.00
STIL 1994 01 01 105 0.00
STIL 1994 01 01 110 0.00
STIL 1994 01 01 115 0.00
STIL 1994 01 01 120 0.00
STIL 1994 01 01 125 0.00
STIL 1994 01 01 130 0.00
STIL 1994 01 01 135 0.00
STIL 1994 01 01 140 0.00
STIL 1994 01 01 145 0.00
STIL 1994 01 01 150 0.00
STIL 1994 01 01 155 0.00
STIL 1994 01 01 160 0.00
STIL 1994 01 01 165 0.00
STIL 1994 01 01 170 0.00
STIL 1994 01 01 175 0.00
STIL 1994 01 01 180 0.00
STIL 1994 01 01 185 0.00
STIL 1994 01 01 190 0.00
STIL 1994 01 01 195 0.00
STIL 1994 01 01 200 0.00
STIL 1994 01 01 205 0.00
STIL 1994 01 01 210 0.00
STIL 1994 01 01 215 0.00
STIL 1994 01 01 220 0.00
STIL 1994 01 01 225 0.00
STIL 1994 01 01 230 0.00
STIL 1994 01 01 235 0.00
STIL 1994 01 01 240 0.00
STIL 1994 01 01 245 0.00
STIL 1994 01 01 250 0.00
STIL 1994 01 01 255 0.00
STIL 1994 01 01 260 0.00

Any ideas? I just need it to run through 10 more years of data!

Thanks!

You may want to

- process the years one by one, than concatenate the filed results;
- go find some freak with a more powerful machine running idle

Hi,

Please can you run the script in the background and provide us with the details of

Cheers,
Shazin