Shell Programming and Scripting

Hello,
I have two files:
file1
1
2
3
4
5
file2
"a","b",,,,,"c","1",.....
"s","d",,,,,"s","1",.....
"a","c",,,,,"d","1",....
"f","v",,,,,,"f","2",.....
etc

I have to read "1" from file1 and grab all records in file2 (say column 6 in file2 is "1") until the column changes its value ( here it changes to 2 ).
go back to file 1, get next value (2) and then grab all values in file 2 till it gets a different value (may not be 3, different than 2) etc..and continue.

I tried this with While ...do ..done...it takes lot of time due to the size of file 2.
(file2 will have 30 columns). I hope you understand my question.

this is a rough script on which i was working on. i do not want to use two while loops
while read number
do
while read column
do
var = `echo read column | cut -d -6f "," |
if $number == $var
then
do ......

done < file2

done < file1

Code:

#read first file, pull out 6th field
#place at beginning with pipe delimiter for easier retrieval
while read zf
  do
  var = $(echo "$zf" | cut -d"," -f6)
  echo "$var""|""$zf" >> file2a
done <file2

#read thru first file, creating a temp file for ultimate processing
while read yf
  do
#select the appropriate variable data
  chk=$(echo "$yf" | cut -f1)
  cat $file2 | cut "^$chk|" >file2b
done <file1

#do the processing
while read xf
   do
   ** whatever
done <$file2b

This should decrease the number of times you need to cycle-thru that large second file. Or, is a different thought-logic for trying to solve the problem at hand. (Warning - I have not tested the coding; just off the top of my head thinking thru your issue.)

I think I was not clear about my question. Let us say we have file1 and file2.

file1
1234
1235
1236
1237
1238
etc

file2

"1","1234","4","3",,,,,," 123.34","qw","as",...etc
"1","1235","4","3",,,,,," 213.34","qw","as",...etc
"1","1235","4","3",,,,,," 765.84","qw","as",...etc
"1","1236","4","3",,,,,," -123.74","qw","as",...etc
"1","1236","4","3",,,,,," 123.90","qw","as",...etc
"1","1236","4","3",,,,,," -213.34","qw","as",...etc
"1","1237","4","3",,,,,," 123.34","qw","as",...etc
"1","1237","4","3",,,,,," 123.34","qw","as",...etc
"1","1238","4","3",,,,,," 123.34","qw","as",...etc
etc

now goto file1, get first number....corr number in file 2 is column2.
grab all those records in file2 which corr to this number in column 2 of file2.
Also calculate the net amount for those records of column in bold.

Assuming file1 and file2 are sorted . (file2 based on 2nd column), start with first number in file 1 and grab all records corr to column 2 in file2. If not matched, leave file2. Now goto file1 , next number...comeback to file2 at the same point where we left earlier.(since they are sorted) and grab all records corr. to second column....continue.
I am trying something like above...pls suggest

Code:

awk 'BEGIN{FS=","}
FNR==NR{
 gsub("\"","",$2)
 gsub(/\"| +/,"",$10)
 total[$2]+=$10
 next
} 
{
  print $0,total[$0]
}
' file2 file1

thank you. i am not good at awk. but i can understand what you did.

can we write the output to a file ?
for example...in the same example it prints the totals. so can we say print the entire row from file2 into a new file say file3 if total is positive ?

I am not sure how we can redirect the output ?
And how to obtain the entire row (file2) using awk ?


thanks.

use redirection ">"

Code:

....
 {
   if ( total[$0] > 0 ){
    print $0,total[$0]  > "file3"
   }
}
....

just use $0

How to get file2 rows ?
Can I just say print $0 in above loop ?
when i do above print $0 first file rows are printing and not second file rows.
Do i need to give the print command outside the loop ?

What I mean is If total is more than zero... print all rows of file2 corr. to this number.

file2
"1","1","4","3",,,,,," 123.34"
"1","1","4","3",,,,,," 213.34"
"1","2","4","3",,,,,," -765.84"
"1","2","4","3",,,,,," -123.74"
"1","3","4","3",,,,,," 123.90"
"1","3","4","3",,,,,," -213.34"
"1","4","4","3",,,,,," -123.34"
"1","5","4","3",,,,,," 123.34"
"1","6","4","3",,,,,," -123.34"

file1
1
2
3
4
5
6

output should be like
"1","1","4","3",,,,,," 123.34"
"1","1","4","3",,,,,," 213.34"
"1","5","4","3",,,,,," 123.34"