|
|
Script to compare 1 file with all others
12-24-2011
12-24-2011
12-24-2011
12-24-2011
#-F used to indicate the field separator.
#By default it is space, I am assigning "," to it, since your records are separated by comma
awk -F, '
#NR,FNR - inbuilt variable
#NR = record/line number of all the files starting from 1
#FNR = Current files record/line number
#NR==FNR means execute the code within the block only for the first file i.e. 20111222.csv
#NR=1, FNR=1 for the first line of the first file
#NR=2, FNR=2 for the first line of the first file
#NR=10, FNR=1 for the first line of the second file, got it?
NR==FNR{
#FILENAME is a inbuilt variable which will have the name of the file currently being processed
#I am stoing the filename to a variable "file"
file=FILENAME;
#Creating an associative array with first three elements separated by OFS and assigning value 1 as the key
#It will look something like this, a[NE:223334,SHELF:9,SLOT:5]=1, a[NE:22554,SHELF:9,SLOT:5]=1 etc
#There will not be any duplicate entries in this. If duplicate entries comes, the value will get incremented to 2 or 3 etc
#i.e. Lets say NE:223334,SHELF:9,SLOT:5 comes twice, then a[NE:223334,SHELF:9,SLOT:5]=2
#OFS - inbuilt variable, Output Field Separator. I have set it to , towards the end of the script.
#By default FS amd OFS will be space
a[$1 OFS $2 OFS $3]++;
#continue, i.e. do not execute the code below
next
}
#From second file onwards, the control will come here. So from the first file we have stored all the data
#into the array "a" with fields as the key
#From the second file, the first three fields will be extracted and checked if it present in the array "a" we populated,
#if it is present it will enter the code block.
a[$1 OFS $2 OFS $3]{
#Since we are giving *.csv, 20111222.csv will come again, so we need to skip it and hence this check.
if(FILENAME~file){
next
}
#If the entry is found in the array means, they are old and now we need to keep a count of that
#So I create another but similar associative array to hold the old data and the count.
#If more than one entry is detected, the count will be incremented.
b[$1 OFS $2 OFS $3]++
}
#END - inbuild keyword. This block will be processed at the last
END{
#Now we print the results.
#array "a" will have all the data from the first file, we need to find the data which is new.
#So for each key in array "a" i.e. i, varible i will have the keys from the for loop,
#it should have a value in the array "a" and should not be present in the array "b", since array "b" holds old data.
#and thus we will get the new data
for(i in a){
if(a[i] && !b[i]){
print "NEW: "i
}
}
#Now print the old data, take the keys from array "b" and print it.
#i will have the value "NE:223334,SHELF:9,SLOT:5" and b[i] will have the count
for(i in b){
if(b[i]){
print i"\t\t"b[i]
}
}
}' OFS=, 20111222.csv *.csv
12-24-2011
#-F used to indicate the field separator.
#By default it is space, I am assigning "," to it, since your records are separated by comma
awk -F, '
#NR,FNR - inbuilt variable
#NR = record/line number of all the files starting from 1
#FNR = Current files record/line number
#NR==FNR means execute the code within the block only for the first file i.e. 20111222.csv
#NR=1, FNR=1 for the first line of the first file
#NR=2, FNR=2 for the first line of the first file
#NR=10, FNR=1 for the first line of the second file, got it?
NR==FNR{
#FILENAME is a inbuilt variable which will have the name of the file currently being processed
#I am stoing the filename to a variable "file"
file=FILENAME;
#Creating an associative array with first three elements separated by OFS and assigning value 1 as the key
#It will look something like this, a[NE:223334,SHELF:9,SLOT:5]=1, a[NE:22554,SHELF:9,SLOT:5]=1 etc
#There will not be any duplicate entries in this. If duplicate entries comes, the value will get incremented to 2 or 3 etc
#i.e. Lets say NE:223334,SHELF:9,SLOT:5 comes twice, then a[NE:223334,SHELF:9,SLOT:5]=2
#OFS - inbuilt variable, Output Field Separator. I have set it to , towards the end of the script.
#By default FS amd OFS will be space
a[$1 OFS $2 OFS $3]++;
#continue, i.e. do not execute the code below
next
}
#From second file onwards, the control will come here. So from the first file we have stored all the data
#into the array "a" with fields as the key
#From the second file, the first three fields will be extracted and checked if it present in the array "a" we populated,
#if it is present it will enter the code block.
a[$1 OFS $2 OFS $3]{
#Since we are giving *.csv, 20111222.csv will come again, so we need to skip it and hence this check.
if(FILENAME~file){
next
}
#If the entry is found in the array means, they are old and now we need to keep a count of that
#So I create another but similar associative array to hold the old data and the count.
#If more than one entry is detected, the count will be incremented.
b[$1 OFS $2 OFS $3]++
}
#END - inbuild keyword. This block will be processed at the last
END{
#Now we print the results.
#array "a" will have all the data from the first file, we need to find the data which is new.
#So for each key in array "a" i.e. i, varible i will have the keys from the for loop,
#it should have a value in the array "a" and should not be present in the array "b", since array "b" holds old data.
#and thus we will get the new data
for(i in a){
if(a[i] && !b[i]){
print "NEW: "i
}
}
#Now print the old data, take the keys from array "b" and print it.
#i will have the value "NE:223334,SHELF:9,SLOT:5" and b[i] will have the count
for(i in b){
if(b[i]){
print i"\t\t"b[i]
}
}
}' OFS=, 20111222.csv *.csv
|
|
|
|