Comparing two huge files on field basis.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Comparing two huge files on field basis.
# 1  
Old 05-17-2011
Comparing two huge files on field basis.

Hi all,
I have two large files and i want a field by field comparison for each record in it.
All fields are tab seperated.

file1:
Code:
Email   SELVAKUMAR      RAMACHANDRAN
Email   SHILPA  SAHU
Web     NIYATI  SONI
Web     NIYATI  SONI
Email   VIINII  DOSHI
Web     RAJNISH KUMAR
Web     SUMIT   BHATI
Email   MADHURA VAIDYA
Phone   PRITI   BANE
Web     SHIVALI PATEL
Web     SANDESH PAWAN

file2:
Code:
Email   SELVAKUMAR      RAMACHANDRAN
Email   SHILPA  SAHU
Web     NIYATI  SONI
Web     NIYATI  SONI
Email   VIINII  DOSHI
Web     RAJNISH KUMAR
Web     SUMIT   BHATI
Email   MADHAVI VAIDYA
Phone   PRITI   BANE
Web     SHIVALI PATEL
Web     SANDESH PAWAR

Output :
Code:
Record no.8 is different and the field is ( field no/field value)
Record no.11 is different and the field is ( field no/field value)

or

Record no.8,11 is different and the field is ( field no/field value) , ( field no/field value) respectively.

Will be thankful if the solution comes in AWK script.


Thanks,
Suman Singh
# 2  
Old 05-17-2011
You want old value, new or both? There are never new lines?
# 3  
Old 05-17-2011
Try this if you have gnu diff available.

Code:
diff -y indj indj2  | awk '/\|/ { print "Record no."NR " value "$0 }'

# 4  
Old 05-18-2011
Both the values. both new and old one.
# 5  
Old 05-18-2011
Code:
# ./justdoit file1 file2
Record no.8 is different and the field is ( field no 2 / field value MADHAVI ) in file2
Record no.11 is different and the field is ( field no 3 / field value PAWAR ) in file2

Code:
# ./justdoit file2 file1
Record no.8 is different and the field is ( field no 2 / field value MADHURA ) in file1
Record no.11 is different and the field is ( field no 3 / field value PAWAN ) in file1

Code:
# cat justdoit
#!/bin/bash
ix=0
while read -r linef
do
compar[ix]=$linef
((ix++))
done<$1
ix=0
while read -r lineff
do
comparr[ix]=$lineff
((ix++))
done<$2
x=0;ix=0;c=0
for i in "${compar[@]}"
do
 ((c++))
if [ "$i" != "${comparr[x]}" ] ; then
 chk=(${compar[x]}) ; chk2=(${comparr[x]})
 for j in ${chk[@]}
  do
  ((fno++))
  if [ "$j" != "${chk2[ix]}" ] ; then
  echo "Record no.$c is different and the field is ( field no $fno / field value ${chk2[ix]} ) in $2"
  fi
  ((ix++))
  done
fi
((x++)) ; ix=0;fno=0
done

regards
ygemici
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Comparing two files field and position

I have two files : file1 and file2 file1 format: 7026-H70 7026-1017685 7026-H70 7026-1017687 7026-B80 7026-108D65A 7026-B80 7026-108D67A 7046-B50 7026-1034B4A File2 format : mt01cp01 7026-B80 01108D69A mt01cp02 7026-B80 01108D68A mt01sv01 7046-B50... (7 Replies)
Discussion started by: amir07
7 Replies

2. Shell Programming and Scripting

Perl: Need help comparing huge files

What do i need to do have the below perl program load 205 million record files into the hash. It currently works on smaller files, but not working on huge files. Any idea what i need to do to modify to make it work with huge files: #!/usr/bin/perl $ot1=$ARGV; $ot2=$ARGV; open(mfileot1,... (12 Replies)
Discussion started by: mrn6430
12 Replies

3. Shell Programming and Scripting

Comparing the matches in two files using awk when both files have their own field separators

I've two files with data like below: file1.txt: AAA,Apples,123 BBB,Bananas,124 CCC,Carrot,125 file2.txt: Store1|AAA|123|11 Store2|BBB|124|23 Store3|CCC|125|57 Store4|DDD|126|38 So,the field separator in file1.txt is a comma and in file2.txt,it is | Now,the output should be... (2 Replies)
Discussion started by: asyed
2 Replies

4. Shell Programming and Scripting

Comparing 2 huge text files

I have this 2 files: k5login sanwar@systems.nyfix.com jjamnik@systems.nyfix.com nisha@SYSTEMS.NYFIX.COM rdpena@SYSTEMS.NYFIX.COM service/backups-ora@SYSTEMS.NYFIX.COM ivanr@SYSTEMS.NYFIX.COM nasapova@SYSTEMS.NYFIX.COM tpulay@SYSTEMS.NYFIX.COM rsueno@SYSTEMS.NYFIX.COM... (11 Replies)
Discussion started by: linuxgeek
11 Replies

5. UNIX for Advanced & Expert Users

Sort on basis of particular field

05.50.25:AIRE.S:RESTRICTED:S2: 05.50.25:ANDR.VI:RESTRICTED:S2: 05.50.25:BASF.MI:RESTRICTED:N: 05.50.25:BMWG.DE:RESTRICTED:N: 05.50.25:BORE.ST:RESTRICTED:N: I can sort of the basis of second field using " sort -t: -k2,2 " but what i want to sort the data is on the basis its extension e.g.:... (9 Replies)
Discussion started by: manishma71
9 Replies

6. Shell Programming and Scripting

comparing files with field using awk

hi, i have 1 files a.csv temp.out a.cvs looks like add,16390,180,674X,HALIFAX_COMMONS_X,902,497,902-209 add,16390,180,674X,HALIFAX_COMMONS_X,902,497,902-219 add,16390,180,674X,HALIFAX_COMMONS_X,902,497,902-220 add,16390,180,674X,HALIFAX_COMMONS_X,902,497,902-221 and temp.out looks... (1 Reply)
Discussion started by: raghavendra.cse
1 Replies

7. Shell Programming and Scripting

Comparing two huge files

Hi, I have two files file A and File B. File A is a error file and File B is source file. In the error file. First line is the actual error and second line gives the information about the record (client ID) that throws error. I need to compare the first field (which doesnt start with '//') of... (11 Replies)
Discussion started by: kmkbuddy_1983
11 Replies

8. Shell Programming and Scripting

Field comparing in files

Guys trying to compare field in two files. For Ex: demo.txt 23.33.4.2 hostname 3.2.4.2 hostname12 demo1.txt 3.3.3.3 hostname23 45.23.23.23 hostname 323 I would like to compare the ips b/w these two files.any... (2 Replies)
Discussion started by: coolkid
2 Replies

9. Shell Programming and Scripting

Last field problem while comparing two csv files

Hi All, I've two .csv files as below file1.csv abc, tdf, 223, tpx jgsd, tex, 342, rpy a, jdjdsd, 423, djfkld Where as file2.csv is the new version of file1.csv with some added fields in the end of each line and some additional lines. lfj, eru, 98, jkldj, 39, jdkj9 abc, tdf, 223, tpx,... (3 Replies)
Discussion started by: ganapati
3 Replies

10. UNIX for Dummies Questions & Answers

comparing Huge Files - Performance is very bad

Hi All, Can you please help me in resolving the following problem? My requirement is like this: 1) I have two files YESTERDAY_FILE and TODAY_FILE. Each one is having nearly two million data. 2) I need to check each record of TODAY_FILE in YESTERDAY_FILE. If exists we can skip that by... (5 Replies)
Discussion started by: madhukalyan
5 Replies
Login or Register to Ask a Question