Compare files column to column based on keys Post: 302408874

Sponsored Content

Top Forums Shell Programming and Scripting Compare files column to column based on keys Post 302408874 by blackjack101 on Tuesday 30th of March 2010 05:50:30 PM

03-30-2010

Registered User

Compare files column to column based on keys

Here is my situation. I need to compare two tab separated files (diff is not useful since there could be known difference between files).

I have found similar posts , but not fully matching.I was thinking of writing a shell script using cut and grep and while loop but after going thru posts it appears awk or perl would be more appropriate. Thanks

req 1- I need to extract keys from file1 (two columns) and match with file2 based on keys. This is done by the following which I found in other posts.

Code:

 
awk -F"\t" '
    FILENAME=="f1.txt" {
        Keys[$1 $2]++
    }
    FILENAME=="f2.txt" {
        if (Keys[$1 $2] == 0) {
            print $0
        }
    }
' f1.txt f2.txt > rf.txt

req 2- match rows based on keys for all columns except keys one at a time and produce a report if column value mismatch between files. Report will be examined to ignore some known differences.

File samples -

Code:

f1.txt

210	998877	phone	9981128209	add	111 nw st.
310	998877	usg	650	ex	11
410	998877	web	1003		

f2.txt

210	998877	phone	9981128209	add	111 nw st.
310	998877	usg	650	ex	11.00
410	998877	web	1203		

report -

f2	310	998877	column6 11	11.00
f2	410	998877	column4	1003	1203

blackjack101

View Public Profile for blackjack101

Find all posts by blackjack101

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sum a column value based on multiple keys

Hi, I have below as i/p file: 5ABC 36488989 K 000010000ASB BYTRES 5PQR 45757754 K 000200005KPC HGTRET 5ABC 36488989 K 000045000ASB HGTRET 5GTH 36488989 K 000200200ASB BYTRES 5FTU ...

2. Shell Programming and Scripting

Compare Two Files(Column By Column) In Perl or shell

Hi, I am writing a comparator script, which comapre two txt files(column by column) below are the precondition of this comparator 1)columns of file are not seperated Ex. file1.txt 8888812341181892 1243548895685687 8945896789897789 1111111111111111 file2.txt 9578956789567897...

3. Shell Programming and Scripting

Nawk script to compare records of a file based on a particular column.

Hi Gurus, I am struggling with nawk command where i am processing a file based on columns. Here is the sample data file. UM113570248|24-AUG-11|4|man1|RR211 Alert: Master Process failure |24-AUG-11 UM113570624|24-AUG-11|4|man1| Alert: Pattern 'E_DCLeDAOException' found |24-AUG-11...

4. Shell Programming and Scripting

Compare 2 files and match column data and align data from 3 column

Hello experts, Please help me in achieving this in an easier way possible. I have 2 csv files with following data: File1 08/23/2012 12:35:47,JOB_5330 08/23/2012 12:35:47,JOB_5330 08/23/2012 12:36:09,JOB_5340 08/23/2012 12:36:14,JOB_5340 08/23/2012 12:36:22,JOB_5350 08/23/2012...

5. Shell Programming and Scripting

Compare based on column value

Hi Experts, I want to compare 2 text files based on their column values text1 is like prd-1234 yes no yes yes prd-2345 no no no yes prd-6475 yes yes yes no and test 2 is prd-1234 no no no yes prd-2345 yes no no no desired out put as follows prd-1234 1 3 prd-235 1 4 basically it shows...

6. Shell Programming and Scripting

Compare two files based on column

Hi, I have two files roughly 1200 fields in length for each row, sorted on the 2nd field. I need to compare based on that 2nd column between file1 and file2 and print lines that exist in both files into separate files (I can't guarantee that every line in file1 is in file2). Example: File1: ...

7. Shell Programming and Scripting

Combine multiple rows based on selected column keys

Hello I want to collapse a file with multiple rows into consolidated lines of entries based on selected columns as the 'key'. Example: 1 2 3 Abc def ghi 1 2 3 jkl mno p qrts 6 9 0 mno def Abc 7 8 4 Abc mno mno abc 7 8 9 mno mno abc 7 8 9 mno j k So if columns 1, 2 and 3 are...

8. Shell Programming and Scripting

Compare two csv's with column based

Hi, I am having below two CSV's col_1,col_2,col_3 1,2,4 1,3,6 col_1,col_3,col2,col_5,col_6 1,2,3,4,5 1,6,3,,, I need to compare based on the columns where the mismatch is expected output col_1,col_2,col_3 1,2,4

9. Shell Programming and Scripting

Need awk or Shell script to compare Column-1 of two different CSV files and print if column-1 matche

Example: I have files in below format file 1: zxc,133,joe@example.com cst,222,xyz@example1.com File 2 Contains: hxd hcd jws zxc cst File 1 has 50000 lines and file 2 has around 30000 lines : Expected Output has to be : hxd hcd jws

10. UNIX for Beginners Questions & Answers

UNIX script to compare 3rd column value with first column and display

Hello Team, My source data (INput) is like below EPIC1 router EPIC2 Targetdefinition Exp1 Expres rtr1 Router SQL SrcQual Exp1 Expres rtr1 Router EPIC1 Targetdefinition My output like SQL SrcQual Exp1 Expres Exp1 Expres rtr1 Router rtr1 Router EPIC1 Targetdefinition...

LEARN ABOUT DEBIAN

x2sys_merge

X2SYS_MERGE(1gmt)					       Generic Mapping Tools						 X2SYS_MERGE(1gmt)

NAME

       x2sys_merge - Merge an updated COEs tables

SYNOPSIS

       x2sys_merge -Amain_COElist.d -Mnew_COElist.d

DESCRIPTION

       x2sys_merge  will read two crossovers data base and output the contents of the main one updated with the COEs in the second one. The second
       file should only contain updated COEs relatively to the first one.  That is, it MUST NOT contain any new  two  tracks  intersections  (This
       point  is  NOT checked in the code). This program is useful when, for any good reason like file editing NAV correction or whatever, one had
       to recompute only the COEs between the edited files and the rest of the database.

       -A     Specify the file main_COElist.d with the main crossover error data base.

       -M     Specify the file new_COElist.d with the newly computed crossover error data base.

OPTIONS

       No space between the option flag and the associated arguments.

       EXAMPLES
	      To update the main COE_data.txt with the new COEs estimations saved in the smaller COE_fresh.txt, try

	      x2sys_merge -ACOE_data.txt -MCOE_fresh.txt > COE_updated.txt

SEE ALSO

       x2sys_binlist(1), x2sys_cross(1), x2sys_datalist(1), x2sys_get(1), x2sys_init(1), x2sys_list(1), x2sys_put(1), x2sys_report(1)

GMT 4.5.7							    15 Jul 2011 						 X2SYS_MERGE(1gmt)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sum a column value based on multiple keys

Discussion started by: nirnkv

2. Shell Programming and Scripting

Compare Two Files(Column By Column) In Perl or shell

Discussion started by: kumar96877

3. Shell Programming and Scripting

Nawk script to compare records of a file based on a particular column.

Discussion started by: usha rao

4. Shell Programming and Scripting

Compare 2 files and match column data and align data from 3 column

Discussion started by: asnandhakumar