compare 2 CSV fields from same diff output


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting compare 2 CSV fields from same diff output
# 1  
Old 02-08-2012
compare 2 CSV fields from same diff output

Attached is a file called diff.txt

It is the output from this command:
diff -y --suppress-common-lines --width=5000 1.txt 2.txt > diff.txt

I have also attached 1.txt and 2.txt for your convenience.

Both 1.txt and 2.txt contain one very long CSV string.

File 1.txt is a CSV dump of one DB record before changes
File 2.txt is a CSV dump of the same DB record after it was altered
The table fields are: id, t_client, t_filelocation, t_remotescript, t_destination, t_enabled, t_search, t_compression

I want to find "exactly" (down to the charaters) of what was changed in 2.txt.

I can tell you; for your results should you take on this task; is that 2.txt contains these extra characters sequencially:
<where>newgjv<yesterday>

So what I need to do:
1. Create a report (to a file) (not to screen)
2. Format the report so that it looks like this

CHANGED RULES:
BEFORE:
19,gmp,/ureports/exportgmp/extracts,/home/dslmain/gmp/scripts/copylsoftosftp.sh,gmpsftp@sftp1Smilierod/gmp/export/extracts,YES,<where>acaact<yesterday> <where>acafee<yesterday> <wher
e>acagrp<yesterday> <where>acpos<yesterday> <where>actran<yesterday> <where>bil<yesterday><any> <where>bk<yesterday> <where>bk1<yesterday> <where>clcd<yesterday> <where>clientnam
e<yesterday> <where>corpbil<yesterday><any> <where>cqpyee<yesterday> <where>tr<yesterday> <where>tf<yesterday> <where>tt<yesterday> <where>tpcont<yesterday> <where>tpcont1<yester
day> <where>tpcont2<yesterday> <where>oldbil<yesterday><any> <where>ol<yesterday> <where>oipos<yesterday> <where>dc<yesterday><any> <where>dc2<yesterday> <where>di<yesterday> <wh
ere>dierr<yesterday> <where>espos<any><yesterday> <where>fdxref<yesterday> <where>fx<yesterday> <where>gnexch<yesterday> <where>kyfldc<yesterday> <where>kyfldd<yesterday> <where
>mfaccn<yesterday> <where>mfbr<yesterday> <where>mfcl<yesterday> <where>mfclac<yesterday> <where>mfclac1<yesterday> <where>mfclac2<yesterday> <where>mfclad1<yesterday> <where>mfc
lad2<yesterday> <where>mfclis<yesterday> <where>mfcsip<yesterday> <where>mfdoc<yesterday> <where>mfendt<yesterday> <where>mffact<yesterday> <where>mfhs<yesterday> <where>mfhsac<
yesterday> <where>mfhshd<yesterday> <where>mfinrt<yesterday> <where>mfirs<yesterday> <where>mfobj<yesterday> <where>mfrlac<yesterday> <where>mfrr<yesterday> <where>mfsc<yesterday
> <where>mfsc1<yesterday> <where>mfsc2<yesterday> <where>mfsc3<yesterday> <where>mfscdates<yesterday> <where>mfscen<yesterday> <where>mfscmf1<yesterday> <where>mfscmf2<yesterday>
<where>mfscnm<yesterday> <where>mfsymb<yesterday> <where>mm<yesterday> <where>ncadj<yesterday> <where>ncrr<yesterday> <where>ncscl<yesterday> <where>ncsplt<yesterday> <where>omf
i<any><yesterday> <where>omhist1<yesterday> <where>omhist2<yesterday> <where>omordl<yesterday> <where>omordr<yesterday> <where>omordr1<yesterday> <where>omordr2<yesterday> <where
>sm<yesterday> <where>spc<yesterday> <where>sr<yesterday> <where>sr1<yesterday> <where>sr2<yesterday> <where>gnexpd<yesterday> <where>tpcdtl<yesterday> <where>tpkptl<yesterday> <
where>jejldc<yesterday> <where>nalog<yesterday> <where>cqhead<yesterday> <where>cqform<yesterday> <where>mfasst<yesterday> <where>mfrras<yesterday> <where>tpgb<yesterday> <where>
mfcdtl<yesterday> <where>achstm<yesterday> <where>mfacct<yesterday> <where>gncode<yesterday>,YES
AFTER:
19,gmp,/ureports/exportgmp/extracts,/home/dslmain/gmp/scripts/copylsoftosftp.sh,gmpsftp@sftp1Smilierod/gmp/export/extracts,YES,<where>newgjv<yesterday> <where>acaact<yesterday> <wher
e>acafee<yesterday> <where>acagrp<yesterday> <where>acpos<yesterday> <where>actran<yesterday> <where>bil<yesterday><any> <where>bk<yesterday> <where>bk1<yesterday> <where>clcd<ye
sterday> <where>clientname<yesterday> <where>corpbil<yesterday><any> <where>cqpyee<yesterday> <where>tr<yesterday> <where>tf<yesterday> <where>tt<yesterday> <where>tpcont<yesterd
ay> <where>tpcont1<yesterday> <where>tpcont2<yesterday> <where>oldbil<yesterday><any> <where>ol<yesterday> <where>oipos<yesterday> <where>dc<yesterday><any> <where>dc2<yesterday>
<where>di<yesterday> <where>dierr<yesterday> <where>espos<any><yesterday> <where>fdxref<yesterday> <where>fx<yesterday> <where>gnexch<yesterday> <where>kyfldc<yesterday> <where>
kyfldd<yesterday> <where>mfaccn<yesterday> <where>mfbr<yesterday> <where>mfcl<yesterday> <where>mfclac<yesterday> <where>mfclac1<yesterday> <where>mfclac2<yesterday> <where>mfcl
ad1<yesterday> <where>mfclad2<yesterday> <where>mfclis<yesterday> <where>mfcsip<yesterday> <where>mfdoc<yesterday> <where>mfendt<yesterday> <where>mffact<yesterday> <where>mfhs<
yesterday> <where>mfhsac<yesterday> <where>mfhshd<yesterday> <where>mfinrt<yesterday> <where>mfirs<yesterday> <where>mfobj<yesterday> <where>mfrlac<yesterday> <where>mfrr<yesterd
ay> <where>mfsc<yesterday> <where>mfsc1<yesterday> <where>mfsc2<yesterday> <where>mfsc3<yesterday> <where>mfscdates<yesterday> <where>mfscen<yesterday> <where>mfscmf1<yesterday>
<where>mfscmf2<yesterday> <where>mfscnm<yesterday> <where>mfsymb<yesterday> <where>mm<yesterday> <where>ncadj<yesterday> <where>ncrr<yesterday> <where>ncscl<yesterday> <where>ncs
plt<yesterday> <where>omfi<any><yesterday> <where>omhist1<yesterday> <where>omhist2<yesterday> <where>omordl<yesterday> <where>omordr<yesterday> <where>omordr1<yesterday> <where>
omordr2<yesterday> <where>sm<yesterday> <where>spc<yesterday> <where>sr<yesterday> <where>sr1<yesterday> <where>sr2<yesterday> <where>gnexpd<yesterday> <where>tpcdtl<yesterday> <
where>tpkptl<yesterday> <where>jejldc<yesterday> <where>nalog<yesterday> <where>cqhead<yesterday> <where>cqform<yesterday> <where>mfasst<yesterday> <where>mfrras<yesterday> <wher
e>tpgb<yesterday> <where>mfcdtl<yesterday> <where>achstm<yesterday> <where>mfacct<yesterday> <where>gncode<yesterday>,YES
SPECIFICS:
Added to t_search: <where>newgjv<yesterday>

-----------------------------------------------------------------------
SUMMARY REPORT
-----------------
CHANGED=1
RULE: 19(gmp)
-----------------------------------------------------------------------

Report specifics:
1. You can see that for "CHANGED RULES" I need to show the entire old and new RULE and then in the SPECIFICS show exactly where the change took place (i.e. field).
2. In the Summary Report I need to show the "t_client" field after each "rule"
3. This will also need to work if data in 2.txt was removed; so if I removed say string <where>gncode<yesterday> the report should read:

CHANGED RULES:
BEFORE:
19,gmp,/ureports/exportgmp/extracts,/home/dslmain/gmp/scripts/copylsoftosftp.sh,gmpsftp@sftp1Smilierod/gmp/export/extracts,YES,<where>acaact<yesterday> <where>acafee<yesterday> <wher
e>acagrp<yesterday> <where>acpos<yesterday> <where>actran<yesterday> <where>bil<yesterday><any> <where>bk<yesterday> <where>bk1<yesterday> <where>clcd<yesterday> <where>clientnam
e<yesterday> <where>corpbil<yesterday><any> <where>cqpyee<yesterday> <where>tr<yesterday> <where>tf<yesterday> <where>tt<yesterday> <where>tpcont<yesterday> <where>tpcont1<yester
day> <where>tpcont2<yesterday> <where>oldbil<yesterday><any> <where>ol<yesterday> <where>oipos<yesterday> <where>dc<yesterday><any> <where>dc2<yesterday> <where>di<yesterday> <wh
ere>dierr<yesterday> <where>espos<any><yesterday> <where>fdxref<yesterday> <where>fx<yesterday> <where>gnexch<yesterday> <where>kyfldc<yesterday> <where>kyfldd<yesterday> <where
>mfaccn<yesterday> <where>mfbr<yesterday> <where>mfcl<yesterday> <where>mfclac<yesterday> <where>mfclac1<yesterday> <where>mfclac2<yesterday> <where>mfclad1<yesterday> <where>mfc
lad2<yesterday> <where>mfclis<yesterday> <where>mfcsip<yesterday> <where>mfdoc<yesterday> <where>mfendt<yesterday> <where>mffact<yesterday> <where>mfhs<yesterday> <where>mfhsac<
yesterday> <where>mfhshd<yesterday> <where>mfinrt<yesterday> <where>mfirs<yesterday> <where>mfobj<yesterday> <where>mfrlac<yesterday> <where>mfrr<yesterday> <where>mfsc<yesterday
> <where>mfsc1<yesterday> <where>mfsc2<yesterday> <where>mfsc3<yesterday> <where>mfscdates<yesterday> <where>mfscen<yesterday> <where>mfscmf1<yesterday> <where>mfscmf2<yesterday>
<where>mfscnm<yesterday> <where>mfsymb<yesterday> <where>mm<yesterday> <where>ncadj<yesterday> <where>ncrr<yesterday> <where>ncscl<yesterday> <where>ncsplt<yesterday> <where>omf
i<any><yesterday> <where>omhist1<yesterday> <where>omhist2<yesterday> <where>omordl<yesterday> <where>omordr<yesterday> <where>omordr1<yesterday> <where>omordr2<yesterday> <where
>sm<yesterday> <where>spc<yesterday> <where>sr<yesterday> <where>sr1<yesterday> <where>sr2<yesterday> <where>gnexpd<yesterday> <where>tpcdtl<yesterday> <where>tpkptl<yesterday> <
where>jejldc<yesterday> <where>nalog<yesterday> <where>cqhead<yesterday> <where>cqform<yesterday> <where>mfasst<yesterday> <where>mfrras<yesterday> <where>tpgb<yesterday> <where>
mfcdtl<yesterday> <where>achstm<yesterday> <where>mfacct<yesterday> <where>gncode<yesterday>,YES
AFTER:
19,gmp,/ureports/exportgmp/extracts,/home/dslmain/gmp/scripts/copylsoftosftp.sh,gmpsftp@sftp1Smilierod/gmp/export/extracts,YES,<where>newgjv<yesterday> <where>acaact<yesterday> <wher
e>acafee<yesterday> <where>acagrp<yesterday> <where>acpos<yesterday> <where>actran<yesterday> <where>bil<yesterday><any> <where>bk<yesterday> <where>bk1<yesterday> <where>clcd<ye
sterday> <where>clientname<yesterday> <where>corpbil<yesterday><any> <where>cqpyee<yesterday> <where>tr<yesterday> <where>tf<yesterday> <where>tt<yesterday> <where>tpcont<yesterd
ay> <where>tpcont1<yesterday> <where>tpcont2<yesterday> <where>oldbil<yesterday><any> <where>ol<yesterday> <where>oipos<yesterday> <where>dc<yesterday><any> <where>dc2<yesterday>
<where>di<yesterday> <where>dierr<yesterday> <where>espos<any><yesterday> <where>fdxref<yesterday> <where>fx<yesterday> <where>gnexch<yesterday> <where>kyfldc<yesterday> <where>
kyfldd<yesterday> <where>mfaccn<yesterday> <where>mfbr<yesterday> <where>mfcl<yesterday> <where>mfclac<yesterday> <where>mfclac1<yesterday> <where>mfclac2<yesterday> <where>mfcl
ad1<yesterday> <where>mfclad2<yesterday> <where>mfclis<yesterday> <where>mfcsip<yesterday> <where>mfdoc<yesterday> <where>mfendt<yesterday> <where>mffact<yesterday> <where>mfhs<
yesterday> <where>mfhsac<yesterday> <where>mfhshd<yesterday> <where>mfinrt<yesterday> <where>mfirs<yesterday> <where>mfobj<yesterday> <where>mfrlac<yesterday> <where>mfrr<yesterd
ay> <where>mfsc<yesterday> <where>mfsc1<yesterday> <where>mfsc2<yesterday> <where>mfsc3<yesterday> <where>mfscdates<yesterday> <where>mfscen<yesterday> <where>mfscmf1<yesterday>
<where>mfscmf2<yesterday> <where>mfscnm<yesterday> <where>mfsymb<yesterday> <where>mm<yesterday> <where>ncadj<yesterday> <where>ncrr<yesterday> <where>ncscl<yesterday> <where>ncs
plt<yesterday> <where>omfi<any><yesterday> <where>omhist1<yesterday> <where>omhist2<yesterday> <where>omordl<yesterday> <where>omordr<yesterday> <where>omordr1<yesterday> <where>
omordr2<yesterday> <where>sm<yesterday> <where>spc<yesterday> <where>sr<yesterday> <where>sr1<yesterday> <where>sr2<yesterday> <where>gnexpd<yesterday> <where>tpcdtl<yesterday> <
where>tpkptl<yesterday> <where>jejldc<yesterday> <where>nalog<yesterday> <where>cqhead<yesterday> <where>cqform<yesterday> <where>mfasst<yesterday> <where>mfrras<yesterday> <wher
e>tpgb<yesterday> <where>mfcdtl<yesterday> <where>achstm<yesterday> <where>mfacct<yesterday> <where>gncode<yesterday>,YES
SPECIFICS:
Removed from t_search: <where>gncode<yesterday>

Thanks
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Is there a UNIX command that can compare fields of files with differing number of fields?

Hi, Below are the sample files. x.txt is from an Excel file that is a list of users from Windows and y.txt is a list of database account. $ head -500 x.txt y.txt ==> x.txt <== TEST01 APP_USER_PROFILE USER03 APP_USER_PROFILE TEST02 APP_USER_EXP_PROFILE TEST04 APP_USER_PROFILE USER01 ... (3 Replies)
Discussion started by: newbie_01
3 Replies

2. Shell Programming and Scripting

Trying to use diff output to compare to a separate file

I have two files: smw:/working/iso_testing # cat a QConvergeConsoleCLI-1.1.03-49.x86_64.rpm aaa_base-13.2+git20140911.61c1681-1.3.i586.rpm acpica-20140724-2.1.2.i586.rpm test.rpm smw:/working/iso_testing # cat b QConvergeConsoleCLI-1.1.03-49.x86_64.rpm... (12 Replies)
Discussion started by: jedlund21
12 Replies

3. Shell Programming and Scripting

awk - compare 1st 15 fields of record with 20 fields

I'm trying to compare 2 files for differences in a selct number of fields. When differnces are found it will write the whole record of the second file including appending '|C' out to a delta file. Each record will have 20 fields, but only want to do comparison of 1st 15 fields. The 1st field of... (7 Replies)
Discussion started by: sljnk
7 Replies

4. Shell Programming and Scripting

Compare 2 csv files in ksh and o/p the difference in a new csv file

(say) I have 2 csv files - file1.csv & file2.csv as mentioned below: file1.csv ID,version,cost 1000,1,30 2000,2,40 3000,3,50 4000,4,60 file2.csv ID,version,cost 1000,1,30 2000,2,45 3000,4,55 6000,5,70 The... (7 Replies)
Discussion started by: Naresh101
7 Replies

5. Shell Programming and Scripting

Compare two CSV files and put the difference in third file with line no,field no and diff value.

I am having two csv files i need to compare these files and the output file should have the information of the differences at the field level. For Example, File 1: A,B,C,D,E,F 1,2,3,4,5,6 File 2: A,C,B,D,E,F 1,2,4,5,5,6 out put file: (12 Replies)
Discussion started by: karingulanagara
12 Replies

6. Shell Programming and Scripting

awk to compare diff output by fields

Diff output as follows: < AAA BBB CCC DDD EEE 123 > PPP QQQ RRR SSS TTT 111 > VVV WWW XXX YYY ZZZ 333 > AAA BBB CCC DDD EEE 124 How can i use awk to compare the last field to determine if the counter has increased, and need to ensure that the first 4 fields must have the same... (15 Replies)
Discussion started by: ux4me
15 Replies

7. Shell Programming and Scripting

AWK Compare files, different fields, output

Hi All, Looking for a quick AWK script to output some differences between two files. FILE1 device1 1.1.1.1 PINGS device1 2.2.2.2 PINGS FILE2 2862 SITE1 device1-prod 1.1.1.1 icmp - 0 ... (4 Replies)
Discussion started by: stacky69
4 Replies

8. Shell Programming and Scripting

Need to compare two csv files values and write into another csv file

Hi all, Am new to scripting. So i just need your ideas to help me out. Here goes my requirement. I have two csv files 1.csv 2.csv abc,1.24 abc,1 def,2.13 def,1 I need to compare the first column of 1.csv with 2.csv and if matches then need to compare... (2 Replies)
Discussion started by: chinnahyd
2 Replies

9. Shell Programming and Scripting

Compare two files and output diff to third file

I have made several attempts to read two files of ip addresses and eliminate records from file1 that are in file2. My latest attempt follows. Everything works except my file3 is exactly the same as file1 and it should not be. # !/usr/bin/bash # # NoInterfaces # Utility will create a file... (8 Replies)
Discussion started by: altamaha
8 Replies

10. Shell Programming and Scripting

diff 2 files; output diff's to 3rd file

Hello, I want to compare two files. All records in file 2 that are not in file 1 should be output to file 3. For example: file 1 123 1234 123456 file 2 123 2345 23456 file 3 should have 2345 23456 I have looked at diff, bdiff, cmp, comm, diff3 without any luck! (2 Replies)
Discussion started by: blt123
2 Replies
Login or Register to Ask a Question

Featured Tech Videos