Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Compare a file with all others then print off data

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Compare a file with all others then print off data
# 1  
Old 12-29-2011
Compare a file with all others then print off data
my script so far

Code:
nawk -F, 'NR==FNR{a[$1OFS$2OFS$3]++;next} a[$1OFS$2OFS$3]{b[$1OFS$2OFS$3]++}
END{for(i in b){if(b[i]-1){print i";\t\t"b[i]}else{print "NEW:"i";\t\t1"} } }' OFS=, 20111228.csv *.csv | sort
NE:221478,SHELF:13,SLOT:4;              2
NE:221726,SHELF:8,SLOT:1;               2
NE:222318,SHELF:14,SLOT:1;              9
NE:222799,SHELF:5,SLOT:2;               2
NE:230478,SHELF:10,SLOT:6;              2
NE:252848,SHELF:3,SLOT:5;               3
NE:267909,SHELF:1,SLOT:6;               2
NE:267909,SHELF:4,SLOT:4;               4
NE:509033,SHELF:2,SLOT:1;               11
NE:509622,SHELF:12,SLOT:2;              2
NE:541469,SHELF:11,SLOT:6;              31
NE:565538,SHELF:7,SLOT:4;               15
NE:571528,SHELF:8,SLOT:2;               12
NE:593848,SHELF:3,SLOT:1;               3
NE:594728,SHELF:3,SLOT:5;               32
NE:597708,SHELF:12,SLOT:6;              2
NE:602306,SHELF:18,SLOT:6;              4
NE:800189,SHELF:8,SLOT:3;               3
NE:801048,SHELF:3,SLOT:1;               2
NE:825890,SHELF:5,SLOT:1;               2
NE:852991,SHELF:11,SLOT:4;              20
NE:877073,SHELF:3,SLOT:3;               6
NE:984177,SHELF:1,SLOT:4;               6
NEW:NE:220072,SHELF:6,SLOT:3;           1
NEW:NE:221514,SHELF:14,SLOT:1;          1
NEW:NE:230293,SHELF:6,SLOT:2;           1
NEW:NE:239160,SHELF:15,SLOT:4;          1
NEW:NE:509860,SHELF:8,SLOT:3;           1
NEW:NE:602306,SHELF:6,SLOT:3;           1
NEW:NE:801048,SHELF:6,SLOT:2;           1
NEW:NE:825890,SHELF:2,SLOT:4;           1
NEW:NE:826030,SHELF:8,SLOT:3;           1
NEW:NE:841068,SHELF:8,SLOT:4;           1
NEW:NE:854479,SHELF:6,SLOT:5;           1
NEW:NE:895435,SHELF:5,SLOT:5;           1
NEW:NE:993180,SHELF:3,SLOT:1;           1

#cat 20111228.csv
NE:221726,SHELF:8,SLOT:1,01:00:02,Wed Dec 28 2011
NE:801048,SHELF:3,SLOT:1,01:30:02,Wed Dec 28 2011
NE:841068,SHELF:8,SLOT:4,02:00:03,Wed Dec 28 2011
NE:593848,SHELF:3,SLOT:1,02:30:09,Wed Dec 28 2011
NE:801048,SHELF:6,SLOT:2,04:00:01,Wed Dec 28 2011
NE:252848,SHELF:3,SLOT:5,04:30:03,Wed Dec 28 2011
NE:852991,SHELF:11,SLOT:4,05:30:01,Wed Dec 28 2011
NE:509033,SHELF:2,SLOT:1,06:30:01,Wed Dec 28 2011
NE:509033,SHELF:2,SLOT:1,09:00:01,Wed Dec 28 2011
NE:222318,SHELF:14,SLOT:1,10:00:01,Wed Dec 28 2011
NE:267909,SHELF:4,SLOT:4,12:30:02,Wed Dec 28 2011
NE:800189,SHELF:8,SLOT:3,13:30:02,Wed Dec 28 2011
NE:509622,SHELF:12,SLOT:2,14:30:01,Wed Dec 28 2011
NE:852991,SHELF:11,SLOT:4,14:30:01,Wed Dec 28 2011
NE:602306,SHELF:6,SLOT:3,15:00:01,Wed Dec 28 2011
NE:597708,SHELF:12,SLOT:6,15:30:01,Wed Dec 28 2011
NE:239160,SHELF:15,SLOT:4,15:30:01,Wed Dec 28 2011
NE:800189,SHELF:8,SLOT:3,16:00:01,Wed Dec 28 2011
NE:825890,SHELF:5,SLOT:1,17:00:02,Wed Dec 28 2011
NE:571528,SHELF:8,SLOT:2,17:30:01,Wed Dec 28 2011
NE:594728,SHELF:3,SLOT:5,17:30:02,Wed Dec 28 2011
NE:565538,SHELF:7,SLOT:4,18:30:03,Wed Dec 28 2011
NE:825890,SHELF:2,SLOT:4,18:30:04,Wed Dec 28 2011
NE:220072,SHELF:6,SLOT:3,19:00:03,Wed Dec 28 2011
NE:826030,SHELF:8,SLOT:3,19:00:03,Wed Dec 28 2011
NE:895435,SHELF:5,SLOT:5,19:00:06,Wed Dec 28 2011
NE:222799,SHELF:5,SLOT:2,19:00:06,Wed Dec 28 2011
NE:267909,SHELF:1,SLOT:6,19:30:01,Wed Dec 28 2011
NE:800189,SHELF:8,SLOT:3,19:30:01,Wed Dec 28 2011
NE:984177,SHELF:1,SLOT:4,20:00:02,Wed Dec 28 2011
NE:230293,SHELF:6,SLOT:2,20:30:02,Wed Dec 28 2011
NE:602306,SHELF:18,SLOT:6,20:30:02,Wed Dec 28 2011
NE:541469,SHELF:11,SLOT:6,20:30:02,Wed Dec 28 2011
NE:854479,SHELF:6,SLOT:5,20:30:02,Wed Dec 28 2011
NE:825890,SHELF:5,SLOT:1,20:30:02,Wed Dec 28 2011
NE:221478,SHELF:13,SLOT:4,21:00:01,Wed Dec 28 2011
NE:252848,SHELF:3,SLOT:5,21:00:01,Wed Dec 28 2011
NE:230478,SHELF:10,SLOT:6,21:00:01,Wed Dec 28 2011
NE:509860,SHELF:8,SLOT:3,21:00:02,Wed Dec 28 2011
NE:565538,SHELF:7,SLOT:4,21:30:02,Wed Dec 28 2011
NE:993180,SHELF:3,SLOT:1,22:00:01,Wed Dec 28 2011
NE:852991,SHELF:11,SLOT:4,22:00:02,Wed Dec 28 2011
NE:221514,SHELF:14,SLOT:1,22:00:02,Wed Dec 28 2011
NE:267909,SHELF:4,SLOT:4,22:30:01,Wed Dec 28 2011
NE:877073,SHELF:3,SLOT:3,23:00:02,Wed Dec 28 2011

see existing script in first output, current script is designed to pick up line highlighted in red as a repeat but because it has not been in previous files it should therefore be a NEW with a 2 count, if you understand?

Smilie
# 2  
Old 12-29-2011
Hi llcooljatt,

I'm afraid that you will have to give us more details.

I've tested your script but output is different, did you miss any step?
Code:
$ awk -F, 'NR==FNR{a[$1OFS$2OFS$3]++;next} a[$1OFS$2OFS$3]{b[$1OFS$2OFS$3]++} END{for(i in b){if(b[i]-1){print i";\t\t"b[i]}else{print "NEW:"i";\t\t1"} } }' OFS=, 20111228.csv *.csv | sort
NE:252848,SHELF:3,SLOT:5;               2
NE:267909,SHELF:4,SLOT:4;               2
NE:509033,SHELF:2,SLOT:1;               2
NE:565538,SHELF:7,SLOT:4;               2
NE:800189,SHELF:8,SLOT:3;               3
NE:825890,SHELF:5,SLOT:1;               2
NE:852991,SHELF:11,SLOT:4;              3
NEW:NE:220072,SHELF:6,SLOT:3;           1
NEW:NE:221478,SHELF:13,SLOT:4;          1
NEW:NE:221514,SHELF:14,SLOT:1;          1
NEW:NE:221726,SHELF:8,SLOT:1;           1
NEW:NE:222318,SHELF:14,SLOT:1;          1
NEW:NE:222799,SHELF:5,SLOT:2;           1
NEW:NE:230293,SHELF:6,SLOT:2;           1
NEW:NE:230478,SHELF:10,SLOT:6;          1
NEW:NE:239160,SHELF:15,SLOT:4;          1
NEW:NE:267909,SHELF:1,SLOT:6;           1
NEW:NE:509622,SHELF:12,SLOT:2;          1
NEW:NE:509860,SHELF:8,SLOT:3;           1
NEW:NE:541469,SHELF:11,SLOT:6;          1
NEW:NE:571528,SHELF:8,SLOT:2;           1
NEW:NE:593848,SHELF:3,SLOT:1;           1
NEW:NE:594728,SHELF:3,SLOT:5;           1
NEW:NE:597708,SHELF:12,SLOT:6;          1
NEW:NE:602306,SHELF:18,SLOT:6;          1
NEW:NE:602306,SHELF:6,SLOT:3;           1
NEW:NE:801048,SHELF:3,SLOT:1;           1
NEW:NE:801048,SHELF:6,SLOT:2;           1
NEW:NE:825890,SHELF:2,SLOT:4;           1
NEW:NE:826030,SHELF:8,SLOT:3;           1
NEW:NE:841068,SHELF:8,SLOT:4;           1
NEW:NE:854479,SHELF:6,SLOT:5;           1
NEW:NE:877073,SHELF:3,SLOT:3;           1
NEW:NE:895435,SHELF:5,SLOT:5;           1
NEW:NE:984177,SHELF:1,SLOT:4;           1
NEW:NE:993180,SHELF:3,SLOT:1;           1

Regards,
Birei
# 3  
Old 12-29-2011
the output is different as I have many of *.csv files, you will not have the same
# 4  
Old 12-29-2011
Quote:
it has not been in previous files
What does this mean?

Regards,
Birei
This User Gave Thanks to birei For This Post:
llcooljatt
# 5  
Old 12-29-2011
NE:825890,SHELF:5,SLOT:1 has not been in *.csv
# 6  
Old 12-29-2011
I don't like much your approach to the problem. It is difficult to understand, and more if I can't test it.

This is my shot in the dark:
Code:
nawk -F, '
NR==FNR {
    a[$1OFS$2OFS$3]++;
    next
}

NR != FNR {
    b[$1OFS$2OFS$3]++;
}

END {
    for ( key in a ) {
        if ( b[key] ) {
            b[key]++
        } else {
            print "NEW: " key ";\t\t" a[key]
        }
    }

    for ( key in b ) {
        if ( b[key] == 1 ) {
            print "NEW: " key ";\t\t" b[key]
        } else {
            print key ";\t\t" b[key]
        }
    }
}' OFS=, 20111228.csv *.csv | sort

Regards,
Birei
This User Gave Thanks to birei For This Post:
llcooljatt
# 7  
Old 12-29-2011
I am sorry I am new to programming hence not the best at understanding.

Have you got line output please?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare 2 columns from the same file and print a value depending on the result

Hello Unix gurus, I have a file with this format (example values): label1 1 0 label2 1 0 label3 0.4 0.6 label4 0.5 0.5 label5 0.1 0.9 label6 0.9 0.1 in which: column 1 is a row label column 2 and 3 are values I would like to do a simple operation on this table and get the... (8 Replies)
Discussion started by: ksennin
8 Replies

2. Shell Programming and Scripting

Compare & print content of file which is not matching

Hi All I want to compare 2 files using awk and get output of content which is not matching I have 2 files a.txt 123 456 780 143 b.txt A|B|C|167|D|E C|K|D|123|D|E A|B|D|789|G|F C|D|G|143|A|B Not matching line from b.txt O/P A|B|C|167|D|E A|B|D|789|G|F (3 Replies)
Discussion started by: aaysa123
3 Replies

3. Shell Programming and Scripting

Compare 2 text file with 1 column in each file and write mismatch data to 3rd file

Hi, I need to compare 2 text files with around 60000 rows and 1 column. I need to compare these and write the mismatch data to 3rd file. File1 - file2 = file3 wc -l file1.txt 58112 wc -l file2.txt 55260 head -5 file1.txt 101214200123 101214700300 101250030067 101214100500... (10 Replies)
Discussion started by: Divya Nochiyil
10 Replies

4. UNIX for Dummies Questions & Answers

Compare 2 files print the lines of file 2 that contain a string from file 1

Hello I am a new unix user, and I have a work related task to compare 2 files and print all of the lines in file 2 that contain a string from file 1 Note: the fields are in different columns in the files. I suspect the is a good use for awk? Thanks for your time & help File 1 123 232 W343... (6 Replies)
Discussion started by: KevinRidley
6 Replies

5. Shell Programming and Scripting

Compare and print out data only appear in file 1 problem

Below is the data content of file_1 and file_2: file_1 >sample_1 FKGJGPOPOPOQA ASDADWEEWERE ASDAWEWQWRW ASDASDASDASDD file_2 >sample_1 DRTOWPFPOPOQA ASDADWEEASDF ASDADRTYWRW ASDASDASDASDD I got try the following perl script. Unfortunately, it can't give my desired output result... (7 Replies)
Discussion started by: patrick87
7 Replies

6. Shell Programming and Scripting

Compare two file and print same line

i want to compare two file and print same line file1 12345 a 23456 a 45678 a 45679 a file2 23456 a 34567 a 45679 a output 23456 a 45679 a any one can help me? Thank you (7 Replies)
Discussion started by: bleach8578
7 Replies

7. Shell Programming and Scripting

compare two columns of different files and print the matching second file..

Hi, I have two tab separated files; file1: S.No ddi fi cu o/l t+ t- 1 0.5 0.6 o 0.1 0.2 2 0.2 0.3 l 0.3 0.4 3 0.5 0.8 l 0.1 0.6 ... (5 Replies)
Discussion started by: vasanth.vadalur
5 Replies

8. Shell Programming and Scripting

Compare selected columns from a file and print difference

I have learned file comparison from my previous post here. Then, it is comparing the whole line. Now, i have a new problem. I have two files with 3 columns separated with a "|". What i want to do is to compare the second and third column of file 1, and the second and third column of file 2. And... (4 Replies)
Discussion started by: kingpeejay
4 Replies

9. Shell Programming and Scripting

compare 2 file and print difference in the third file URG PLS

Hi I have two files in unix. I need to compare two files and print the differed lines in other file Eg file1 1111 2222 3333 file2 1111 2222 3333 4444 5555 newfile 4444 5555 Thanks In advance (3 Replies)
Discussion started by: evvander
3 Replies

10. UNIX for Dummies Questions & Answers

Compare Data in the same file

Dear Unix-Gurus, I'm trying to write a script to compare the data in a log file. Here's how my logfile will look like: 'List All A0 Data in Destination Server' A0567 A0678 A0789 List A0 Files in Source Server A0567 A0678 A0789 So if the file match in Source Server match Destination... (1 Reply)
Discussion started by: lweegp
1 Replies
Login or Register to Ask a Question