Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Script to compare 1 file with all others

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to compare 1 file with all others
# 15  
Old 12-23-2011
ok here's my problem your script compares say two files fine but when there are multiple files it prints of wierd stuff for example:


Code:
#cat 20111224.csv
NE:852068,SHELF:1,SLOT:5,00:00:01,Sat Dec 24 2011
NE:564127,SHELF:9,SLOT:3,00:30:02,Sat Dec 24 2011
NE:225827,SHELF:8,SLOT:5,01:00:02,Sat Dec 24 2011

#nawk -F, 'NR==FNR{a[$1$2$3]++;next} !a[$1$2$3]{print "NEW: "$1,$2,$3}' OFS="," 20111224.csv *.csv
NEW: NE:851528,SHELF:3,SLOT:5
NEW: NE:598771,SHELF:9,SLOT:4
NEW: NE:828273,SHELF:2,SLOT:3
NEW: NE:219865,SHELF:4,SLOT:6
NEW: NE:237642,SHELF:12,SLOT:3
NEW: NE:848649,SHELF:4,SLOT:2
NEW: NE:851608,SHELF:5,SLOT:5
NEW: NE:219961,SHELF:5,SLOT:2
NEW: NE:877072,SHELF:2,SLOT:5
NEW: NE:222544,SHELF:15,SLOT:3
NEW: NE:560725,SHELF:1,SLOT:4
NEW: NE:560725,SHELF:1,SLOT:4
NEW: NE:851528,SHELF:3,SLOT:4
NEW: NE:858436,SHELF:4,SLOT:1
NEW: NE:221809,SHELF:12,SLOT:6
NEW: NE:220172,SHELF:1,SLOT:5
NEW: NE:560725,SHELF:13,SLOT:6
NEW: NE:565538,SHELF:6,SLOT:1
NEW: NE:220172,SHELF:1,SLOT:5
NEW: NE:594728,SHELF:2,SLOT:2
NEW: NE:505303,SHELF:3,SLOT:1
NEW: NE:571456,SHELF:5,SLOT:5
NEW: NE:560725,SHELF:16,SLOT:4
NEW: NE:220172,SHELF:1,SLOT:5
NEW: NE:860788,SHELF:2,SLOT:6


The list just goes on and on...

by 33 i mean

Code:
NE:860788,SHELF:2,SLOT:6                        33
NE:828273,SHELF:2,SLOT:3                        84


i.e matches the same text then increments the counter for that match.

really appreciate the help on this.
# 16  
Old 12-23-2011
Try this...
Code:
awk -F, 'NR==FNR{a[$1$2$3]++;next} !a[$1$2$3]{b[$1OFS$2OFS$3]++}END{for(i in b){print "NEW: "i"\t"b[i]}}' OFS="," 20111222.csv *.csv

--ahamed
Last edited by ahamed101; 12-23-2011 at 11:55 PM..
This User Gave Thanks to ahamed101 For This Post:
llcooljatt
# 17  
Old 12-23-2011
Great ahamed were getting there, so close - you are a genius !

but two problems now

1. I need only the counter for file 20111224.csv in the output & not every other .csv file (but want it to compare every other .csv file if that makes sense?)
2. I don't see any "NEW" lines with that last script

In summary

1.It needs to be a summary of 20111224.csv file comparing against all other records (with a numeric value at the end of each line for only 20111224.csv file)
2.If no match is found for a line in the 20111224.csv file comparing against all other files then the word "NEW" appears for that line
# 18  
Old 12-23-2011
Quote:
Originally Posted by llcooljatt
Great ahamed were getting there, so close - you are a genius !

but two problems now

1. I need only the counter for file 20111224.csv in the output & not every other .csv file (but want it to compare every other .csv file if that makes sense?)
2. I don't see any "NEW" lines with that last script
No, your #1 does not make sense. Because the output will not have anything from 20111224.csv, only lines not present in 20111224.csv are displayed and that is what you have asked for.

#2 - Check the previous post, I have updated it.

--ahamed

---------- Post updated at 07:59 PM ---------- Previous update was at 07:55 PM ----------

Try this..
Code:
awk -F, 'NR==FNR{a[$1OFS$2OFS$3]++;next}!a[$1OFS$2OFS$3]{b[$1OFS$2OFS$3]++}
END{for(i in b){print "NEW: "i}for(i in a){if(a[i]>0)print i"\t"a[i]}}' OFS="," 20111222.csv *.csv

Above code will eliminate the duplicates in other files. If you want to retain the duplicate, try this...

Code:
awk -F, 'NR==FNR{a[$1OFS$2OFS$3]++;next}!a[$1OFS$2OFS$3]{print "NEW: "$1,$2,$3}
END{for(i in a){if(a[i]>0)print i"\t"a[i]}}' OFS="," 20111222.csv *.csv

--ahamed
Last edited by ahamed101; 12-24-2011 at 12:05 AM..
This User Gave Thanks to ahamed101 For This Post:
llcooljatt
# 19  
Old 12-24-2011
Quote:
Originally Posted by ahamed101
No, your #1 does not make sense. Because the output will not have anything from 20111224.csv, only lines not present in 20111224.csv are displayed and that is what you have asked for.

#2 - Check the previous post, I have updated it.

--ahamed

---------- Post updated at 07:59 PM ---------- Previous update was at 07:55 PM ----------

Try this..
Code:
awk -F, 'NR==FNR{a[$1OFS$2OFS$3]++;next}!a[$1OFS$2OFS$3]{b[$1OFS$2OFS$3]++}
END{for(i in b){print "NEW: "i}for(i in a){if(a[i]>0)print i"\t"a[i]}}' OFS="," 20111222.csv *.csv




ok can I flip this around and ask that the 20111224.csv file (which is the latest data) has it's lines with numerics against them and "NEW" for no matches against all other files
# 20  
Old 12-24-2011
Check the previous post, it should give you want you want.

--ahamed
This User Gave Thanks to ahamed101 For This Post:
llcooljatt
# 21  
Old 12-24-2011
Quote:
Originally Posted by ahamed101
No, your #1 does not make sense. Because the output will not have anything from 20111224.csv, only lines not present in 20111224.csv are displayed and that is what you have asked for.

#2 - Check the previous post, I have updated it.

--ahamed

---------- Post updated at 07:59 PM ---------- Previous update was at 07:55 PM ----------

Try this..
Code:
awk -F, 'NR==FNR{a[$1OFS$2OFS$3]++;next}!a[$1OFS$2OFS$3]{b[$1OFS$2OFS$3]++}
END{for(i in b){print "NEW: "i}for(i in a){if(a[i]>0)print i"\t"a[i]}}' OFS="," 20111222.csv *.csv

Above code will eliminate the duplicates in other files. If you want to retain the duplicate, try this...

Code:
awk -F, 'NR==FNR{a[$1OFS$2OFS$3]++;next}!a[$1OFS$2OFS$3]{print "NEW: "$1,$2,$3}
END{for(i in a){if(a[i]>0)print i"\t"a[i]}}' OFS="," 20111222.csv *.csv

--ahamed




this still shows many "NEW" lines which compare against the 20111224.csv file and print them as new, I need it the other way around
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with file compare and move script

I'm running debian (the raspbian version of it) and working on a script to compare files in 2 directories, source and target, move files with duplicate names to a 3rd directory, then move remaining files in source to target. I can't get the syntax right, keep getting syntax errors and can't get... (7 Replies)
Discussion started by: mattz40
7 Replies

2. Shell Programming and Scripting

Shell script (sh file) logic to compare contents of one file with another file and output to file

Shell script logic Hi I have 2 input files like with file 1 content as (file1) "BRGTEST-242" a.txt "BRGTEST-240" a.txt "BRGTEST-219" e.txt File 2 contents as fle(2) "BRGTEST-244" a.txt "BRGTEST-244" b.txt "BRGTEST-231" c.txt "BRGTEST-231" d.txt "BRGTEST-221" e.txt I want to get... (22 Replies)
Discussion started by: pottic
22 Replies

3. Shell Programming and Scripting

Bash script to compare 2 file

Hello Friends please help me to create script to compare 2 fiile which has rpm info . File 1: glibc-2.12.1.149.el6_6.5.x86_64.rpm glibc-common-2.12-1.149.el6_6.5.x86_64.rpm File 2 : glibc-2.12.123.el6_6.5.x86_64.rpm glibc-common-2.12-123.el6_6.5.x86_64.rpm To compare file1... (1 Reply)
Discussion started by: rnary
1 Replies

4. Shell Programming and Scripting

Script to compare lines in a file

Need help to create the script that does the following : - 1. Compare the current line "column B and C" with next line "column B and C" 2. If they are the same, print output to a file Input file 2014-08-25 04:45:56.673|T1|JO|Begin|10 2014-08-25 04:55:56.673|T1|JO|Begin|11 2014-08-25... (8 Replies)
Discussion started by: chailee
8 Replies

5. Shell Programming and Scripting

script to grep a pattern from file compare contents with another file and replace

Hi All, Need help on this I have 2 files one file file1 which has several entries as : define service{ hostgroup_name !host1,!host5,!host6,.* service_description check_nrpe } define service{ hostgroup_name !host2,!host4,!host6,.* service_description check_opt } another... (2 Replies)
Discussion started by: namitai
2 Replies

6. Shell Programming and Scripting

script to compare two columns in a file

Dear everyone, I need any sort of shell script or perl script would do the following. I have a txt file as follows: ;Stretnumber Resident Resdient (not in file) 16 John Mary 16 Mary Parker 16 Nancy Smith 16 Mary John 18 Trey ... (5 Replies)
Discussion started by: sasharma
5 Replies

7. Shell Programming and Scripting

Script to compare file sizes

I need to write a bash script larger X Y that compares the sizes of two specified files X and Y, and reports which file is larger. For example, if X is larger, the output should be "File X is larger", while if Y is larger, the output should be "File Y is larger". If the files are exactly the... (3 Replies)
Discussion started by: julia_21436
3 Replies

8. Shell Programming and Scripting

Unix script to compare the two file

Hi, I want to compare two | delimited files.Awk is not working in my unix box.So plz give alternate solutions. Please see the below code: file1=$1 file2=$2 num_of_records_file1=`awk ' END { print NR } ' $file1` num_of_records_file2=`awk ' END { print NR } ' $file2` i=1 while do... (4 Replies)
Discussion started by: autosys_nm
4 Replies

9. Shell Programming and Scripting

Check File Exists and compare to previous day file script

We have data files that are ftp'd every morning to a SUN server. The file names are exactly the same except for that each has the date included in its name. I have to write script to do 2 things: STEP 1) Verify that the file arrived in morning. STEP 2) Compare the file size of the current... (3 Replies)
Discussion started by: rbknisely
3 Replies

10. Shell Programming and Scripting

compare file size from a output file from a script

Hi guys, firstly I'm working on SunOS 5.10 Generic_125100-10 sun4u sparc SUNW,Sun-Fire-V240 I've made a script to compress two directory and then send them to an other server via ftp. This is working very well. Inside theis script I decide to log usefull data for troubleshooting in case of... (7 Replies)
Discussion started by: moustik
7 Replies
Login or Register to Ask a Question