File Difference Problem


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting File Difference Problem
# 8  
Old 07-11-2011
Code:
awk '!($0 in a){a[$0];if(FILENAME=="4")print}' 3 4 > new


Last edited by radoulov; 07-11-2011 at 12:03 PM.. Reason: Code tags.
This User Gave Thanks to ltomuno For This Post:
# 9  
Old 07-11-2011
Quote:
Originally Posted by ltomuno
Code:
awk '!($0 in a){a[$0];if(FILENAME=="4")print}' 3 4 > new

That will suppress duplicate records within 4. This could be a problem if one of the new lines in 4 is identical to an older line in 3 and 4.
This User Gave Thanks to alister For This Post:
# 10  
Old 07-12-2011
Amit,
Please check below.I have take the input 3rdjuly.txt and 4thjuly.txt
as per the your very first post
The results are same as you are expecting

Input:

cat 3rdjuly.txt
Code:
~||~172.16.44.102~||~-~||~Unauthenticated~||~[17/Jun/2011:09:02:03 +0100]~||~GET /NestWeb/faces/public/MUA/pages/loginPage.xhtml HTTP/1.1~||~200~||~48583~||~-~||~Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)~||~-~||~0~||~258828~||~
~||~172.16.44.102~||~-~||~Unauthenticated~||~[17/Jun/2011:09:02:03 +0100]~||~GET /NestWeb/includes/common/css/print.css HTTP/1.1~||~200~||~-~||~https://tcsppdsw/schemeweb/NestWeb/faces/public/MUA/pages/loginPage.xhtml~||~Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)~||~org.apache.myfaces.shared_impl.context.flash.FlashImpl.POSTBACKMAP.KEY=-mmdxm8rb8; JSESSIONID=Qw5nN7JSLBgnW2LbGHJd72nMPBT1TNzP8bH13WJwpQvS4QggNm64!-63489622~||~0~||~553~||~
~||~172.16.44.102~||~-~||~Unauthenticated~||~[17/Jun/2011:09:02:03 +0100]~||~GET /NestWeb/includes/common/css/main.css HTTP/1.1~||~200~||~180945~||~https://tcsppdsw/schemeweb/NestWeb/faces/public/MUA/pages/loginPage.xhtml~||~Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)~||~org.apache.myfaces.shared_impl.context.flash.FlashImpl.POSTBACKMAP.KEY=-mmdxm8rb8; JSESSIONID=Qw5nN7JSLBgnW2LbGHJd72nMPBT1TNzP8bH13WJwpQvS4QggNm64!-63489622~||~0~||~782~||~

cat 4thjuly.txt
Code:
~||~172.16.44.102~||~-~||~Unauthenticated~||~[17/Jun/2011:09:02:03 +0100]~||~GET /NestWeb/faces/public/MUA/pages/loginPage.xhtml HTTP/1.1~||~200~||~48583~||~-~||~Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)~||~-~||~0~||~258828~||~
~||~172.16.44.102~||~-~||~Unauthenticated~||~[17/Jun/2011:09:02:03 +0100]~||~GET /NestWeb/includes/common/css/print.css HTTP/1.1~||~200~||~-~||~https://tcsppdsw/schemeweb/NestWeb/faces/public/MUA/pages/loginPage.xhtml~||~Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)~||~org.apache.myfaces.shared_impl.context.flash.FlashImpl.POSTBACKMAP.KEY=-mmdxm8rb8; JSESSIONID=Qw5nN7JSLBgnW2LbGHJd72nMPBT1TNzP8bH13WJwpQvS4QggNm64!-63489622~||~0~||~553~||~
~||~172.16.44.102~||~-~||~Unauthenticated~||~[17/Jun/2011:09:02:03 +0100]~||~GET /NestWeb/includes/common/css/main.css HTTP/1.1~||~200~||~180945~||~https://tcsppdsw/schemeweb/NestWeb/faces/public/MUA/pages/loginPage.xhtml~||~Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)~||~org.apache.myfaces.shared_impl.context.flash.FlashImpl.POSTBACKMAP.KEY=-mmdxm8rb8; JSESSIONID=Qw5nN7JSLBgnW2LbGHJd72nMPBT1TNzP8bH13WJwpQvS4QggNm64!-63489622~||~0~||~782~||~
~||~172.16.44.102~||~-~||~Unauthenticated~||~[17/Jun/2011:09:02:04 +0100]~||~GET /NestWeb/includes/MUA/css/IE.css HTTP/1.1~||~200~||~44~||~https://tcsppdsw/schemeweb/NestWeb/faces/public/MUA/pages/loginPage.xhtml~||~Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)~||~org.apache.myfaces.shared_impl.context.flash.FlashImpl.POSTBACKMAP.KEY=-mmdxm8rb8; JSESSIONID=Qw5nN7JSLBgnW2LbGHJd72nMPBT1TNzP8bH13WJwpQvS4QggNm64!-63489622~||~0~||~603~||~
~||~172.16.44.102~||~-~||~Unauthenticated~||~[17/Jun/2011:09:02:04 +0100]~||~GET /NestWeb/includes/common/js/jquery-1.4.min.js HTTP/1.1~||~200~||~69838~||~https://tcsppdsw/schemeweb/NestWeb/faces/public/MUA/pages/loginPage.xhtml~||~Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)~||~org.apache.myfaces.shared_impl.context.flash.FlashImpl.POSTBACKMAP.KEY=-mmdxm8rb8; JSESSIONID=Qw5nN7JSLBgnW2LbGHJd72nMPBT1TNzP8bH13WJwpQvS4QggNm64!-63489622~||~0~||~525~||~

Code:
diff 3rdjuly.txt 4thjuly.txt > newfile.txt

Output:

cat newfile.txt
3a4,5
> ~||~172.16.44.102~||~-~||~Unauthenticated~||~[17/Jun/2011:09:02:04 +0100]~||~GET /NestWeb/includes/MUA/css/IE.css HTTP/1.1~||~200~||~44~||~https://tcsppdsw/schemeweb/NestWeb/f...||~Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)~||~org.apache.myfaces.shared_impl.context.flash.FlashImpl.POSTBACKMAP.KEY=-mmdxm8rb8; JSESSIONID=Qw5nN7JSLBgnW2LbGHJd72nMPBT1TNzP8bH13WJwpQvS4QggNm64!-63489622~||~0~||~603~||~
> ~||~172.16.44.102~||~-~||~Unauthenticated~||~[17/Jun/2011:09:02:04 +0100]~||~GET /NestWeb/includes/common/js/jquery-1.4.min.js HTTP/1.1~||~200~||~69838~||~https://tcsppdsw/schemeweb/NestWeb/f...||~Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)~||~org.apache.myfaces.shared_impl.context.flash.FlashImpl.POSTBACKMAP.KEY=-mmdxm8rb8; JSESSIONID=Qw5nN7JSLBgnW2LbGHJd72nMPBT1TNzP8bH13WJwpQvS4QggNm64!-63489622~||~0~||~525~||~


If you are expecting anything else let me know


Cheers
Harish
This User Gave Thanks to harish612 For This Post:
# 11  
Old 07-13-2011
Amit,
You can also try with below code
But this will search the 3rdjuly.txt for each line of the 4thjuly.txt

For eg :
Even if the 4thjuly.txt line 2 and 3rdjuly.txt line 2 are not same but that line is present in 3rdjuly.txt then it won't be displayed in the output

Code:
rm -f  newfile.txt
while read line
do
  line1=`fgrep -ie "${line}" 3rdjuly.txt `
  if [ $? -ne 0 ] ; then
    echo "$line" >> newfile.txt
  fi
done <4thjuly.txt

In case you want to get the difference on each line of both the files you can refer to my above post i.e using diff command


Cheers
Harish
This User Gave Thanks to harish612 For This Post:
# 12  
Old 07-13-2011
MySQL

Thanks Alister and Harish Smilie

Its working fine Smilie cool..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk code to find difference in second file which is not present in first file .

Hi All, I want to find difference between two files and output only lines which are not present in second file .I am using awk and I am getting only the first difference but I want to get all the lines which are not present in file2 .Below is the code I am using . Please help to get the desired... (7 Replies)
Discussion started by: srinivasrao
7 Replies

2. Shell Programming and Scripting

awk to calculate difference of split and sum the difference

In the awk I am trying to subtract the difference $3-$2 of each matching $4 before the first _ (underscore) and print that value in $13. I think the awk will do that, but added comments. What I am not sure off is how to add a line or lines that will add sum each matching $13 value and put it in... (2 Replies)
Discussion started by: cmccabe
2 Replies

3. UNIX for Beginners Questions & Answers

UNIX utility to find difference in folder, file and contents of file against a base version

Hi, I am trying to find out whether there are any Unix utilities that compares folders, files and contents within the file and provides a comprehensive report. The comparison can be against base version of a folder and file with content. Can you please let me know of such a utility? Thanks,... (6 Replies)
Discussion started by: Sripathi_ks
6 Replies

4. Shell Programming and Scripting

Compare large file and identify difference in separate file

I have a very large system generated file containing around 500K rows size 100MB like following HOME|ALICE STREET|3||NEW LISTING HOME|NEWPORT STREET|1||NEW LISTING HOME|KING STREET|5||NEW LISTING HOME|WINSOME AVENUE|4||MODIFICATION CAR|TOYOTA|4||NEW LISTING CAR|FORD|4||NEW... (9 Replies)
Discussion started by: jubaier
9 Replies

5. Shell Programming and Scripting

Three Difference File Huge Data Comparison Problem.

I got three different file: Part of File 1 ARTPHDFGAA . . Part of File 2 ARTGHHYESA . . Part of File 3 ARTPOLYWEA . . (4 Replies)
Discussion started by: patrick87
4 Replies

6. Shell Programming and Scripting

Calculate the time difference between a local file and a remote file.

I m stuck with a issue. I need to calculate the time difference between two files.. one on the local machine and one on the remote machine using a script. Can any one suggest the way this can be achevied Thanks, manohar (1 Reply)
Discussion started by: meetmano143
1 Replies

7. Shell Programming and Scripting

Weird date difference problem

I am trying to find the difference in days between 2 dates. I have to extract the 1st date from a filename, which i did using the awk command. I have to compare this date to today's date and if the difference is greater than 30 days, do something, else do something else. This is what i wrote... (22 Replies)
Discussion started by: meeraKh
22 Replies

8. Shell Programming and Scripting

compare 2 file and print difference in the third file URG PLS

Hi I have two files in unix. I need to compare two files and print the differed lines in other file Eg file1 1111 2222 3333 file2 1111 2222 3333 4444 5555 newfile 4444 5555 Thanks In advance (3 Replies)
Discussion started by: evvander
3 Replies

9. Filesystems, Disks and Memory

Strange difference in file size when copying LARGE file..

Hi, Im trying to take a database backup. one of the files is 26 GB. I am using cp -pr to create a backup copy of the database. after the copying is complete, if i do du -hrs on the folders i saw a difference of 2GB. The weird fact is that the BACKUP folder was 2 GB more than the original one! ... (1 Reply)
Discussion started by: 0ktalmagik
1 Replies

10. UNIX for Dummies Questions & Answers

Newbie question about difference between executable file and ordinary file

Hi, I am newbie in unix and just started learning it. I want to know what is the difference between an executable file and a file (say text file). How to create executable file? What is the extension for that? How to differentiate ? How does it get executed? Thanks (1 Reply)
Discussion started by: Balaji
1 Replies
Login or Register to Ask a Question