Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

File Comparision

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting File Comparision
# 1  
Old 08-12-2010
File Comparision
Hi All,

I want to write a script which will compare two files and tell me if the files are different.

Actually my files will be same but order of lines will be different,so diff is not working.

I have written a script to do this:-

Code:
while read line; do
cnt=`grep -i $line /etl/symdev/hp/load/$1 | wc -l`
[ ${cnt} -le 1 ] && { echo "$line" >> /home/pkuma93/Error_log.log}
done < /etl/symdev/td/hp/load/$1

But its giving an error message Like:-

Code:
+ grep -i -12007122118835640491 P 2 4373 4373 26 2 79795 2 2 4359 4359 5 2 4359 4359 9 1069 78492 5 4373 3396 528 1 850 403 242 3063 -1 3063 1001378554 38 1 1000515551236727527 1177397160601 47.32 1 134.28 134.28 86.96 0.00 47.32 0.00 0.00 0.00 0.00 1 0 134.28 0.00 47.32 1 0.00 0 47.320000000010594994 0.000000075471949 0.00Y 2 0 14 542 2 1 0.00NM 0001N 6270 47.32 134.28 1 1 78492 210 133 210 210 210 1467405274 1001 11350733EE /etl/symdev/hp/load/HP74.LOAD.unet_phy_mkt_seg_refresh.load
grep: illegal option -- 1
grep: illegal option -- 2
grep: illegal option -- 0
grep: illegal option -- 0
grep: illegal option -- 7
grep: illegal option -- 1
grep: illegal option -- 2
grep: illegal option -- 2
grep: illegal option -- 1
grep: illegal option -- 1
grep: illegal option -- 8
grep: illegal option -- 8
grep: illegal option -- 3
grep: illegal option -- 5
grep: illegal option -- 6
grep: illegal option -- 4
grep: illegal option -- 0
grep: illegal option -- 4
grep: illegal option -- 9
grep: illegal option -- 1
usage: grep [-r] [-R] [-H] [-L] [-E|-F] [-c|-l|-q] [-insvxbhwy] [-p[parasep]] -e pattern_list...
        [-f pattern_file...] [file...]
usage: grep [-E|-F] [-c|-l|-q] [-insvxbhwy] [-p[parasep]] [-e pattern_list...]
        -f pattern_file... [file...]
usage: grep [-E|-F] [-c|-l|-q] [-insvxbhwy] [-p[parasep]] pattern_list [file...]
cnt=       0
+ echo 0
0

Can anyone help me to resolve this issue?

Thanks
Last edited by pludi; 08-12-2010 at 06:19 AM.. Reason: code tags, please...
# 2  
Old 08-12-2010
try to sort the files first, then use diff
# 3  
Old 08-12-2010
Quote:
Originally Posted by funksen
try to sort the files first, then use diff
Sorry i m new to unix,there are mutiple columns in files then how can i sort....and i want generic script for all the files which will be of different columns....
# 4  
Old 08-12-2010
an example:

Code:
# cat file1
line1 a b c
line3 c d e
line2 q l r
# cat file2
line1 a b c
line2 q l r
line3 c d e
# cat file3
line3 c d e
line2  l r
line1 a b c

without sort

Code:
# diff file1 file2
2d1
< line3 c d e
3a3
> line3 c d e

echo $? # output return code
1


with sort:

Code:
#sort file1 > file1.sort
#sort file2 >file2.sort
#diff file1.sort file2.sort
#echo $?
0




to put this in a script:

Code:
#!/bin/ksh
sort ${1} > /tmp/${1}.sort
sort ${2} > /tmp/${2}.sort

diff /tmp/${1}.sort /tmp/${2}.sort >/dev/null 2>&1
if [ $? -eq 0 ] 
then echo "files are equal"  ; RC=0
else echo "files are not equal" ; RC=1
fi

rm /tmp/${1}.sort
rm /tmp/${2}.sort 

exit $RC

usage:

Code:
# ./compare.ksh file1 file3
files are not equal
# ./compare.ksh file2 file1
files are equal

they way you sort the files doesn't matter, unless you sort both files the same way
# 5  
Old 08-26-2010
Quote:
Originally Posted by funksen
an example:

Code:
# cat file1
line1 a b c
line3 c d e
line2 q l r
# cat file2
line1 a b c
line2 q l r
line3 c d e
# cat file3
line3 c d e
line2  l r
line1 a b c

without sort

Code:
# diff file1 file2
2d1
< line3 c d e
3a3
> line3 c d e
 
echo $? # output return code
1


with sort:

Code:
#sort file1 > file1.sort
#sort file2 >file2.sort
#diff file1.sort file2.sort
#echo $?
0




to put this in a script:

Code:
#!/bin/ksh
sort ${1} > /tmp/${1}.sort
sort ${2} > /tmp/${2}.sort
 
diff /tmp/${1}.sort /tmp/${2}.sort >/dev/null 2>&1
if [ $? -eq 0 ] 
then echo "files are equal"  ; RC=0
else echo "files are not equal" ; RC=1
fi
 
rm /tmp/${1}.sort
rm /tmp/${2}.sort 
 
exit $RC

usage:

Code:
# ./compare.ksh file1 file3
files are not equal
# ./compare.ksh file2 file1
files are equal

they way you sort the files doesn't matter, unless you sort both files the same way
Hi,

I have written file comparision script using diff and sort command,and its working fine for small files but i m facing issue with the large files having size around 300MB.

Is there any other way to compare two large files?

---------- Post updated at 01:45 PM ---------- Previous update was at 12:00 PM ----------
Quote:
Originally Posted by prasson_ibm
Hi,

I have written file comparision script using diff and sort command,and its working fine for small files but i m facing issue with the large files having size around 300MB.

Is there any other way to compare two large files?
While sorting large file i m getting error message
sort: 0653-657 A write error occurred while sorting.
sort: 0653-657 A write error occurred while sorting.
# 6  
Old 08-26-2010
hm sort/diff may take longer, but there should be no write error

perhaps diff or sort temporary store information in the filesystem

check the amount of free disk space in /tmp I guess

or you have too less memory/paging space?
# 7  
Old 08-26-2010
Quote:
Originally Posted by funksen
hm sort/diff may take longer, but there should be no write error

perhaps diff or sort temporary store information in the filesystem

check the amount of free disk space in /tmp I guess

or you have too less memory/paging space?
Hi,

I have checked the space

Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 0.50 0.10 81% 5526 20% /
/dev/hd2 4.94 1.04 79% 72352 22% /usr
/dev/hd9var 0.50 0.09 82% 5159 19% /var
/dev/hd3 0.50 0.49 2% 740 1% /tmp
/dev/hd1 0.12 0.02 88% 2190 27% /home
/proc - - - - - /proc

Space is less....so is there any other alternative to do this..??
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

File comparision with AWK / SED

Hi all I need to compare two separate product lists that are changed weekly. New products are added, old products are removed and prices change. I have found various Windows programs that do this function but it's not as clean as I like and just wondered if there was a simpler way with... (1 Reply)
Discussion started by: mrpugster
1 Replies

2. Shell Programming and Scripting

File comparision

Hi All I have to files cat a.txt AAA BBB CCC DDD and cat b.txt AAA CCC EEE i want to compare these two files and o/p should have content of file a.txt which is not in file b.txt c.txt BBB DDD Please help me (3 Replies)
Discussion started by: aaysa123
3 Replies

3. Shell Programming and Scripting

File comparision

Hi All I have to files cat a.txt AAA BBB CCC DDD and cat b.txt AAA CCC EEE i want to compare these two files and o/p should have content of file a.txt which is not in file b.txt c.txt BBB DDD Please help me (1 Reply)
Discussion started by: aaysa123
1 Replies

4. Shell Programming and Scripting

Date Comparision in the File

Hi All, I have thefollowing files in the directory inbox/sat ras.sat.trn.20090103.001902.00004358 ras.sat.trn.20090612.001903.00005339 ras.sat.trn.20090723.001902.00004358 The above file contains the date of the file creation. We just need to write a ksh shell script to check the... (5 Replies)
Discussion started by: satheesh_color
5 Replies

5. Shell Programming and Scripting

File comparision

HI, I would like to know how to compare two files and replace non-matching lines with "_" . I can get non-mathing lines with grep -v -f file1 file2 i just want to knw how to display 'file2' with non-matching lines from 'file1' replaced by "_" for exmaple file1: a b c d ... (2 Replies)
Discussion started by: maddy81
2 Replies

6. Shell Programming and Scripting

Email sending attachement with two file comparision

Hi guys, I have two files . One file contains log record and another file contains emailids which is supposed to send it users File1 : Name : abc_xyz_data.txt Which contains log record File 2 : Name : abc_xyz_mailids.txt Which contains emailids. abc@test.com bcd@test.com I... (4 Replies)
Discussion started by: orabalu
4 Replies

7. Shell Programming and Scripting

file comparision by line

i have two files and i want to compare these two it shoud print those lines which are not in 2nd file a.txt 1236,AB,0 2345,CD,1 5679,EF,1 9123,AA,1 9223,AA,0 b.txt 1234,AB,0 2345,CD,1 5678,EF,1 9123,AA,0 o/p 1236,AB,0 5679,EF,1 9123,AA,1 9223,AA,0 (6 Replies)
Discussion started by: aaysa123
6 Replies

8. Shell Programming and Scripting

file size comparision local file and remote file

Hi, I have written a script which would FTP a dump file to the FTP server and log the whole activity into a file. to confirm the success of the file copy i grep for "226 file receive OK" and then send out an email saying success. Now i want to make sure the bytes of the local file and... (4 Replies)
Discussion started by: dba.admin2008
4 Replies

9. Shell Programming and Scripting

Unix File Comparision

I have a file named file1 which contains numbers in sequence like... 1 2 3 7 8 Then i have file File 2 that contains 4 5 ........so i need to compare both files so that i can find out the missing entry in the sequence is 6.......These files are flat files which contain and so i need to... (5 Replies)
Discussion started by: coolguy01
5 Replies

10. Shell Programming and Scripting

File Comparision by using Shell Script

Hello All, I am trying to find 2 file comparision by using Shell Script. For example, I am having 2 directories namely DAY1 & DAY2. DAY1 directory contains file1.dat, file2.dat, file3.dat, file4.dat, file5.dat & DAY2 directory contains file1.dat, file2.dat, file3.dat, file4.dat, file5.dat. Now,... (3 Replies)
Discussion started by: nvkuriseti
3 Replies
Login or Register to Ask a Question