Shell Programming and Scripting

Hi all,

i have 50 files .data should be same in these 50 files , so my task is to find the difference. i need a logic , which finds difference between all files and print in output file with file name where it found that difference .
i tried below logic , but its not giving me what i want.
let me give example
i tried this logic:-
cat $TMPDIR/*lis* | sort | egrep -ve $EXCEPTIONS_LIST_GREP | uniq -u| while read line
do
echo "Unique value found : ${line} : in file : `grep -lx "${line}" $TMPDIR/CELL*`"
aa=`grep -lx "${line}" *list*`
echo "$aa"
done >>$TMPDIR/ai see below data in 6 files which have differ by 1 file :-

cat CELL*|sort|egrep -ve HOTFIXED -e old -e B4 -e b4|sort|grep libsyncsor
1213068711 8303736 pbin/lib/libsyncsort.so
1213068711 8303736 pbin/lib/libsyncsort.so
2424480061 8303736 pbin/lib/libsyncsort.so
330122325 8303736 pbin/lib/libsyncsort.so
330122325 8303736 pbin/lib/libsyncsort.so

now above logic which i gave here , is giving 1 file only (bold 1), see the example below:-

cat CELL*|sort|egrep -ve HOTFIXED -e old -e B4 -e b4|sort|uniq -u
2424480061 8303736 pbin/lib/libsyncsort.so

echo from script :-
+ echo Unique value found : 2424480061 8303736 pbin/lib/libsyncsort.so : in file : /tmp/check_sync_20120207_231421_PID_29642/CELLTADM_LIST_2_celltadm@fra-siapp01_list

But we see three type of files in (1st column differes) , 1st-1213068711 , 2nd -2424480061 , 3rd-330122325
so it should give all these 3 difference.

actually i am comparing one directory structure on multiple servers, i copied all files name to different files (server name wise), now remaining thing to get the difference , on which server ,what file is different

Dear deepakiniimt, I'm a little confused, so please forgive me if I have missed the point. Perhaps putting code tags in your message would help clarity. It seems you may have chopped a bit out too, or i just don't "get it"

You say you have a bunch of files that you expect to be the same apart from a few differences. Could you consider the logic that if one file is the standard, then you could diff each one against it? Something like:-

Code:

for file in `ls -1 $dir/$files`
do
   echo $file
   diff standard.file $file
done > outputfile

Would that thinking suffice?

If you are looking for records that are unique in your list of files, then perhaps you should think about:You could:-

Code:

cat $dir/$files|uniq -u>/tmp/unique_records
grep -f /tmp/unique_records $dir/$files

Depending on your version of unix, you may need fgrep instead.

If this matches too many lines, then it may be that the lines are sometimes part of larger lines, eg:-

Code:

cat /tmp/unique_records
Hello said dad
Hello said mum
Hello Hello Hello
Hello

In this case, the last record would match the other three, along with any other record in any file that had the simple string "Hello". You could edit the file of unique records to make them all start with a carat symbol ^ and end with a dollar $ to explicitly signify begginning and end of line and the two ends of your string. You code would grow to:-

Code:

cat $dir/$files|uniq -u|while read line
do
   echo "\^${line}\$"
done >/tmp/unique_records
grep -f /tmp/unique_records $dir/$files

It's a bit clumsy and perhaps a sed or an awk would be quicker, just depending how much data you have.


Does that meet your needs, or have I gone off track completely?

I hope that it helps,
Robin
Liverpool/Blackburn
UK

Thanks a lot for prompt reply,
let me explain a bit more:-
1. each file is having 50K lines , though size of file is not too much.
2. i have no standard file , so that i cant compaer with standard file
3.to get unique lines first and then search again through these 50 lines is not fast , m not sure because no of lines are too much.
4. files are having 3 columns, actually this is 'cksum' value in files, so third column(file name), will be same.
well just now , i got some output ,looks meaningful :-
cat $TMPDIR/*list*|egrep -ve $EXCEPTIONS_LIST_GREP| sort | uniq -c|grep -vw ^"[ ]*5"|cut -d' ' -f 5-7|while read line
do
export aa=`grep -lx "${line}" $TMPDIR/*list*`
echo "Unique value found : ${line} : in file : $aa"
done >>$TMPDIR/aaaaaajust need to sort , i guess

output :-


Unique value found : 1213068711 8303736 pbin/lib/libsyncsort.so : in file : /tmp/check_sync_20120208_004328_PID_29046/CELLTADM_LIST_2_celltadm@avm-siapp01_list
/tmp/check_sync_20120208_004328_PID_29046/CELLTADM_LIST_2_celltadm@rpa-siapp01_list
/tmp/check_sync_20120208_004328_PID_29046/CELLTADM_LIST_2_celltadm@wlg-siapp01_list
Unique value found : 330122325 8303736 pbin/lib/libsyncsort.so : in file : /tmp/check_sync_20120208_004328_PID_29046/CELLTADM_LIST_2_celltadm@bat-siapp01_list
/tmp/check_sync_20120208_004328_PID_29046/CELLTADM_LIST_2_celltadm@rpb-siapp01_list