Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Search compare and determine duplicate files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Search compare and determine duplicate files
# 1  
Old 03-27-2011
Search compare and determine duplicate files
Hi

May i ask if someone know a package that will search a directory recursively and compare determine duplicate files according to each filename, date modified or any attributes that will determine its duplicity

If none where should i start or what are those command in shell scripting that can be usefull for this task. Or such scripting language has the advantage of the others like perl is more advantage that shell scripting it just an example.

Can some one share some of there sample script for this task if they had..

thanks in advance.
# 2  
Old 03-27-2011
Some knowledge of your goals might help us in recommending packages, what are you trying to achieve with this?
# 3  
Old 03-28-2011
@chubler_XL: My goals is if there a package available or should i write a script that will accomplished the FF. search and specified directory recursively and identify the files inside for duplication. Duplicate means the same content with respect to each filename or the same filename in different directory, the same size but different filename but the same content. for example and ebook pdf reside in different directory but the same ebooks. or the ebooks pdf with the same but different filename reside in different directory.

Thanks for the reply
# 4  
Old 03-28-2011
I'm not aware of any packages that do it, how about this script to find identical files:

Code:
if [ $# -ne 1 ]
then
    echo "usage: $0 <directory>"
    exit 1
fi
find $1 -type f -ls | awk '$7 > 0 { if($7 in sizes) { sizes[$7]=sizes[$7] SUBSEP $11; dup[$7]++} else sizes[$7]=$11 } END {for(i in dup) print sizes[i] }' | while read
do
   OIFS="$IFS"
   IFS=$(printf \\034)
   F=( $REPLY )
   IFS="$OIFS"
   i=0
   while [ $i -lt ${#F[@]} ]
   do
       let j=i+1
       while [ $j -lt ${#F[@]} ]
       do
            if cmp -s "${F[i]}" "${F[j]}"
            then
                echo ${F[i]} and ${F[j]} are identical
            fi
            let j=j+1
       done
       let i=i+1
   done
done

It does a find dir -type f -ls and stores filename in an awk array with size as the index. If a file with the same size is found the size is written into the dup array. At the end all duplicate sizes are output in a SUBSEP list.

This list is read into the F array and cmp with the -s flag (ie no output, just exit status) is used to compare files - this command will stop comparing as soon as the first difference is found which is better than calculating CRCs for each file.

Note: if files A B C are identical you will get output
A is identical to B
A is identical to C
B is identical to C
This can be fixed with more logic, but I don't consider it an issue.
Last edited by Chubler_XL; 03-28-2011 at 07:20 PM..
# 5  
Old 03-28-2011
Quote:
Originally Posted by jao_madn
@chubler_XL: My goals is if there a package available or should i write a script that will accomplished the FF. search and specified directory recursively and identify the files inside for duplication. Duplicate means the same content with respect to each filename or the same filename in different directory, the same size but different filename but the same content. for example and ebook pdf reside in different directory but the same ebooks. or the ebooks pdf with the same but different filename reside in different directory.
What about fduppes
These 3 Users Gave Thanks to danmero For This Post:
jao_madn PoopNoodle pravin27
# 6  
Old 04-03-2011
Hi Chuber_XL:

Thanks for your script i now digesting your script..thanks very much..

---------- Post updated at 06:07 AM ---------- Previous update was at 05:29 AM ----------

hi chubler_xl:

your script runs very well. Can i ask additional questions or requirements on the script. the script will scan filename and compare each file with respect to their size right.. how can add in the script for which in addition of comparing the file size it will compare also or first compare the filename for the found list of files and add it on the output..for example display
dir1/filename1 and /dir1/dir2/**/filename1 are the same in filename

thanks in advance..

And also how to accomplished if i execute <find . -type f> or the same command and wants only to get the filename
Example:
dir1/dir2/filename1
dir1/filename2
dir1/dir2/dir3/filename3

Output
filename1
filename2
filename3
Last edited by jao_madn; 04-03-2011 at 07:15 PM..
# 7  
Old 04-03-2011
Glad script was usefull, see my post in This thread for an update that also supports spaces in filenames.

I Can't understand the enhancement you are are after, can you reword it please, something liike this:

Say we have 3 different contents (A, B and C)
Code:
Files with contents A:
dir1/dir2/filename1
dir1/filename2
dir1/dir2/dir3/filename3

Code:
Files with contents B:
dir1/File_one
dir2/File_two

Code:
Files with contents C:
dir3/File_one

In this situation, what would the input and output to the script be like?

Eg:
input
Code:
dir1/dir2/filename1
dir1/filename2
dir1/dir2/dir3/filename3
dir1/File_one
dir2/File_two
dir3/File_one

Output:
Code:
"dir1/dir2/filename1" and "dir1/filename2" are identical
"dir1/filename1" and "dir1/dir2/dir3/filename3" are identical
"dir1/filename2" and "dir1/dir2/dir3/filename3" are identical
"dir1/File_one" and "dir2/File_two" are identical
"dir1/File_one" and "dir3/File_one" have same name but are different

Last edited by Chubler_XL; 04-03-2011 at 07:41 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to compare two files for duplicate..??

Hi , I had a requirement to compare two files whether the two files are same or different .... like(files contaisn of two columns each) file1.txt 121343432213 1234 64564564646 2345 343423424234 2456 file2.txt 121343432213 1234 64564564646 2345 31231313123 3455 how to... (2 Replies)
Discussion started by: hemanthsaikumar
2 Replies

2. Shell Programming and Scripting

To search duplicate sequence in file

Hi, I want to search only duplicate sequence number in file e.g 4757610 4757610 should display only duplicate sequence number in file. file contain is: 4757610 6zE:EXPNL ORDER_PRIORITY='30600022004757610' ORDER_IDENTIFIER='4257771056' MM_ASK_VOLUME='273' MM_ASK_PRICE='1033.0000' m='GBX'... (5 Replies)
Discussion started by: ashfaque
5 Replies

3. Shell Programming and Scripting

Search pattern on logfile and search for day/dates and skip duplicate lines if any

Hi, I've written a script to search for an Oracle ORA- error on a log file, print that line and the .trc file associated with it as well as the dateline of when I assumed the error occured. In most it is the first dateline previous to the error. Unfortunately, this is not a fool proof script.... (2 Replies)
Discussion started by: newbie_01
2 Replies

4. UNIX for Dummies Questions & Answers

Search for string in a file then compare it with excel files entry

All, i have a file text.log: cover6 cover3 cover2 cover4 other file is abc.log as : 0 0 1 0 Then I have a excel file result.xls that contains: Name Path Pass cover2 cover3 cover6 cover4 (1 Reply)
Discussion started by: Anamika08
1 Replies

5. Shell Programming and Scripting

Search and compare files from two paths

Hi All, I have a 2 path, one with oldfile path in which has several sub folders,each sub folders contains a config file(basically text file), likewise there will be another newfile path which will have sub folders, each sub folders contains a config file. Need to read files from oldfile... (6 Replies)
Discussion started by: Optimus81
6 Replies

6. Shell Programming and Scripting

Search duplicate field and replace one of them with new value

Dear All, I have file with 4 columns: 1 AA 0 21 2 BB 0 31 3 AA 0 21 4 CC 0 41 I would like to find the duplicate record based on column 2 and replace the 4th column of the duplicate by a new value. So, the output will be: 1 AA 0 21 2 BB 0 31 3 AA 0 -21 4 CC 0 41 Any suggestions... (3 Replies)
Discussion started by: ezhil01
3 Replies

7. Shell Programming and Scripting

compare two files and search keyword and print output

You have two files to compare by searching keyword from one file into another file File A 23 >pp_ANSWER 24 >aa hello 25 >jau head wear 66 >jss oops 872 >aqq olps ploww oww sss 722 >GG_KILLER ..... large files File B Beta done KILLER John Mayor calix meyers ... (5 Replies)
Discussion started by: cdfd123
5 Replies

8. Shell Programming and Scripting

How to search & compare paragraphs between two files

Hello Guys, Greetings to All. I am stuck in my work here today while trying to comapre paragraphs between two files, I need your help on urgent basis, without your inputs I can not proceed. Kindly find some time to answer my question, I'll be grateful to you for ever. My detailed issue is as... (10 Replies)
Discussion started by: NARESH1302
10 Replies

9. Shell Programming and Scripting

compare fields in a file with duplicate records

Hi: I've been searching the net but didnt find a clue. I have a file in which, for some records, some fields coincide. I want to compare one (or more) of the dissimilar fields and retain the one record that fulfills a certain condition. For example, on this file: 99 TR 1991 5 06 ... (1 Reply)
Discussion started by: rleal
1 Replies

10. UNIX for Dummies Questions & Answers

Compare and Remove duplicate lines from txt

Hello, I am a total linux newbie and I can't seem to find a solution to this little problem. I have two text files with a huge list of URLS. Let's call them file1.txt and file2.txt What I want to do is grab an URL from file2.txt, search file1.txt for the URL and if found, delete it from... (11 Replies)
Discussion started by: rmarcano
11 Replies
Login or Register to Ask a Question