UNIX for Dummies Questions & Answers

Hello, my main goal is to find duplicate files, I dug up a simple code on the internet and my only question is will it work on large files and/or directories with numerous files.

Code:

#!/bin/bash  DIR="/pwd"
  for file1 in ${DIR}; do
      for file2 in ${DIR}; do
          if [ $file1 != $file2 ]; then
            DIFF=`diff "$file1" "$file2" -q`
          if [ "${DIFF%% *}" != "Files" ]; then
             echo "duplicate: $file1 $file2"
               break
          fi
          fi
      done
  done
  echo "done"

Thank you in advance !

I don't think that script's going to work. It certainly won't work on directories with lots of files in them, or work very efficiently.

I'd suggest the fdupes utility, which makes an efficient tree to compare files with.

You could do something like:

Code:

 find /dir -type f -exec md5sum {} \; | awk '{ if(cc[$1]) { print $2; } else { cc[$1]++; }}'

This would give you the name of the duplicate files (not the first, but the duplicates thereafter)
This will traverse down the whole tree from /dir - to stop it traversing child subdirs add the flag "-maxdepth 1"

@BionicMonk
The code you posted is not of the best quality. Even ignoring the mispost on line 1 and the fact that it does not work.
It will fail with large numbers of files or if any filename contains a space character. How quickly it will fail depends on the Operating System.
On further examination we realise that it does not work at all. There is no code in the script to process a list of files.

What Operating System and version do you have?
What Shell do you use?

As previous poster implied the only way to determine whether a file is a duplicate is to checksum the file. The name of the checksum program has much variation depending on what Operating System you are running.

Can you give us a feel of the highest number of files you will be checking in a particular directory tree and how much disc workspace you have to store temporary files. I have a technique which does about 20,000 files in 5-6 minutes but which is impractical for say 200,000 files.

Hmm. Before anybody spends any time on this issue, please post definitive examples of duplicate files and files which are not duplicates stating clearly how you define a "duplicate".