remove duplicate files in a directory

asinha63 · #1 (**permalink**) 03-13-2006

Hi ppl.
I have to check for duplicate files in a directory .
the directory has following files
/the/folder /containing/the/file
a1.yyyymmddhhmmss
a1.yyyyMMddhhmmss
b1.yyyymmddhhmmss
b2.yyyymmddhhmmss
c.yyyymmddhhmmss
d.yyyymmddhhmmss
d.yyyymmddhhmmss

where the date time stamp can be different for the same file hence the risk of duplicate files.
How do i make the validate so that there are no duplicate files to be processed .
Anubha

jim mcnamara · #2 (**permalink**) 03-13-2006

PS: one directory doesn't have duplicated names...

if you run cksum or another hash like md5 you can find duplicates that way:

Code:

cd /path/to/wherever
for file in `ls *`
do
      cksum $file
done | awk '{ 
         arr[$1]++
         if(arr[$1]>1)  {print $0 }
         } ' > ./duplicate.files

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Remove duplicate lines in log files	karthikn7974	Shell Programming and Scripting	4	03-21-2009 06:41 PM
How to remove directory with subdirectories and files?	ppa108	UNIX for Dummies Questions & Answers	9	11-25-2008 02:02 AM
remove old files from a directory	2davidc8	UNIX for Dummies Questions & Answers	2	07-03-2007 02:06 PM
script that detects duplicate files in directory	trueman82	Shell Programming and Scripting	1	12-04-2006 11:58 AM
Want to remove files from unix directory	yogi_chavan	HP-UX	1	08-12-2005 04:27 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by