Unix/Linux Go Back    

Shell Programming and Scripting BSD, Linux, and UNIX shell scripting Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

remove duplicate files in a directory

Shell Programming and Scripting

Thread Tools Search this Thread Display Modes
Old Unix and Linux 03-13-2006
asinha63 asinha63 is offline
Registered User
Join Date: Mar 2006
Last Activity: 30 March 2006, 11:31 AM EST
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
CPU & Memory remove duplicate files in a directory

Hi ppl.
I have to check for duplicate files in a directory .
the directory has following files
/the/folder /containing/the/file

where the date time stamp can be different for the same file hence the risk of duplicate files.
How do i make the validate so that there are no duplicate files to be processed .

Last edited by asinha63; 03-14-2006 at 09:25 AM.. Reason: to make the issue clearer
Sponsored Links
Old Unix and Linux 03-13-2006
jim mcnamara jim mcnamara is offline Forum Staff  
Join Date: Feb 2004
Last Activity: 24 October 2016, 6:50 PM EDT
Location: NM
Posts: 10,839
Thanks: 451
Thanked 971 Times in 902 Posts
PS: one directory doesn't have duplicated names...

if you run cksum or another hash like md5 you can find duplicates that way:

cd /path/to/wherever
for file in `ls *`
      cksum $file
done | awk '{ 
         if(arr[$1]>1)  {print $0 }
         } ' > ./duplicate.files

Sponsored Links

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Remove duplicate files corfuitl Shell Programming and Scripting 5 03-28-2012 05:31 AM
Remove Duplicate Files On Remote Servers jaysunn Shell Programming and Scripting 5 03-10-2010 01:27 PM
Remove duplicate files in same directory coolatt Shell Programming and Scripting 7 02-05-2010 01:17 AM
remove all duplicate lines from all files in one folder lowmaster Shell Programming and Scripting 8 05-30-2009 07:45 AM

All times are GMT -4. The time now is 06:11 AM.