remove duplicate files in a directory | Unix Linux Forums | Shell Programming and Scripting

  Go Back    

Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

remove duplicate files in a directory

Shell Programming and Scripting

Closed Thread    
Thread Tools Search this Thread Display Modes
Old 03-13-2006
asinha63 asinha63 is offline
Registered User
Join Date: Mar 2006
Last Activity: 30 March 2006, 11:31 AM EST
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
CPU & Memory remove duplicate files in a directory

Hi ppl.
I have to check for duplicate files in a directory .
the directory has following files
/the/folder /containing/the/file

where the date time stamp can be different for the same file hence the risk of duplicate files.
How do i make the validate so that there are no duplicate files to be processed .

Last edited by asinha63; 03-14-2006 at 10:25 AM.. Reason: to make the issue clearer
Sponsored Links
Old 03-13-2006
jim mcnamara jim mcnamara is offline Forum Staff  
Join Date: Feb 2004
Last Activity: 31 January 2015, 10:45 PM EST
Location: NM
Posts: 10,360
Thanks: 315
Thanked 839 Times in 779 Posts
PS: one directory doesn't have duplicated names...

if you run cksum or another hash like md5 you can find duplicates that way:

cd /path/to/wherever
for file in `ls *`
      cksum $file
done | awk '{ 
         if(arr[$1]>1)  {print $0 }
         } ' > ./duplicate.files

Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Remove duplicate files corfuitl Shell Programming and Scripting 5 03-28-2012 06:31 AM
Remove Duplicate Files On Remote Servers jaysunn Shell Programming and Scripting 5 03-10-2010 02:27 PM
Remove duplicate files in same directory coolatt Shell Programming and Scripting 7 02-05-2010 02:17 AM
remove all duplicate lines from all files in one folder lowmaster Shell Programming and Scripting 8 05-30-2009 08:45 AM
Remove duplicate lines in log files karthikn7974 Shell Programming and Scripting 4 03-21-2009 07:41 PM

All times are GMT -4. The time now is 10:43 AM.