Unix/Linux Go Back    

Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

remove duplicate files in a directory

Shell Programming and Scripting

Thread Tools Search this Thread Display Modes
Old Unix and Linux 03-13-2006
asinha63 asinha63 is offline
Registered User
Join Date: Mar 2006
Last Activity: 30 March 2006, 11:31 AM EST
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
CPU & Memory remove duplicate files in a directory

Hi ppl.
I have to check for duplicate files in a directory .
the directory has following files
/the/folder /containing/the/file

where the date time stamp can be different for the same file hence the risk of duplicate files.
How do i make the validate so that there are no duplicate files to be processed .

Last edited by asinha63; 03-14-2006 at 09:25 AM.. Reason: to make the issue clearer
Sponsored Links
Old Unix and Linux 03-13-2006
jim mcnamara jim mcnamara is offline Forum Staff  
Join Date: Feb 2004
Last Activity: 21 October 2017, 7:54 PM EDT
Location: NM
Posts: 11,215
Thanks: 567
Thanked 1,110 Times in 1,024 Posts
PS: one directory doesn't have duplicated names...

if you run cksum or another hash like md5 you can find duplicates that way:

cd /path/to/wherever
for file in `ls *`
      cksum $file
done | awk '{ 
         if(arr[$1]>1)  {print $0 }
         } ' > ./duplicate.files

Sponsored Links

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Remove duplicate files corfuitl Shell Programming and Scripting 5 03-28-2012 05:31 AM
Remove Duplicate Files On Remote Servers jaysunn Shell Programming and Scripting 5 03-10-2010 01:27 PM
Remove duplicate files in same directory coolatt Shell Programming and Scripting 7 02-05-2010 01:17 AM
remove all duplicate lines from all files in one folder lowmaster Shell Programming and Scripting 8 05-30-2009 07:45 AM

All times are GMT -4. The time now is 04:25 AM.