remove duplicate files in a directory | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

remove duplicate files in a directory

Shell Programming and Scripting


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 03-13-2006
asinha63 asinha63 is offline
Registered User
 
Join Date: Mar 2006
Last Activity: 30 March 2006, 11:31 AM EST
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
CPU & Memory remove duplicate files in a directory

Hi ppl.
I have to check for duplicate files in a directory .
the directory has following files
/the/folder /containing/the/file
a1.yyyymmddhhmmss
a1.yyyyMMddhhmmss
b1.yyyymmddhhmmss
b2.yyyymmddhhmmss
c.yyyymmddhhmmss
d.yyyymmddhhmmss
d.yyyymmddhhmmss

where the date time stamp can be different for the same file hence the risk of duplicate files.
How do i make the validate so that there are no duplicate files to be processed .
Anubha

Last edited by asinha63; 03-14-2006 at 09:25 AM.. Reason: to make the issue clearer
Sponsored Links
    #2  
Old 03-13-2006
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 23 April 2014, 8:57 AM EDT
Location: NM
Posts: 10,066
Thanks: 253
Thanked 760 Times in 714 Posts
PS: one directory doesn't have duplicated names...

if you run cksum or another hash like md5 you can find duplicates that way:

Code:
cd /path/to/wherever
for file in `ls *`
do
      cksum $file
done | awk '{ 
         arr[$1]++
         if(arr[$1]>1)  {print $0 }
         } ' > ./duplicate.files

Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Remove duplicate files corfuitl Shell Programming and Scripting 5 03-28-2012 05:31 AM
Remove Duplicate Files On Remote Servers jaysunn Shell Programming and Scripting 5 03-10-2010 01:27 PM
Remove duplicate files in same directory coolatt Shell Programming and Scripting 7 02-05-2010 01:17 AM
remove all duplicate lines from all files in one folder lowmaster Shell Programming and Scripting 8 05-30-2009 07:45 AM
Remove duplicate lines in log files karthikn7974 Shell Programming and Scripting 4 03-21-2009 06:41 PM



All times are GMT -4. The time now is 07:25 AM.