Unix/Linux Go Back    


Shell Programming and Scripting Unix shell scripting - KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and shell scripts and shell scripting languages here.

remove duplicate files in a directory

Shell Programming and Scripting


Closed Linux or Unix Question    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 03-13-2006
asinha63 asinha63 is offline
Registered User
 
Join Date: Mar 2006
Last Activity: 30 March 2006, 11:31 AM EST
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
CPU & Memory remove duplicate files in a directory

Hi ppl.
I have to check for duplicate files in a directory .
the directory has following files
/the/folder /containing/the/file
a1.yyyymmddhhmmss
a1.yyyyMMddhhmmss
b1.yyyymmddhhmmss
b2.yyyymmddhhmmss
c.yyyymmddhhmmss
d.yyyymmddhhmmss
d.yyyymmddhhmmss

where the date time stamp can be different for the same file hence the risk of duplicate files.
How do i make the validate so that there are no duplicate files to be processed .
Anubha

Last edited by asinha63; 03-14-2006 at 09:25 AM.. Reason: to make the issue clearer
Sponsored Links
    #2  
Old Unix and Linux 03-13-2006
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 1 September 2015, 3:14 PM EDT
Location: NM
Posts: 10,531
Thanks: 353
Thanked 880 Times in 818 Posts
PS: one directory doesn't have duplicated names...

if you run cksum or another hash like md5 you can find duplicates that way:

Code:
cd /path/to/wherever
for file in `ls *`
do
      cksum $file
done | awk '{ 
         arr[$1]++
         if(arr[$1]>1)  {print $0 }
         } ' > ./duplicate.files

Sponsored Links
Closed Linux or Unix Question

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Remove duplicate files corfuitl Shell Programming and Scripting 5 03-28-2012 05:31 AM
Remove Duplicate Files On Remote Servers jaysunn Shell Programming and Scripting 5 03-10-2010 01:27 PM
Remove duplicate files in same directory coolatt Shell Programming and Scripting 7 02-05-2010 01:17 AM
remove all duplicate lines from all files in one folder lowmaster Shell Programming and Scripting 8 05-30-2009 07:45 AM
Remove duplicate lines in log files karthikn7974 Shell Programming and Scripting 4 03-21-2009 06:41 PM



All times are GMT -4. The time now is 02:26 AM.