Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Find Duplicate files, not by name
Top Forums Shell Programming and Scripting Find Duplicate files, not by name Post 302299623 by jim mcnamara on Friday 20th of March 2009 02:27:48 PM
Old 03-20-2009
use md5 or another checksum or hash, I just used cksum:
Code:
cksum  *.jpg | sort -n > filelist

change the sort command if you use md5.

The files with identical checksums are identical. Read the file over before you go on to part 2 below:
Code:
old=""
while read sum lines filename
do
      if [[ "$sum" != "$old" ]] ; then
            old="$sum"
            continue
      fi
      rm -f "$filename"
          
done < filelist

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to find duplicate files with find ?

hello all I like to make search on files , and the result need to be the files that are duplicated? (8 Replies)
Discussion started by: umen
8 Replies

2. Shell Programming and Scripting

Find duplicate value comparing 2 files and create an output

I need a perl script which will create an output file after comparing two diff file in a directory path: /export/home/abc/file1 /export/home/abc/file2 File Format: <IP>TAB<DeviceName><TAB>DESCRIPTIONS file1: 10.1.2.1.3<tab>abc123def<tab>xyz.mm1.ppp.... (2 Replies)
Discussion started by: ricky007
2 Replies

3. Shell Programming and Scripting

Find duplicate files

What utility do you recommend for simply finding all duplicate files among all files? (4 Replies)
Discussion started by: kiasas
4 Replies

4. Shell Programming and Scripting

Find duplicate files by file size

Hi! I want to find duplicate files (criteria: file size) in my download folder. I try it like this: find /Users/frodo/Downloads \! -type d -exec du {} \; | sort > /Users/frodo/Desktop/duplicates_1.txt; cut -f 1 /Users/frodo/Desktop/duplicates_1.txt | uniq -d | grep -hif -... (9 Replies)
Discussion started by: Dirk Einecke
9 Replies

5. Shell Programming and Scripting

find duplicate string in many different files

I have more than 100 files like this: SVEAVLTGPYGYT 2 SVEGNFEETQY 10 SVELGQGYEQY 28 SVERTGTGYT 6 SVGLADYNEQF 21 SVGQGYEQY 32 SVKTVLGYEQF 2 SVNNEQF 12 SVRDGLTNSPLH 3 SVRRDREGLEQF 11 SVRTSGSYEQY 17 SVSVSGSPLQETQY 78 SVVHSTSPEAF 59 SVVPGNGYT 75 (4 Replies)
Discussion started by: xshang
4 Replies

6. Shell Programming and Scripting

Find duplicate files but with different extensions

Hi ! I wonder if anyone can help on this : I have a directory: /xyz that has the following files: chsLog.107.20130603.gz chsLog.115.20130603 chsLog.111.20130603.gz chsLog.107.20130603 chsLog.115.20130603.gz As you ca see there are two files that are the same but only with a minor... (10 Replies)
Discussion started by: fretagi
10 Replies

7. Shell Programming and Scripting

Find duplicate rows between files

Hi champs, I have one of the requirement, where I need to compare two files line by line and ignore duplicates. Note, I hav files in sorted order. I have tried using the comm command, but its not working for my scenario. Input file1 srv1..development..employee..empname,empid,empdesg... (1 Reply)
Discussion started by: Selva_2507
1 Replies

8. Shell Programming and Scripting

Find help in shell - that clears away duplicate files

I am so frustrated!!! I want a nice command that clears away duplicate files: find . -type f -regex '.*{1,3}\..*' | xargs -I## rm -v '##' should work in my opinion. But it finds nothing even though I have files that have the file name: Scooby-Doo-1.txt Himalaya-2.jpg Camping... (8 Replies)
Discussion started by: Mr.Glaurung
8 Replies

9. Shell Programming and Scripting

To Find Duplicate files using latest in Linux

I have tried the following code and with that i couldnt achieve what i want. #!/usr/bin/bash find ./ -type f \( -iname "*.xml" \) | sort -n > fileList sed -i '/\.\/fileList/d' fileList NAMEOFTHISFILE=$(echo $0|sed -e 's/\/()$*.^|/\\&/g') sed -i "/$NAMEOFTHISFILE/d"... (2 Replies)
Discussion started by: gold2k8
2 Replies

10. UNIX for Advanced & Expert Users

AIX find duplicate backup files

I would like find and delete old backup files in aix. How would I go about doing this? For example: server1_1-20-2020 server1_1-21-2020 server1_1-22-2020 server1_1-23-2020 server2_1-20-2020 server2_1-21-2020 server2_1-22-2020 server2_1-23-2020 How would I go about finding and... (3 Replies)
Discussion started by: cokedude
3 Replies

LEARN ABOUT DEBIAN

mrename

MRENAME(1)						      General Commands Manual							MRENAME(1)

NAME

       mrename - program to rename files

SYNOPSIS

       mrename 'pattern' prefix [option]

DESCRIPTION

       This  manual  page  documents  briefly the mrename command.  This manual page was written for the Debian GNU/Linux distribution because the
       original program does not have a manual page.

       mrename is a tool for easy and automatic renaming of many files.  The 'pattern' is the pattern to search files to rename (quoted  to  avoid
       that  bash  resolve  it),  and prefix is the prefix that will be added to the name of each file. The two alternative options for copying or
       moving files in the new name are explained below.  All parameters are needed, and you have to stay and launch the script in the same direc-
       tory of the files to be renamed. The program should be able to write in this directory.

OPTIONS

       There are only the following three options.

       -c     The option -c will copy each file with the new filename.

       -m     The option -m will move each file in the new filename.

       -h     Display help.

EXAMPLE

       If  you	have  a  directory  with two jpeg images prof.jpg and forp.jpg and you want to add them a prefix like item0, item1 etc..  (that is
       item0prof.jpg, item1forp.jpg etc..) do this:

	  cd /path/to/the/images
	  mrename '*.jpg' item -c

       to copy each matching file into another with the new name

	  mrename '*.jpg' item -m

       to rename each file without keeping a copy with the previous name

Word-Wide-Web:
       http://alfalinux.sourceforge.net/mrename.php3

AUTHOR
:
       Giancarlo -rofus- Erra
       e-mail: rofus@mindless.com

       This manual page was written by Dr. Guenter Bechly <gbechly@debian.org>, for the Debian GNU/Linux system (but may be used by others). It is
       distributed under the GPL just like mrename itself.

								 October  22, 2000							MRENAME(1)
All times are GMT -4. The time now is 11:05 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy