Sponsored Content
Top Forums Shell Programming and Scripting Find Duplicate files, not by name Post 302299623 by jim mcnamara on Friday 20th of March 2009 02:27:48 PM
Old 03-20-2009
use md5 or another checksum or hash, I just used cksum:
Code:
cksum  *.jpg | sort -n > filelist

change the sort command if you use md5.

The files with identical checksums are identical. Read the file over before you go on to part 2 below:
Code:
old=""
while read sum lines filename
do
      if [[ "$sum" != "$old" ]] ; then
            old="$sum"
            continue
      fi
      rm -f "$filename"
          
done < filelist

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to find duplicate files with find ?

hello all I like to make search on files , and the result need to be the files that are duplicated? (8 Replies)
Discussion started by: umen
8 Replies

2. Shell Programming and Scripting

Find duplicate value comparing 2 files and create an output

I need a perl script which will create an output file after comparing two diff file in a directory path: /export/home/abc/file1 /export/home/abc/file2 File Format: <IP>TAB<DeviceName><TAB>DESCRIPTIONS file1: 10.1.2.1.3<tab>abc123def<tab>xyz.mm1.ppp.... (2 Replies)
Discussion started by: ricky007
2 Replies

3. Shell Programming and Scripting

Find duplicate files

What utility do you recommend for simply finding all duplicate files among all files? (4 Replies)
Discussion started by: kiasas
4 Replies

4. Shell Programming and Scripting

Find duplicate files by file size

Hi! I want to find duplicate files (criteria: file size) in my download folder. I try it like this: find /Users/frodo/Downloads \! -type d -exec du {} \; | sort > /Users/frodo/Desktop/duplicates_1.txt; cut -f 1 /Users/frodo/Desktop/duplicates_1.txt | uniq -d | grep -hif -... (9 Replies)
Discussion started by: Dirk Einecke
9 Replies

5. Shell Programming and Scripting

find duplicate string in many different files

I have more than 100 files like this: SVEAVLTGPYGYT 2 SVEGNFEETQY 10 SVELGQGYEQY 28 SVERTGTGYT 6 SVGLADYNEQF 21 SVGQGYEQY 32 SVKTVLGYEQF 2 SVNNEQF 12 SVRDGLTNSPLH 3 SVRRDREGLEQF 11 SVRTSGSYEQY 17 SVSVSGSPLQETQY 78 SVVHSTSPEAF 59 SVVPGNGYT 75 (4 Replies)
Discussion started by: xshang
4 Replies

6. Shell Programming and Scripting

Find duplicate files but with different extensions

Hi ! I wonder if anyone can help on this : I have a directory: /xyz that has the following files: chsLog.107.20130603.gz chsLog.115.20130603 chsLog.111.20130603.gz chsLog.107.20130603 chsLog.115.20130603.gz As you ca see there are two files that are the same but only with a minor... (10 Replies)
Discussion started by: fretagi
10 Replies

7. Shell Programming and Scripting

Find duplicate rows between files

Hi champs, I have one of the requirement, where I need to compare two files line by line and ignore duplicates. Note, I hav files in sorted order. I have tried using the comm command, but its not working for my scenario. Input file1 srv1..development..employee..empname,empid,empdesg... (1 Reply)
Discussion started by: Selva_2507
1 Replies

8. Shell Programming and Scripting

Find help in shell - that clears away duplicate files

I am so frustrated!!! I want a nice command that clears away duplicate files: find . -type f -regex '.*{1,3}\..*' | xargs -I## rm -v '##' should work in my opinion. But it finds nothing even though I have files that have the file name: Scooby-Doo-1.txt Himalaya-2.jpg Camping... (8 Replies)
Discussion started by: Mr.Glaurung
8 Replies

9. Shell Programming and Scripting

To Find Duplicate files using latest in Linux

I have tried the following code and with that i couldnt achieve what i want. #!/usr/bin/bash find ./ -type f \( -iname "*.xml" \) | sort -n > fileList sed -i '/\.\/fileList/d' fileList NAMEOFTHISFILE=$(echo $0|sed -e 's/\/()$*.^|/\\&/g') sed -i "/$NAMEOFTHISFILE/d"... (2 Replies)
Discussion started by: gold2k8
2 Replies

10. UNIX for Advanced & Expert Users

AIX find duplicate backup files

I would like find and delete old backup files in aix. How would I go about doing this? For example: server1_1-20-2020 server1_1-21-2020 server1_1-22-2020 server1_1-23-2020 server2_1-20-2020 server2_1-21-2020 server2_1-22-2020 server2_1-23-2020 How would I go about finding and... (3 Replies)
Discussion started by: cokedude
3 Replies
Wanted(3pm)						User Contributed Perl Documentation					       Wanted(3pm)

NAME
File::Find::Wanted - More obvious wrapper around File::Find VERSION
Version 1.00 SYNOPSIS
File::Find is a great module, except that it doesn't actually find anything. Its "find()" function walks a directory tree and calls a callback function. Unfortunately, the callback function is deceptively called "wanted", which implies that it should return a boolean saying whether you want the file. That's not how it works. Most of the time you call "find()", you just want to build a list of files. There are other modules that do this for you, most notably Richard Clamp's great File::Find::Rule, but in many cases, it's overkill, and you need to learn a new syntax. With the "find_wanted" function, you supply a callback sub and a list of starting directories, but the sub actually should return a boolean saying whether you want the file in your list or not. To get a list of all files ending in .jpg: my @files = find_wanted( sub { -f && /.jpg$/ }, $dir ); For a list of all directories that are not CVS or .svn: my @files = find_wanted( sub { -d && !/^(CVS|.svn)$/ }, $dir ) ); It's easy, direct, and simple. WHY DO THIS
? The cynical may say "that's just the same as doing this": my @files; find( sub { push @files, $File::Find::name if -f && /.jpg$/ }, $dir ); Sure it is, but File::Find::Wanted makes it more obvious, and saves a line of code. That's worth it to me. I'd like it if find_wanted() made its way into the File::Find distro, but for now, this will do. FUNCTIONS
find_wanted( &wanted, @directories ) Descends through @directories, calling the wanted function as it finds each file. The function returns a list of all the files and directories for which the wanted function returned a true value. This is just a wrapper around "File::Find::find()". See File::Find for details on how to modify its behavior. COPYRIGHT &; LICENSE Copyright 2005-2012 Andy Lester. This program is free software; you can redistribute it and/or modify it under the terms of the Artistic License v2.0. perl v5.14.2 2012-06-08 Wanted(3pm)
All times are GMT -4. The time now is 12:02 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy