Sponsored Content
Full Discussion: Find duplicate files
Top Forums Shell Programming and Scripting Find duplicate files Post 302434566 by drl on Saturday 3rd of July 2010 11:20:19 AM
Old 07-03-2010
Hi.

On many Linux systems, such as:
Code:
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0

there is a provided utility:
Code:
fdupes - finds duplicate files in a given set of directories

see man fdupes for details ... cheers, drl
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to find duplicate files with find ?

hello all I like to make search on files , and the result need to be the files that are duplicated? (8 Replies)
Discussion started by: umen
8 Replies

2. Shell Programming and Scripting

Find duplicate value comparing 2 files and create an output

I need a perl script which will create an output file after comparing two diff file in a directory path: /export/home/abc/file1 /export/home/abc/file2 File Format: <IP>TAB<DeviceName><TAB>DESCRIPTIONS file1: 10.1.2.1.3<tab>abc123def<tab>xyz.mm1.ppp.... (2 Replies)
Discussion started by: ricky007
2 Replies

3. Shell Programming and Scripting

Find Duplicate files, not by name

I have a directory with images: -rw-r--r-- 1 root root 26216 Mar 19 21:00 020109.210001.jpg -rw-r--r-- 1 root root 21760 Mar 19 21:15 020109.211502.jpg -rw-r--r-- 1 root root 23144 Mar 19 21:30 020109.213002.jpg -rw-r--r-- 1 root root 31350 Mar 20 00:45 020109.004501.jpg -rw-r--r-- 1 root... (2 Replies)
Discussion started by: Ikon
2 Replies

4. Shell Programming and Scripting

Find duplicate files by file size

Hi! I want to find duplicate files (criteria: file size) in my download folder. I try it like this: find /Users/frodo/Downloads \! -type d -exec du {} \; | sort > /Users/frodo/Desktop/duplicates_1.txt; cut -f 1 /Users/frodo/Desktop/duplicates_1.txt | uniq -d | grep -hif -... (9 Replies)
Discussion started by: Dirk Einecke
9 Replies

5. Shell Programming and Scripting

find duplicate string in many different files

I have more than 100 files like this: SVEAVLTGPYGYT 2 SVEGNFEETQY 10 SVELGQGYEQY 28 SVERTGTGYT 6 SVGLADYNEQF 21 SVGQGYEQY 32 SVKTVLGYEQF 2 SVNNEQF 12 SVRDGLTNSPLH 3 SVRRDREGLEQF 11 SVRTSGSYEQY 17 SVSVSGSPLQETQY 78 SVVHSTSPEAF 59 SVVPGNGYT 75 (4 Replies)
Discussion started by: xshang
4 Replies

6. Shell Programming and Scripting

Find duplicate files but with different extensions

Hi ! I wonder if anyone can help on this : I have a directory: /xyz that has the following files: chsLog.107.20130603.gz chsLog.115.20130603 chsLog.111.20130603.gz chsLog.107.20130603 chsLog.115.20130603.gz As you ca see there are two files that are the same but only with a minor... (10 Replies)
Discussion started by: fretagi
10 Replies

7. Shell Programming and Scripting

Find duplicate rows between files

Hi champs, I have one of the requirement, where I need to compare two files line by line and ignore duplicates. Note, I hav files in sorted order. I have tried using the comm command, but its not working for my scenario. Input file1 srv1..development..employee..empname,empid,empdesg... (1 Reply)
Discussion started by: Selva_2507
1 Replies

8. Shell Programming and Scripting

Find help in shell - that clears away duplicate files

I am so frustrated!!! I want a nice command that clears away duplicate files: find . -type f -regex '.*{1,3}\..*' | xargs -I## rm -v '##' should work in my opinion. But it finds nothing even though I have files that have the file name: Scooby-Doo-1.txt Himalaya-2.jpg Camping... (8 Replies)
Discussion started by: Mr.Glaurung
8 Replies

9. Shell Programming and Scripting

To Find Duplicate files using latest in Linux

I have tried the following code and with that i couldnt achieve what i want. #!/usr/bin/bash find ./ -type f \( -iname "*.xml" \) | sort -n > fileList sed -i '/\.\/fileList/d' fileList NAMEOFTHISFILE=$(echo $0|sed -e 's/\/()$*.^|/\\&/g') sed -i "/$NAMEOFTHISFILE/d"... (2 Replies)
Discussion started by: gold2k8
2 Replies

10. UNIX for Advanced & Expert Users

AIX find duplicate backup files

I would like find and delete old backup files in aix. How would I go about doing this? For example: server1_1-20-2020 server1_1-21-2020 server1_1-22-2020 server1_1-23-2020 server2_1-20-2020 server2_1-21-2020 server2_1-22-2020 server2_1-23-2020 How would I go about finding and... (3 Replies)
Discussion started by: cokedude
3 Replies
File::Find::Rule::Perl(3pm)				User Contributed Perl Documentation			       File::Find::Rule::Perl(3pm)

NAME
File::Find::Rule::Perl - Common rules for searching for Perl things SYNOPSIS
use File::Find::Rule (); use File::Find::Rule::Perl (); # Find all Perl files smaller than 10k my @files = File::Find::Rule->perl_file ->size('<10Ki') ->in('dir'); # Locate all the modules that PAUSE will index my @mod = File::Find::Rule->no_index ->perl_module ->in('My-Distribution'); DESCRIPTION
I write a lot of things that muck with Perl files. And it always annoyed me that finding "perl files" requires a moderately complex File::Find::Rule pattern. File::Find::Rule::Perl provides methods for finding various types Perl-related files, or replicating search queries run on a distribution in various parts of the CPAN ecosystem. METHODS
perl_module The "perl_module" rule locates perl modules. That is, files that are named "*.pm". This rule is equivalent to "->"file->name( '*.pm' )> and is included primarily for completeness. perl_test The "perl_test" rule locates perl test scripts. That is, files that are named "*.t". This rule is equivalent to "->"file->name( '*.t' )> and is included primarily for completeness. perl_installer The "perl_installer" rule locates perl distribution installers. That is, it locates "Makefile.PL" and "Build.PL" files. perl_script The "perl_script" rule locates perl scripts. This is any file that ends in .pl, or any files without extensions that have a perl "hash-bang" line. perl_file The "perl_file" rule locates all files containing Perl code. This includes all the files matching the above "perl_module", "perl_test", "perl_installer" and "perl_script" rules. no_index # Provide the rules directly $rule->no_index( directory => [ 'inc', 't', 'examples' ], file => [ 'Foo.pm', 'lib/Foo.pm' ], ); # Provide a META.yml to use $rule->no_index( 'META.yml' ); # Provide a dist root directory to look for a META.yml in $rule->no_index( 'My-Distribution' ); # Automatically pick up a META.yml from the target directory $rule->no_index->in( 'My-Distribution' ); The "no_index" method applies a set of rules as per the no_index section in a "META.yml" file. SUPPORT
Bugs should always be submitted via the CPAN bug tracker http://rt.cpan.org/NoAuth/ReportBug.html?Queue=File-Find-Rule-Perl <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=File-Find-Rule-Perl> For other issues, contact the maintainer AUTHOR
Adam Kennedy <adamk@cpan.org> SEE ALSO
<http://ali.as/>, File::Find::Rule, File::Find::Rule::PPI COPYRIGHT
Copyright 2006 - 2011 Adam Kennedy. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. The full text of the license can be found in the LICENSE file included with this module. perl v5.12.4 2011-10-31 File::Find::Rule::Perl(3pm)
All times are GMT -4. The time now is 02:40 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy