Sponsored Content
Full Discussion: Finds all duplicate files
Top Forums Shell Programming and Scripting Finds all duplicate files Post 302968197 by Scrutinizer on Saturday 5th of March 2016 04:25:25 AM
Old 03-05-2016
Also using checksum hashes (not a bitwise comparison), try with your favorite checksum utility (in this example shasum) :

Code:
find /some/dir -type f -exec shasum {} + |
awk '{i=$1; $1=x; C[i]++; A[i]=A[i] $0 FS} END{for(i in C) if(C[i]>1) print A[i]}'

This assumes that duplicates may also be found in sub directories...
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

PS finds a ghost?

Hello, I have problems executing a script in ksh with this script named process.sh: ps -ef | grep process.sh | grep -v grep | wc -l | read a if then echo "The script is running" exit 0 fiThe problem is that when I execute the script, sometimes it shows the message "The script is... (10 Replies)
Discussion started by: jguirao
10 Replies

2. Shell Programming and Scripting

getting rid of duplicate files

i have a bad problem with multiple occurances of the same file in different directories.. how this happened i am not sure! but I know that i can use awk to scan multiple directory trees to find an occurance of the same file... some of these files differ somwhat but that does not matter! the... (4 Replies)
Discussion started by: moxxx68
4 Replies

3. Shell Programming and Scripting

Finding Duplicate files

How do you delete and and find duplicate files? (1 Reply)
Discussion started by: Jicom4
1 Replies

4. Shell Programming and Scripting

Find Duplicate files, not by name

I have a directory with images: -rw-r--r-- 1 root root 26216 Mar 19 21:00 020109.210001.jpg -rw-r--r-- 1 root root 21760 Mar 19 21:15 020109.211502.jpg -rw-r--r-- 1 root root 23144 Mar 19 21:30 020109.213002.jpg -rw-r--r-- 1 root root 31350 Mar 20 00:45 020109.004501.jpg -rw-r--r-- 1 root... (2 Replies)
Discussion started by: Ikon
2 Replies

5. Shell Programming and Scripting

Find duplicate files

What utility do you recommend for simply finding all duplicate files among all files? (4 Replies)
Discussion started by: kiasas
4 Replies

6. UNIX for Dummies Questions & Answers

For Loop To Rename Multiple Files Finds One Non-existant File

Okay so here's something that's confusing me: I have a script that's designed to remove the words "new_" from the front of any file except two exceptions and it looks something like this... for i in new_* do if ] && ]; then j=`echo "$i"|cut -c5-` mv $i $j fi done ... (5 Replies)
Discussion started by: Korn0474
5 Replies

7. UNIX for Dummies Questions & Answers

find -size -7M finds files, but won't cp them all

If I run: find /somefolder -type f -size -7M | wc -l I get 73594 files But when I run find /somefolder -type f -size -7M -exec /bin/cp -v {} /someotherfolder/ \; it only copies 38891 of the files to the folder, why? There's a mix of all types of files in /somefolder. Is there some other... (12 Replies)
Discussion started by: unclecameron
12 Replies

8. Shell Programming and Scripting

Remove duplicate files

Hi, In a directory, e.g. ~/corpus is a lot of files and subdirectories. Some of the files are named: 12345___PP___0902___AA.txt 12346___PP___0902___AA. txt 12347___PP___0902___AA. txt The amount of files varies. I need to keep the highest (12347___PP___0902___AA. txt) and remove... (5 Replies)
Discussion started by: corfuitl
5 Replies

9. Shell Programming and Scripting

Duplicate files

Hi Gents, I have 1 files as seen below. 44571009 100 42381900 101 23482389 102 44571009 103 28849007 104 28765648 105 25689908 106 28765648 107 42381900 108 44571009 109 17298799 110 44571009 111 I would like to get something like it 44571009 100 103 109 111 (3 Replies)
Discussion started by: jiam912
3 Replies

10. UNIX for Beginners Questions & Answers

Command finds some, misses some

The contents of my home directory: bin Desktop Documents Downloads folders Music Pictures Public Templates Videos When I run the command for file in /home/myself/*d*; do if ; then echo $file; fi; doneit finds /home/myself/Downloads /home/myself/Videos but not "folders". ... (5 Replies)
Discussion started by: Xubuntu56
5 Replies
CKDIST(1)						    BSD General Commands Manual 						 CKDIST(1)

NAME
ckdist -- check software distributions SYNOPSIS
ckdist [-airsx] [-d dir] [-n name] [-t type] file ... DESCRIPTION
The ckdist utility reads ``checksum'' files (which are assumed to specify components of a software distribution) and verifies the integrity of the distribution by validating the checksum of each component file. Both MD5 (128-bit ``message digest'') and .inf (32-bit CRC) checksum formats are supported. The file operands may refer to regular files or to directories. Regular files named md5, or which have an .md5 or an .inf extension, are assumed to be of the implied type, otherwise format is determined from content. If a directory is specified, it is searched for appropri- ately-named files only. The options are as follows: -a Report on all distribution components, not just those in respect of which errors are detected. -i Ignore missing distribution components. -r Search specified directories recursively. -s Suppress complaints about inaccessible checksum files and directories. -x Verify the existence of distribution components (and also check sizes, in the case of .inf files), but omit the more time- consuming step of actually computing and comparing checksums. -d dir Look for distribution components in the directory dir. -n name Access distribution components using the filename name. When accessing .inf file components, append the appropriate exten- sion to the filename. -t type Assume that all specified checksum files are of the format type, and search directories only for files in this format (where type is either md5 or inf). EXIT STATUS
The ckdist utility exits with one of the following values: 0 No errors were detected. 1 Errors were found in a distribution. 2 Usage errors, inaccessible input files, or other system errors were encountered. SEE ALSO
cksum(1), md5(1) NOTES
Both BSD and DOS versions of ckdist are available. BSD
January 20, 1997 BSD
All times are GMT -4. The time now is 09:02 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy