01-17-2009
Finding Duplicate files
How do you delete and and find duplicate files?
Last edited by Jicom4; 01-18-2009 at 04:14 PM..
9 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
A pogram named LOGGEDON returns an output of:
Ref_num IP Address Logged on User
12000 10.10.12.12 12-02-2002 11:00 john
12004 10.10.12.13 12-03-2002 14:00 mary
12012 10.10.12.14 12-03-2002 11:30 bob
12024 ... (1 Reply)
Discussion started by: dinplant
1 Replies
2. Shell Programming and Scripting
Hi,
I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern.
Note:These are the samples of two files,but i can have more... (2 Replies)
Discussion started by: jerome Sukumar
2 Replies
3. Shell Programming and Scripting
Hi,
I have research data, which is organized to 100 folders numbered 00-99. I have many sets of 100 folders, for different values of initial parameters. For some reason, the computer that ran the program to gather the data, didn't always create a unique seed for each folder. I anticipated that... (1 Reply)
Discussion started by: Jopi
1 Replies
4. Shell Programming and Scripting
Hi Unix Guru's
I had generated the uniqe code for every day date ranging from 20000101 to 21990101(200 years alomost 73000 uniqe codes ) and redirected it to text file.
Now My problem is i want to check whether there are any duplicates in unique code not PRESENT in the textfile ?
... (2 Replies)
Discussion started by: Reddy482
2 Replies
5. Shell Programming and Scripting
Hello,
I am working with a perl script that tries to find the average "frequency" in which lines are duplicated. So far I've only managed to find the way to count how many times the lines are repeated, the code is as follows:
perl -ae'
my $filename= $ENV{'i'};
open (FILE, "$filename") or... (10 Replies)
Discussion started by: acsg
10 Replies
6. Shell Programming and Scripting
Hi guys can you please help me with a script to find files with one row/1 line of content then move the file to another directory my script below runs but nothing happens to the files....Alternatively Ca I get a script to find the *.csv files with "wc -1" results = 1 then create a list of those... (5 Replies)
Discussion started by: Dj Moi
5 Replies
7. Shell Programming and Scripting
Hi Gents,
I have 1 files as seen below.
44571009 100
42381900 101
23482389 102
44571009 103
28849007 104
28765648 105
25689908 106
28765648 107
42381900 108
44571009 109
17298799 110
44571009 111
I would like to get something like it
44571009 100 103 109 111 (3 Replies)
Discussion started by: jiam912
3 Replies
8. Programming
Hello All,
I have got some assignment to complete till this Monday and problem statement is as follow :-
Problem :- Find duplicate files (especially .c and .cpp) from two project base directories with following requirement :-
1.Should be extendable to search in multiple base... (4 Replies)
Discussion started by: anand.shah
4 Replies
9. Shell Programming and Scripting
I have a project tree like that.
after running find command with the -no -empty option, i am able to have a list of non empty directory
DO_MY_SEARCH="find . -type d -not -empty -print0"
MY_EXCLUDE_DIR1=" -e NOT_IN_USE -e RTMAP -e NOT_USEFULL "
echo " " > $MY_TEMP_RESULT_1
while... (2 Replies)
Discussion started by: jcdole
2 Replies
LEARN ABOUT DEBIAN
fdupes
FDUPES(1) General Commands Manual FDUPES(1)
NAME
fdupes - finds duplicate files in a given set of directories
SYNOPSIS
fdupes [ options ] DIRECTORY ...
DESCRIPTION
Searches the given path for duplicate files. Such files are found by comparing file sizes and MD5 signatures, followed by a byte-by-byte
comparison.
OPTIONS
-r --recurse
for every directory given follow subdirectories encountered within
-R --recurse:
for each directory given after this option follow subdirectories encountered within (note the ':' at the end of option; see the
Examples section below for further explanation)
-s --symlinks
follow symlinked directories
-H --hardlinks
normally, when two or more files point to the same disk area they are treated as non-duplicates; this option will change this behav-
ior
-n --noempty
exclude zero-length files from consideration
-f --omitfirst
omit the first file in each set of matches
-A --nohidden
exclude hidden files from consideration
-1 --sameline
list each set of matches on a single line
-S --size
show size of duplicate files
-m --summarize
summarize duplicate files information
-q --quiet
hide progress indicator
-d --delete
prompt user for files to preserve, deleting all others (see CAVEATS below)
-N --noprompt
when used together with --delete, preserve the first file in each set of duplicates and delete the others without prompting the user
-v --version
display fdupes version
-h --help
displays help
SEE ALSO
md5sum(1)
NOTES
Unless -1 or --sameline is specified, duplicate files are listed together in groups, each file displayed on a separate line. The groups are
then separated from each other by blank lines.
When -1 or --sameline is specified, spaces and backslash characters () appearing in a filename are preceded by a backslash character.
EXAMPLES
fdupes a --recurse: b
will follow subdirectories under b, but not those under a.
fdupes a --recurse b
will follow subdirectories under both a and b.
CAVEATS
If fdupes returns with an error message such as fdupes: error invoking md5sum it means the program has been compiled to use an external
program to calculate MD5 signatures (otherwise, fdupes uses internal routines for this purpose), and an error has occurred while attempting
to execute it. If this is the case, the specified program should be properly installed prior to running fdupes.
When using -d or --delete, care should be taken to insure against accidental data loss.
When used together with options -s or --symlink, a user could accidentally preserve a symlink while deleting the file it points to.
Furthermore, when specifying a particular directory more than once, all files within that directory will be listed as their own duplicates,
leading to data loss should a user preserve a file without its "duplicate" (the file itself!).
AUTHOR
Adrian Lopez <adrian2@caribe.net>
FDUPES(1)