remove duplicate files in a directory

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove all but newest two files (Not a duplicate post)

TARGET_DIR='/media/andy/MAXTOR_SDB1/Ubuntu_Mate_18.04/' REGEX='{4}-{2}-{2}_{2}:{2}' # regular expression that match to: date '+%Y-%m-%d_%H:%M' LATEST_FILE="$(ls "$TARGET_DIR" | egrep "^${REGEX}$" | tail -1)" find "$TARGET_DIR" ! -name "$LATEST_FILE" -type f -regextype egrep -regex...

2. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines.

3. Shell Programming and Scripting

Remove duplicate files

Hi, In a directory, e.g. ~/corpus is a lot of files and subdirectories. Some of the files are named: 12345___PP___0902___AA.txt 12346___PP___0902___AA. txt 12347___PP___0902___AA. txt The amount of files varies. I need to keep the highest (12347___PP___0902___AA. txt) and remove...

4. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>...

5. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to...

6. Shell Programming and Scripting

Remove Duplicate Files On Remote Servers

Hello, I wrote a basic script that works however I am was wondering if it could be sped up. I am comparing files over ssh to remove the file from the source server directory if a match occurs. Please Advise me on my mistakes. #!/bin/bash for file in `ls /export/home/podcast2/"$1" ` ; do ...

7. Shell Programming and Scripting

Remove duplicate files in same directory

Hi all. Am doing continuous backup of mailboxes using rsync. So whenever a new mail arrives it is automatically copied on backup server. When a new mail arrives it is named as xyz:2, when it is read by the email client an S is appended xyz:2,S Eventually , 2 copies of the same file exist on...

8. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete...

9. Shell Programming and Scripting

remove all duplicate lines from all files in one folder

Hi, is it possible to remove all duplicate lines from all txt files in a specific folder? This is too hard for me maybe someone could help. lets say we have an amount of textfiles 1 or 2 or 3 or... maximum 50 each textfile has lines with text. I want all lines of all textfiles...

10. Shell Programming and Scripting

script that detects duplicate files in directory

I need help with a script which accepts one argument and goes through all the files under a directory and prints a list of possible duplicate files As its output, it prints zero or more lines, each one containing a space-separated list of filenames. All the files listed on one line have the same...

LEARN ABOUT DEBIAN

fdupes

FDUPES(1)						      General Commands Manual							 FDUPES(1)

NAME

       fdupes - finds duplicate files in a given set of directories

SYNOPSIS

       fdupes [ options ] DIRECTORY ...

DESCRIPTION

       Searches  the  given  path for duplicate files. Such files are found by comparing file sizes and MD5 signatures, followed by a byte-by-byte
       comparison.

OPTIONS

       -r --recurse
	      for every directory given follow subdirectories encountered within

       -R --recurse:
	      for each directory given after this option follow subdirectories encountered within (note the ':' at the	end  of  option;  see  the
	      Examples section below for further explanation)

       -s --symlinks
	      follow symlinked directories

       -H --hardlinks
	      normally, when two or more files point to the same disk area they are treated as non-duplicates; this option will change this behav-
	      ior

       -n --noempty
	      exclude zero-length files from consideration

       -f --omitfirst
	      omit the first file in each set of matches

       -A --nohidden
	      exclude hidden files from consideration

       -1 --sameline
	      list each set of matches on a single line

       -S --size
	      show size of duplicate files

       -m --summarize
	      summarize duplicate files information

       -q --quiet
	      hide progress indicator

       -d --delete
	      prompt user for files to preserve, deleting all others (see CAVEATS below)

       -N --noprompt
	      when used together with --delete, preserve the first file in each set of duplicates and delete the others without prompting the user

       -v --version
	      display fdupes version

       -h --help
	      displays help

SEE ALSO

       md5sum(1)

NOTES

       Unless -1 or --sameline is specified, duplicate files are listed together in groups, each file displayed on a separate line. The groups are
       then separated from each other by blank lines.

       When -1 or --sameline is specified, spaces and backslash characters  () appearing in a filename are preceded by a backslash character.

EXAMPLES

       fdupes a --recurse: b
	      will follow subdirectories under b, but not those under a.

       fdupes a --recurse b
	      will follow subdirectories under both a and b.

CAVEATS

       If  fdupes  returns  with  an error message such as fdupes: error invoking md5sum it means the program has been compiled to use an external
       program to calculate MD5 signatures (otherwise, fdupes uses internal routines for this purpose), and an error has occurred while attempting
       to execute it. If this is the case, the specified program should be properly installed prior to running fdupes.

       When using -d or --delete, care should be taken to insure against accidental data loss.

       When used together with options -s or --symlink, a user could accidentally preserve a symlink while deleting the file it points to.

       Furthermore, when specifying a particular directory more than once, all files within that directory will be listed as their own duplicates,
       leading to data loss should a user preserve a file without its "duplicate" (the file itself!).

AUTHOR

       Adrian Lopez <adrian2@caribe.net>

																	 FDUPES(1)

Shell Programming and Scripting