Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: remove duplicate files in a directory
Top Forums Shell Programming and Scripting remove duplicate files in a directory Post 101841 by jim mcnamara on Monday 13th of March 2006 02:46:21 PM
Old 03-13-2006
PS: one directory doesn't have duplicated names...

if you run cksum or another hash like md5 you can find duplicates that way:
Code:
cd /path/to/wherever
for file in `ls *`
do
      cksum $file
done | awk '{ 
         arr[$1]++
         if(arr[$1]>1)  {print $0 }
         } ' > ./duplicate.files

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

script that detects duplicate files in directory

I need help with a script which accepts one argument and goes through all the files under a directory and prints a list of possible duplicate files As its output, it prints zero or more lines, each one containing a space-separated list of filenames. All the files listed on one line have the same... (1 Reply)
Discussion started by: trueman82
1 Replies

2. Shell Programming and Scripting

remove all duplicate lines from all files in one folder

Hi, is it possible to remove all duplicate lines from all txt files in a specific folder? This is too hard for me maybe someone could help. lets say we have an amount of textfiles 1 or 2 or 3 or... maximum 50 each textfile has lines with text. I want all lines of all textfiles... (8 Replies)
Discussion started by: lowmaster
8 Replies

3. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

4. Shell Programming and Scripting

Remove duplicate files in same directory

Hi all. Am doing continuous backup of mailboxes using rsync. So whenever a new mail arrives it is automatically copied on backup server. When a new mail arrives it is named as xyz:2, when it is read by the email client an S is appended xyz:2,S Eventually , 2 copies of the same file exist on... (7 Replies)
Discussion started by: coolatt
7 Replies

5. Shell Programming and Scripting

Remove Duplicate Files On Remote Servers

Hello, I wrote a basic script that works however I am was wondering if it could be sped up. I am comparing files over ssh to remove the file from the source server directory if a match occurs. Please Advise me on my mistakes. #!/bin/bash for file in `ls /export/home/podcast2/"$1" ` ; do ... (5 Replies)
Discussion started by: jaysunn
5 Replies

6. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to... (3 Replies)
Discussion started by: arvindng
3 Replies

7. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

8. Shell Programming and Scripting

Remove duplicate files

Hi, In a directory, e.g. ~/corpus is a lot of files and subdirectories. Some of the files are named: 12345___PP___0902___AA.txt 12346___PP___0902___AA. txt 12347___PP___0902___AA. txt The amount of files varies. I need to keep the highest (12347___PP___0902___AA. txt) and remove... (5 Replies)
Discussion started by: corfuitl
5 Replies

9. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

10. Shell Programming and Scripting

Remove all but newest two files (Not a duplicate post)

TARGET_DIR='/media/andy/MAXTOR_SDB1/Ubuntu_Mate_18.04/' REGEX='{4}-{2}-{2}_{2}:{2}' # regular expression that match to: date '+%Y-%m-%d_%H:%M' LATEST_FILE="$(ls "$TARGET_DIR" | egrep "^${REGEX}$" | tail -1)" find "$TARGET_DIR" ! -name "$LATEST_FILE" -type f -regextype egrep -regex... (7 Replies)
Discussion started by: drew77
7 Replies

LEARN ABOUT REDHAT

cksum

cksum(n)							       cksum								  cksum(n)

NAME

       cksum - calculate a cksum(1) compatible checksum

SYNOPSIS

       package require Tcl 8.2

       package require cksum ?1.0?

       ::crc::cksum ?-format format? message

       ::crc::cksum ?-format format? -filename file

DESCRIPTION

       This  package  provides a Tcl-only implementation of the cksum(1) algorithm based upon information provided at in the GNU implementation of
       this program as part of the GNU Textutils 2.0 package.

COMMANDS

       ::crc::cksum ?-format format? message

       ::crc::cksum ?-format format? -filename file
	      The command takes string data or a file name and returns a checksum value calculated using the cksum(1)  algorithm.  The	result	is
	      formatted using the format(n) specifier provided or as an unsigned integer (%u) by default.

OPTIONS

       -filename name
	      Return a checksum for the file contents instead of for parameter data.

       -format string
	      Return the checksum using an alternative format template.

EXAMPLES

       % crc::cksum "Hello, World!"
       2609532967

       % crc::cksum -format 0x%X "Hello, World!"
       0x9B8A5027

       % crc::cksum -file cksum.tcl
       1828321145

SEE ALSO

       sum(n), crc32(n)

AUTHORS

       Pat Thoyts

KEYWORDS

       cksum, checksum, crc, crc32, cyclic redundancy check, data integrity, security

crc									1.0								  cksum(n)
All times are GMT -4. The time now is 04:52 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy