Sponsored Content
Top Forums Shell Programming and Scripting remove duplicate files in a directory Post 101841 by jim mcnamara on Monday 13th of March 2006 02:46:21 PM
Old 03-13-2006
PS: one directory doesn't have duplicated names...

if you run cksum or another hash like md5 you can find duplicates that way:
Code:
cd /path/to/wherever
for file in `ls *`
do
      cksum $file
done | awk '{ 
         arr[$1]++
         if(arr[$1]>1)  {print $0 }
         } ' > ./duplicate.files

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

script that detects duplicate files in directory

I need help with a script which accepts one argument and goes through all the files under a directory and prints a list of possible duplicate files As its output, it prints zero or more lines, each one containing a space-separated list of filenames. All the files listed on one line have the same... (1 Reply)
Discussion started by: trueman82
1 Replies

2. Shell Programming and Scripting

remove all duplicate lines from all files in one folder

Hi, is it possible to remove all duplicate lines from all txt files in a specific folder? This is too hard for me maybe someone could help. lets say we have an amount of textfiles 1 or 2 or 3 or... maximum 50 each textfile has lines with text. I want all lines of all textfiles... (8 Replies)
Discussion started by: lowmaster
8 Replies

3. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

4. Shell Programming and Scripting

Remove duplicate files in same directory

Hi all. Am doing continuous backup of mailboxes using rsync. So whenever a new mail arrives it is automatically copied on backup server. When a new mail arrives it is named as xyz:2, when it is read by the email client an S is appended xyz:2,S Eventually , 2 copies of the same file exist on... (7 Replies)
Discussion started by: coolatt
7 Replies

5. Shell Programming and Scripting

Remove Duplicate Files On Remote Servers

Hello, I wrote a basic script that works however I am was wondering if it could be sped up. I am comparing files over ssh to remove the file from the source server directory if a match occurs. Please Advise me on my mistakes. #!/bin/bash for file in `ls /export/home/podcast2/"$1" ` ; do ... (5 Replies)
Discussion started by: jaysunn
5 Replies

6. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to... (3 Replies)
Discussion started by: arvindng
3 Replies

7. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

8. Shell Programming and Scripting

Remove duplicate files

Hi, In a directory, e.g. ~/corpus is a lot of files and subdirectories. Some of the files are named: 12345___PP___0902___AA.txt 12346___PP___0902___AA. txt 12347___PP___0902___AA. txt The amount of files varies. I need to keep the highest (12347___PP___0902___AA. txt) and remove... (5 Replies)
Discussion started by: corfuitl
5 Replies

9. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

10. Shell Programming and Scripting

Remove all but newest two files (Not a duplicate post)

TARGET_DIR='/media/andy/MAXTOR_SDB1/Ubuntu_Mate_18.04/' REGEX='{4}-{2}-{2}_{2}:{2}' # regular expression that match to: date '+%Y-%m-%d_%H:%M' LATEST_FILE="$(ls "$TARGET_DIR" | egrep "^${REGEX}$" | tail -1)" find "$TARGET_DIR" ! -name "$LATEST_FILE" -type f -regextype egrep -regex... (7 Replies)
Discussion started by: drew77
7 Replies
COLLATOR_SORT(3)							 1							  COLLATOR_SORT(3)

Collator::sort - Sort array using specified collator

	Object oriented style

SYNOPSIS
public bool Collator::sort (array &$arr, [int $sort_flag]) DESCRIPTION
Procedural style bool collator_sort (Collator $coll, array &$arr, [int $sort_flag]) This function sorts an array according to current locale rules. Equivalent to standard PHP sort(3) . PARAMETERS
o $coll -Collator object. o $arr - Array of strings to sort. o $sort_flag - Optional sorting type, one of the following: o Collator::SORT_REGULAR - compare items normally (don't change types) o Collator::SORT_NUMERIC - compare items numerically o Collator::SORT_STRING - compare items as strings Default sorting type is Collator::SORT_REGULAR. It is also used if an invalid $sort_flag value has been specified. RETURN VALUES
Returns TRUE on success or FALSE on failure. EXAMPLES
Example #1 collator_sort(3) example <?php $coll = collator_create( 'en_US' ); $arr = array( 'at', 'as', 'as' ); var_export( $arr ); collator_sort( $coll, $arr ); var_export( $arr ); ?> The above example will output: array ( 0 => 'at', 1 => 'as', 2 => 'as', )array ( 0 => 'as', 1 => 'as', 2 => 'at', ) SEE ALSO
Collator constants, collator_asort(3), collator_sort_with_sort_keys(3). PHP Documentation Group COLLATOR_SORT(3)
All times are GMT -4. The time now is 06:16 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy