Sponsored Content
Top Forums Shell Programming and Scripting remove duplicate files in a directory Post 101841 by jim mcnamara on Monday 13th of March 2006 02:46:21 PM
Old 03-13-2006
PS: one directory doesn't have duplicated names...

if you run cksum or another hash like md5 you can find duplicates that way:
Code:
cd /path/to/wherever
for file in `ls *`
do
      cksum $file
done | awk '{ 
         arr[$1]++
         if(arr[$1]>1)  {print $0 }
         } ' > ./duplicate.files

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

script that detects duplicate files in directory

I need help with a script which accepts one argument and goes through all the files under a directory and prints a list of possible duplicate files As its output, it prints zero or more lines, each one containing a space-separated list of filenames. All the files listed on one line have the same... (1 Reply)
Discussion started by: trueman82
1 Replies

2. Shell Programming and Scripting

remove all duplicate lines from all files in one folder

Hi, is it possible to remove all duplicate lines from all txt files in a specific folder? This is too hard for me maybe someone could help. lets say we have an amount of textfiles 1 or 2 or 3 or... maximum 50 each textfile has lines with text. I want all lines of all textfiles... (8 Replies)
Discussion started by: lowmaster
8 Replies

3. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

4. Shell Programming and Scripting

Remove duplicate files in same directory

Hi all. Am doing continuous backup of mailboxes using rsync. So whenever a new mail arrives it is automatically copied on backup server. When a new mail arrives it is named as xyz:2, when it is read by the email client an S is appended xyz:2,S Eventually , 2 copies of the same file exist on... (7 Replies)
Discussion started by: coolatt
7 Replies

5. Shell Programming and Scripting

Remove Duplicate Files On Remote Servers

Hello, I wrote a basic script that works however I am was wondering if it could be sped up. I am comparing files over ssh to remove the file from the source server directory if a match occurs. Please Advise me on my mistakes. #!/bin/bash for file in `ls /export/home/podcast2/"$1" ` ; do ... (5 Replies)
Discussion started by: jaysunn
5 Replies

6. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to... (3 Replies)
Discussion started by: arvindng
3 Replies

7. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

8. Shell Programming and Scripting

Remove duplicate files

Hi, In a directory, e.g. ~/corpus is a lot of files and subdirectories. Some of the files are named: 12345___PP___0902___AA.txt 12346___PP___0902___AA. txt 12347___PP___0902___AA. txt The amount of files varies. I need to keep the highest (12347___PP___0902___AA. txt) and remove... (5 Replies)
Discussion started by: corfuitl
5 Replies

9. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

10. Shell Programming and Scripting

Remove all but newest two files (Not a duplicate post)

TARGET_DIR='/media/andy/MAXTOR_SDB1/Ubuntu_Mate_18.04/' REGEX='{4}-{2}-{2}_{2}:{2}' # regular expression that match to: date '+%Y-%m-%d_%H:%M' LATEST_FILE="$(ls "$TARGET_DIR" | egrep "^${REGEX}$" | tail -1)" find "$TARGET_DIR" ! -name "$LATEST_FILE" -type f -regextype egrep -regex... (7 Replies)
Discussion started by: drew77
7 Replies
RMM(1)								     [nmh-1.5]								    RMM(1)

NAME
rmm - remove messages SYNOPSIS
rmm [+folder] [msgs] [-unlink | -nounlink] [-version] [-help] DESCRIPTION
By default, rmm will remove the specified messages by renaming each of the message files with a site-dependent prefix (usually a comma). Such files will then need to be removed in some manner after a certain amount of time. Many sites arrange for cron to remove these files once a day, so check with your system administrator. Alternately, if you wish for rmm to really remove the files representing these messages, you can use the -unlink switch. But messages removed by this method cannot be later recovered. If you prefer a more sophisticated method of `removing' messages, you can define the rmmproc profile component. For example, you can add a profile component such as rmmproc: /home/foouser/bin/rmm_msgs then instead of simply renaming the message file, rmm will call the named program or script to handle the files that represent the messages to be deleted. Some users of csh prefer the following: alias rmm 'refile +d' where folder `+d' is a folder for deleted messages, and alias mexp 'rm `mhpath +d all`' is used to "expunge" deleted messages. The current message is not changed by rmm, so a next will advance to the next message in the folder as expected. FILES
$HOME/.mh_profile The user profile PROFILE COMPONENTS
Path: To determine the user's nmh directory Current-Folder: To find the default current folder rmmproc: Program to delete the message SEE ALSO
refile(1), rmf(1) DEFAULTS
`+folder' defaults to the current folder `msgs' defaults to cur `-nounlink' CONTEXT
If a folder is given, it will become the current folder. BUGS
Since refile uses your rmmproc to delete the message, the rmmproc must NOT call refile without specifying -normmproc, or you will create an infinte loop. MH.6.8 11 June 2012 RMM(1)
All times are GMT -4. The time now is 10:44 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy