UNIX for Dummies Questions & Answers

Hello,

I am working with about 500,000 text files and 90% of them are duplicates. I need a way to delete the duplicate ones.

The files are email messages with the following file name examples:

20040129-1457 This is the Subject line.txt
20040129-1457 This is the Subject line-1.txt
20040129-1457 This is the Subject line-2.txt
20040129-1457 This is the Subject line-3.txt
20040129-1457 This is the Subject line-4.txt
20040129-1459 This is the other subject line.txt
20040129-1459 This is the other subject line-1.txt
20040129-1459 This is the other subject line-2.txt
20040129-1459 This is the other subject line-3.txt
20040129-1459 This is the other subject line-4.txt
20040129-1459 This is the other subject line-5.txt

etc etc etc...

Is there any way to delete the ones that are named with the serial at the end?

these files are on a Macintosh running OSX 10.4.3 and it seems that the finder cant handle that many files effectively. A unix script would work better i think.

Thanks for any help.

Ashu

Based on your inputs,

Code:

#!/bin/ksh

ls | while read line; do
        if [ -f "$line" ]; then
                fname=$(echo ${line%%.txt})
                rm -f "${fname}"-*.txt
        fi
done

--NOTE--
The '-f' in rm is to supress all the error messages that you *will* get with this script.
--/NOTE--

That works but it is really slow.

The files are actually closer to 1.1 million files at this point.

I was able to use something like this:

gholkar:/1 ashu$ ls | grep "[1-9].txt$"
20040129-1457 FMP Password-1.txt
20040129-1457 FMP Password-2.txt
20040129-1457 FMP Password-3.txt
20040129-1459 Dude-1.txt
20040129-1459 Dude-2.txt

That works to show me the files, but when i add in the xargs to rm the files i get this:

gholkar:/1 ashu$ ls | grep "[1-9].txt$" | xargs rm
rm: 20040129-1457: No such file or directory
rm: FMP: No such file or directory
rm: Password-1.txt: No such file or directory
rm: 20040129-1457: No such file or directory
rm: FMP: No such file or directory
rm: Password-2.txt: No such file or directory
rm: 20040129-1457: No such file or directory
rm: FMP: No such file or directory
rm: Password-3.txt: No such file or directory
rm: 20040129-1459: No such file or directory
rm: Dude-1.txt: No such file or directory
rm: 20040129-1459: No such file or directory
rm: Dude-2.txt: No such file or directory

so how do i work around the spaces in the file names?

Try this :

ls | grep "[1-9].txt$" | xargs -I {} rm "{}"


If there is a space in the file name, then rm assumes it as more than one file.. so enclose the file name in double quotes to resolve the problem.