I am working with about 500,000 text files and 90% of them are duplicates. I need a way to delete the duplicate ones.
The files are email messages with the following file name examples:
20040129-1457 This is the Subject line.txt
20040129-1457 This is the Subject line-1.txt
20040129-1457 This is the Subject line-2.txt
20040129-1457 This is the Subject line-3.txt
20040129-1457 This is the Subject line-4.txt
20040129-1459 This is the other subject line.txt
20040129-1459 This is the other subject line-1.txt
20040129-1459 This is the other subject line-2.txt
20040129-1459 This is the other subject line-3.txt
20040129-1459 This is the other subject line-4.txt
20040129-1459 This is the other subject line-5.txt
etc etc etc...
Is there any way to delete the ones that are named with the serial at the end?
these files are on a Macintosh running OSX 10.4.3 and it seems that the finder cant handle that many files effectively. A unix script would work better i think.
That works to show me the files, but when i add in the xargs to rm the files i get this:
gholkar:/1 ashu$ ls | grep "[1-9].txt$" | xargs rm
rm: 20040129-1457: No such file or directory
rm: FMP: No such file or directory
rm: Password-1.txt: No such file or directory
rm: 20040129-1457: No such file or directory
rm: FMP: No such file or directory
rm: Password-2.txt: No such file or directory
rm: 20040129-1457: No such file or directory
rm: FMP: No such file or directory
rm: Password-3.txt: No such file or directory
rm: 20040129-1459: No such file or directory
rm: Dude-1.txt: No such file or directory
rm: 20040129-1459: No such file or directory
rm: Dude-2.txt: No such file or directory
so how do i work around the spaces in the file names?
sample .csv files with 7 columns.I want to delete the last column from each of the below files but retain their file names (1_ContractDocuments.csv,2_ContractDocuments.csv etc.)
There can be more files like 3_ContractDocuments.csv , 4_ContractDocuments.csv . Can you please help
source .csv... (5 Replies)
Hi,
I have multiple files in my log folder. e.g:
a_m1.log
b_1.log
c_1.log
d_1.log
b_2.log
c_2.log
d_2.log
e_m1.log
a_m2.log
e_m2.log
I need to keep latest 10 instances of each file.
I can write multiple find commands but looking if it is possible in one line.
m file are monthly... (4 Replies)
Hi
My directory structure is as below.
dir1, dir2, dir3
I have the list of files to be deleted in the below path as below.
/staging/retain_for_2years/Cleanup/log $ ls -lrt
total 0
drwxr-xr-x 2 nobody nobody 256 Mar 01 16:15 01-MAR-2015_SPDBS2
drwxr-xr-x 2 root ... (2 Replies)
Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted!
1. The problem statement, all variables and given/known data:
Hi This is my first post.I am learning Unix and finding it difficult to get a handle on the scripting side of... (5 Replies)
I Want to delete the following files together,what command should i pass for that? (Note:- All Start With .)
.bash_logout
.bashrc
.bash_profile
.rtorrent.rc ... (3 Replies)
Hi ,
I am a newbie!!!
I want to develop a script for deleting files older than x days from multiple paths. Now I could reach upto this piece of code which deletes files older than x days from a particular path. How do I enhance it to have an input from a .txt file or a .dat file? For eg:... (12 Replies)
Greetings!
I'm looking for starting information for a shell script. Here's my scenario:
I have multiple folders(100) for example:
/www/test/applications/app1/logs
/www/test/applications/app2/logs
Within these folders there are log files files that need to be deleted after a month.
... (3 Replies)
Requirement
Several files in remote machines ought to be deleted via sh. Name of the files to be deleted are know
Approach
1) script was written with ftp (requires credential) and delete command. File names were passed as array(iterated via for loop-with ftp+delete commands enclosed within... (1 Reply)
its urgent!!!!!!111
i need a script which can delete data from multiple files.
plz if anybody knows the script plz write a mail to me :
(Email addresses are not allowed) (5 Replies)
Hello
when I try to run rm on multiple files I have problem to delete files with space.
I have this command :
find . -name "*.cmd" | xargs \rm -f
it doing the work fine but when it comes across files with spaces like : "my foo file.cmd"
it refuse to delete it
why? (1 Reply)