Visit Our UNIX and Linux User Community


Delete files listed in text file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Delete files listed in text file
# 1  
Old 02-06-2015
Delete files listed in text file

Hi Team,

Here's the scenario,

I have a text file called "file_list.txt". Its content is as follows.
111.tmp
112.tmp
113.tmp
114.tmp

These files will present in "workdir" directory. It has many files. But only the files present in file_list.txt has to be deleted from the workdir directory. We have the code as follows.

Code:
while read line
do
	rm -f $workdir/${line}
done < file_list.txt

But we are facing performance issue in using while loop. In production, we will have more than 100000 files list in the text file. Can anyone help us to provide a code to improve the performance.

Thanks
# 2  
Old 02-06-2015
I'm afraid deleting 100000 files from a directory will take its time, esp. on a production system where other things are going on as well.
cding to the $workdir would not improve performance significantly; you might want to give
Code:
while read A; read B; read C; do echo rm -f $A $B $C; done < file

a try, as it saves two thirds of process creations. Or even
Code:
< file xargs -n8 echo rm -f

, eliminating 7 process creations.
This User Gave Thanks to RudiC For This Post:
# 3  
Old 02-06-2015
Thank you RudiC for your reply.

There is a small change in the given requirement. Actually the file names will not have .tmp in the "file_list.txt". While deleting I need to append the filename with ".tmp" while deleting.

Can you help me out?
# 4  
Old 02-06-2015
kmanivan82,
To use RudiC's 2nd suggestion, try:
Code:
awk '{print $1".tmp"}' file_list.txt | xargs -n8 echo rm -f

* remove echo once you have tested.

RudiC,
It doesn't appear your 1st suggestion will process the last 3 records/file names from input file if total number of records/file names on file are not evenly divisible by 3.
These 2 Users Gave Thanks to mjf For This Post:
# 5  
Old 02-06-2015
Quote:
Originally Posted by mjf
...

RudiC,
It doesn't appear your 1st suggestion will process the last 3 records/file names from input file if total number of records/file names on file are not evenly divisible by 3.
You're right. But for the original problem:
Code:
while read A; do read B; read C; echo rm -r $A $B $C; done < file

would have worked.

With the new filename modification requirements, this approach needs more work.
This User Gave Thanks to Don Cragun For This Post:
# 6  
Old 02-07-2015
Thanks. When testing with 8 lines, I had an extra empty line in my file which I didn't notice...
# 7  
Old 02-07-2015
Thank you RudiC and mjf. The performance statistics is as follows.

using while loop
**** Script started - 2015-02-07 02:18:52 ****
**** Script ended at 2015-02-07 02:19:45 ****
Duration: 53 seconds to delete 5400 files

using the syntax given by mjf/RudiC
**** Script started - 2015-02-07 02:05:24 ****
**** Script ended at 2015-02-07 02:05:33 ****
Duration: 9 seconds to delete 5400 files

Thank you once again.

Previous Thread | Next Thread
Test Your Knowledge in Computers #367
Difficulty: Medium
The Single UNIX Specification specifies vi, so every conforming UNIX system must have it.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Want to delete the junk files from a directory which are not listed in a TEXT file

Hello Everyone, I want to delete the image files from a directory, which are not listed in a TEXT file. The directory contains large number of image files (in millions) required / not required. I want to delete the image files which are "not required". I have generated a Text file having... (3 Replies)
Discussion started by: Praveen Pandit
3 Replies

2. Shell Programming and Scripting

Cat files listed in text file and redirect to new directory with same filename

I have a directory that is restricted and I cannot just copy the files need, but I can cat them and redirect them to a new directory. The files all have the date listed in them. If I perform a long listing and grep for the date (150620) I can redirect that output to a text file. Now I need to... (5 Replies)
Discussion started by: trigger467
5 Replies

3. Shell Programming and Scripting

How to grep a log file for words listed in separate text file?

Hello, I want to grep a log ("server.log") for words in a separate file ("white-list.txt") and generate a separate log file containing each line that uses a word from the "white-list.txt" file. Putting that in bullet points: Search through "server.log" for lines that contain any word... (15 Replies)
Discussion started by: nbsparks
15 Replies

4. UNIX for Dummies Questions & Answers

Delete files whose file names are listed in a .txt file

hi, I need a help. I used this command to list all the log files which are for more than 10 days to a text file. find /usr/script_test -type f -mtime +10>>/usr/ftprm.txt I want all these files listed in the ftprm.txt to be ftp in another machine and then rm the files. Anyone can help me... (8 Replies)
Discussion started by: kamaldev
8 Replies

5. Shell Programming and Scripting

Copy files listed in text file to new directory

I am trying to write a script that will copy all file listed in a text file (100s of file names) to a new directory Assume script will run with main as current working directory and I know how many files/lines will be in List.txt Im trying to work up a test script using this model Contents of... (2 Replies)
Discussion started by: IAmTheGrass
2 Replies

6. Shell Programming and Scripting

Send a mail to IDs listed in a text file

I have a list of mail ids in text file and want a ksh script that reads this text file and sends a mail to all mail ids with same subject line and content. I am using UX-HP machine and KSH. Thanks for help in advance! (5 Replies)
Discussion started by: Sriranga
5 Replies

7. Shell Programming and Scripting

Moving files listed in a data file to a new directory using Perl

Hi, I have a data file that lists a number of files. I want to move the files named in that one to another directory. Here's what I have: #!/usr/bin/perl -w open(FILE, "<collision.txt"); my @lines=<FILE>; foreach my $lines (@lines) { system("mv $lines collisions/."); } close(FILE); ... (2 Replies)
Discussion started by: renthead720
2 Replies

8. Shell Programming and Scripting

Shellscript to sort duplicate files listed in a text file

I have many pdf's scattered across 4 machines. There is 1 location where I have other Pdf's maintained. But the issues it the 4 machines may have duplicate pdf's among themselves, but I want just 1 copy of each so that they can be transfered to that 1 location. What I have thought is: 1) I have... (11 Replies)
Discussion started by: deaddevil
11 Replies

9. Shell Programming and Scripting

Copy files listed in a text file - whitespace problem.

Hi, Say I have this text file <copy.out> that contains a list of files/directories to be copied out to a different location. $ more copy.out dir1/file1 dir1/file2 dir1/file3 "dir1/white space" dir1/file4 If I do the following: $copy=`more copy.out` $echo $copy dir1/file1... (4 Replies)
Discussion started by: 60doses
4 Replies

10. HP-UX

CVSWeb - Directories listed but files not listed

I am using CVSWeb on HPUnix. When i access it, all directories are listed but files are not listed. I am getting the error "NOTE: There are 51 files, but none matches the current tag. " in tomcat sevrer log i am getting the message "rlog warning: Missing revision or branch number after -r"... (0 Replies)
Discussion started by: ganesh
0 Replies

Featured Tech Videos