Need to delete large set of files (i.e) close to 100K from a directory based on the input file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need to delete large set of files (i.e) close to 100K from a directory based on the input file
# 1  
Old 08-20-2012
Need to delete large set of files (i.e) close to 100K from a directory based on the input file

Hi all,

I need a script to delete a large set of files from a directory under / based on an input file and want to redirect errors into separate file.

I have already prepared a list of files in the input file.

Kndly help me.

Thanks,
Prash
# 2  
Old 08-20-2012
Code:
xargs echo rm < list 2> errlog

Remove the 'echo' once you've tested and are sure it really does what you want.
# 3  
Old 08-20-2012
Quote:
Originally Posted by Corona688
Code:
xargs echo rm < list 2> errlog

Remove the 'echo' once you've tested and are sure it really does what you want.
I have run the below command but i need to print to a file where it shows what files have been deleted?

sudo xargs rm <file

Btw all the files are in directory under "/" . But the input file is something like below (i.e) it dont have any path to the file. So i need a command which deletes these files without storing the input file inside the directory under / where the original files reside.

more file
0012301170
0000013300
0000014100
-
-
-

Thanks for the quick reply.

---------- Post updated at 02:18 PM ---------- Previous update was at 02:13 PM ----------

Corona,

Can you please suggest whether i can use the below script to delete the files?

Code:
for i in `cat file`;
do
     rm -f $i
done

If yes i want an echo statement to be printed on what files have been deleted and what file are not found in the directory?

P.S: I need to delete close to 100K files and the file anmes are listed in the input file.

Thanks

---------- Post updated at 04:10 PM ---------- Previous update was at 02:18 PM ----------

Could you please help me in deleting the large set of files without moving the input file to a directory where original files reside.

Thanks

Last edited by Franklin52; 08-21-2012 at 05:27 AM.. Reason: Please use code tags
# 4  
Old 08-20-2012
If your rm has the -v (verbose) option you could do this, assumption is your file_list.txt and logs are in the /tmp directory.

Code:
cd /your_dir
xargs rm -v < /tmp/file_list.txt > /tmp/removed.list 2> /tmp/errors.list

# 5  
Old 08-20-2012
for i in `cat file`; This will be especially bad for what you want because it will try -- and fail -- to cram all 100,000 filenames into one thing and die with 'too many arguments'.
# 6  
Old 08-20-2012
Quote:
Originally Posted by Chubler_XL
If your rm has the -v (verbose) option you could do this, assumption is your file_list.txt and logs are in the /tmp directory.

Code:
cd /your_dir
xargs rm -v < /tmp/file_list.txt > /tmp/removed.list 2> /tmp/errors.list

Thanks for the post. so if use the below script it is giving me illegal option --v errors, so i have taken -v option and have run the command it worked fine. But is there a way to capture all the files which were deleted in this operation?

I can see the data for which files were not found from errors.list

sudo xargs rm -v < /tmp/file > /tmp/removed.list 2> tmp/errors.list

Thanks
# 7  
Old 08-20-2012
Well any file in the original filelist that isn't in the error list was removed.

Can you output the first few lines of the error file ( head /tmp/errors.list ) file so we can see the format, a simple awk script should be able to produce the removed files list.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to compare files in 2 folders and delete the large file

Hello, my first thread here. I've been searching and fiddling around for about a week and I cannot find a solution.:confused: I have been converting all of my home videos to HEVC and sometimes the files end up smaller and sometimes they don't. I am currently comparing all the video files... (5 Replies)
Discussion started by: Josh52180
5 Replies

2. Shell Programming and Scripting

If no input then set directory to current

Hi GUys I have a script where i am counting the number of files that the user has read permissions to in a folder . They are asked for this folder at the start if they dont enter anything then i want to be able to do the search on the current directory, can you help me with that part please... (25 Replies)
Discussion started by: johnnybananas
25 Replies

3. Shell Programming and Scripting

Script needed to delete to the list of files in a directory based on last created & delete them

Hi My directory structure is as below. dir1, dir2, dir3 I have the list of files to be deleted in the below path as below. /staging/retain_for_2years/Cleanup/log $ ls -lrt total 0 drwxr-xr-x 2 nobody nobody 256 Mar 01 16:15 01-MAR-2015_SPDBS2 drwxr-xr-x 2 root ... (2 Replies)
Discussion started by: prasadn
2 Replies

4. Shell Programming and Scripting

Sed: Splitting A large File into smaller files based on recursive Regular Expression match

I will simplify the explaination a bit, I need to parse through a 87m file - I have a single text file in the form of : <NAME>house........ SOMETEXT SOMETEXT SOMETEXT . . . . </script> MORETEXT MORETEXT . . . (6 Replies)
Discussion started by: sumguy
6 Replies

5. Shell Programming and Scripting

Help needed - Split large file into smaller files based on pattern match

Help needed urgently please. I have a large file - a few hundred thousand lines. Sample CP START ACCOUNT 1234556 name 1 CP END ACCOUNT CP START ACCOUNT 2224444 name 1 CP END ACCOUNT CP START ACCOUNT 333344444 name 1 CP END ACCOUNT I need to split this file each time "CP START... (7 Replies)
Discussion started by: frustrated1
7 Replies

6. Shell Programming and Scripting

Script to delete files with an input for directories and an input for path/file

Hello, I'm trying to figure out how best to approach this script, and I have very little experience, so I could use all the help I can get. :wall: I regularly need to delete files from many directories. A file with the same name may exist any number of times in different subdirectories.... (3 Replies)
Discussion started by: *ShadowCat*
3 Replies

7. Shell Programming and Scripting

Splitting large file into multiple files in unix based on pattern

I need to write a shell script for below scenario My input file has data in format: qwerty0101TWE 12345 01022005 01022005 datainala alanfernanded 26 qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28 qwerty0101TWE 12342 01022005 07022009 datainalc hitalbert 43 qwerty0101CFG 12345... (19 Replies)
Discussion started by: jimmy12
19 Replies

8. Shell Programming and Scripting

Split a file into multiple files based on the input pattern

I have a file with lines something like. ...... 123_start ...... ....... 123_end .... ..... 456_start ...... ..... 456_end .... ..... 789_start .... .... 789_end (6 Replies)
Discussion started by: abinash
6 Replies

9. Shell Programming and Scripting

Sheel script to Delete a set of files from a given directory

I have a file <filestodelete> containing names of files to to be deleted from a directory <filesstore>. I want a script file which accptes the <filestodelete> and also the location of the directory(<filestore>) and deletes all files matching. Thanks in Advance.. (3 Replies)
Discussion started by: VardhiniVenkat
3 Replies

10. Shell Programming and Scripting

i want to delete a file based on existing file in a directory

hi i am having four files in a directory.like 1)sampleRej 2)exampleRej 3)samplemain 4)examplemain my requirement is i have to search for the rejected files (sampleRej,exampleRej) in a directory.if these files in that directory then i have to delete the main files... (3 Replies)
Discussion started by: srivsn
3 Replies
Login or Register to Ask a Question