Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

List Files being deleted in a shell script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers List Files being deleted in a shell script
# 1  
Old 02-12-2017
List Files being deleted in a shell script
Good evening, i need your help please

I've got a file script thet deletes about half millions of files, and i want those files to be printed in a log file as an evidence those files were removed.

Code:
#/bin/sh
rm "/home/e-smith/files/users/bill/Maildir/cur/1392373930.28512.comp01:2,S"
rm "/home/e-smith/files/users/ted/Maildir/cur/1420726198.17690.comp01:2,S"
....

and execute this way:

Code:
./removefiles_2015.sh >> removefiles_2015.log 2>&1

I'm struggling with 2 problems:

1. Log file removefiles_2015.log output is 0, so files being delete are not logged

2. when deleting files takes about 20 minutes which is very slow


Is there any way to approach those things:

I appreciate your help in advanced
# 2  
Old 02-13-2017
Invoking the rm utility once for each of half a million times is going to be a LOT slower than invoking the rm utility fifty thousand times with each invocation removing 100 files.

Although not in the standards, many implementations of the rm utility have a "verbose" option such that adding a -v option to each invocation of rm will print the name of each operand removed after it is removed. If you would tell us what operating system and shell you're using, we might be able to make an easy suggestion that would work in your environment.

How did you get the list of 500,000 files to be removed? If you create the list using find, you could have find remove the files instead of just list files to be removed. If the names of the files are in another file, xargs could be used to create groups of files to be processed by rm.

You have given us very little information about your environment. The less information we have, the fewer options we can suggest to solve your problem. Please help us help you.
# 3  
Old 02-13-2017
Thank you very much for your support

First the enviroment is:


Code:
[pmba1@PRODPMBA3 shell]$ uname -s
Linux
[pmba1@PRODPMBA3 shell]$ uname -a
Linux PRODPMBA3 2.6.18-238.el5 #1 SMP Sun Dec 19 14:22:44 EST 2010 x86_64 x86_64 x86_64 GNU/Linux



2nd How did i get the list of 500,000 files to be removed?

grepping into a backup file (bplist)
Code:
grep  AB_restat_2015  home_Informe_Bplist-PRODPMBA3_15012017_Q626996.txt | grep log | grep 
'/home/pmba1/shell'

--Ouput
Code:
rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-22-01.log.gz
-rw-r--r-- pmba1     pmba1             136 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-23-01.log.gz
-rw-r--r-- pmba1     pmba1             138 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-24-02.log.gz
-rw-r--r-- pmba1     pmba1             139 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-25-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-26-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-27-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-28-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-29-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-30-01.log.gz
-rw-r--r-- root      root              138 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-30-02.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-31-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-32-01.log.gz
-rw-r--r-- pmba1     pmba1             136 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-33-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-34-01.log.gz
-rw-r--r-- pmba1     pmba1             140 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-35-02.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-36-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-37-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-38-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-39-01.log.gz
-rw-r--r-- pmba1     pmba1             140 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-40-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-41-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-42-01.log.gz
-rw-r--r-- pmba1     pmba1             136 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-43-01.log.gz
-rw-r--r-- pmba1     pmba1             137 Jan 15 00:25 /home/pmba1/shell/AB_restat_2015-01-03-01-44-01.log.gz


then construct the script extracting the names of files found in bplist, using vi editor and adding the rm command for each file name

Thanks for your help in advanced
# 4  
Old 02-13-2017
As already suggested by Don Cragun, you might consider piping you list of files into xargs which will group them into fewer actual rm commands. Using the -t flag writes the command being executed to standard error before executing it so you can capture that to a file for review later. It might give you lots of output on few lines, but you can probably make that more readable later on.

You could limit the number of files it processes at once with the -n flag, but that might increase the number of rm command calls and therefore slow your execution. The extreme would be something like:-
Code:
list files here | xargs -tn 1 rm


Does that help?

Robin
# 5  
Old 02-13-2017
Are there other gzipped log files in that quite specific directory that you want to retain? If not,
Code:
rm -v /home/pmba1/shell/AB_restat_2015-01-03-01-*.log.gz

could do the job. The file selection granularity might be adjusted by adaption of the wild card match.
# 6  
Old 02-14-2017
Thanks once again for your help, Ive got a question: The number 1 as an argument in xargs command what is used for ?

Code:
list files here | xargs -tn 1 rm

# 7  
Old 02-15-2017
One might expect that your xargs man page might say something like:
Code:
−n number    Invoke utility using as many standard input arguments as possible, up to number (a
	     positive decimal integer) arguments maximum.  Fewer arguments shall be used if:
	     •	The command line length accumulated exceeds the size specified by the −s
		option (or {LINE_MAX} if there is no −s option).
	     •	The last iteration has fewer than number, but not zero, operands remaining.

Does that help?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to list deleted files in UNIX?

Hi All, Its an interview question. I just want to know the answer of below question. 1) How to list deleted files in unix (13 Replies)
Discussion started by: pspriyanka
13 Replies

2. UNIX for Advanced & Expert Users

Help with a shell script? List files, delete them and log them

Hello, i'm trying to solve this script. List, one at a time, all files larger than 100K in the /home/username directory tree. Give the user the option to delete or compress the file, then proceed to show the next one. Write to a logfile the names of all deleted files and the deletion times. I... (7 Replies)
Discussion started by: jose2802
7 Replies

3. Shell Programming and Scripting

Need help in finding and copying list of files using bash shell script

Dear All, I have a situation where I want to copy some files of type .txt. These files are o/p from one program. Some of the files are named as fileName .txt instead of fileName.txt after fileName by mistake I have specified "space". Now I want to move these files as follows. mv fileName*... (13 Replies)
Discussion started by: linuxUser_
13 Replies

4. Shell Programming and Scripting

Urgent...Need a shell script to list files which belong to particular groups

Hi, I am just new to scripting but got to write a complex scipt please help. i need a shell script which can check the list of data listed in a txt doc and see if they belong to any of the groups that are listed in other list file.... (5 Replies)
Discussion started by: draghun9
5 Replies

5. UNIX for Dummies Questions & Answers

Generate list of deleted files

I copied all JPEGs from my laptop to an external drive using find . -name "*.jpg" -exec cp '{}' ./media/Backup/pictures \; And then deleted all of them from my laptop. Now, I realize that I need the folder path of all the original JPEGs as the path has the important information. I dont... (1 Reply)
Discussion started by: eshwaconsulting
1 Replies

6. AIX

Who deleted my files

Just looking for some guidance on how to figure out who might have deleted some files off one of my systems. These files are not root owned files so could be deleted by a handful of folks in the group responsible for these files besides the root users. Anyway I have been tasked with trying to... (1 Reply)
Discussion started by: juredd1
1 Replies

7. Shell Programming and Scripting

Shell Script Needed to Read a text from a list files

Hi, Below is my issue which I desperately need and I want a shell script which can do this job. I need this script as I m planning to put this for a system health check. Please assist me. 1. There are 10 log files in a particular location. 2. open each log file. Goto to the end of the... (4 Replies)
Discussion started by: kashriram
4 Replies

8. Shell Programming and Scripting

Shell Script Create List of Deleted Files

Hi, I want to put all the deleted files in a txt file. Because i want to backup my image server which has thousands of jpg images. I wrote a shell script which will copies images from image server to backup image server. I setup a cronjob which runs on every five minutes. & through timestamp it... (8 Replies)
Discussion started by: mirfan
8 Replies

9. Shell Programming and Scripting

Getting a list of files on an ftp, via shell script...

G'day, I was wanting to write a shell script that checks an ftp server for the presence of new files, then get those files. In so much as the get, this is pretty straight forward, but I cannot work out how to get a list of files to check. Is it possible for a shell script to get the output of... (1 Reply)
Discussion started by: Elric of Grans
1 Replies

10. Shell Programming and Scripting

Pulling a list of files from FTP site in a shell script

Hi, I am writting a shell script which will pull a list files (mentioned in one file 1.txt) from external FTP site (for ex: ftp://abcd.efghijk.com/). The 1.txt is in my local unix directory. I have username and password to connect the external FTP site. Also before I pull the files, I need... (0 Replies)
Discussion started by: spatra
0 Replies
Login or Register to Ask a Question