Shell Programming and Scripting

Hi Friends,

I have somefiles like

Code:

20180720_1812.tar.gz
20180720_1912.tar.gz 
20180720_2012.tar.gz 
20180720_2112.tar.gz 
20180721_0012.tar.gz 
20180721_0112.tar.gz 
20180721_0212.tar.gz  
20180721_0312.tar.gz

in a directory and so on..these files gets created every 3 hours where as first part of file name is date and second part is time.

however as its occupying more space on disk, its required to delete old files which we manually doing now...in such a way that only last 30 days backup files should be there, any file before that should be deleted

however we have a challenge even after clearing files before 30 days, still lot of disc filled, now idea is to keep only 1 file for any given day which is being last time stamp on that day and delete rest of files for that day , in such way I can retain one file for any day and free up some space.

for example , for 2018-7 -20, i can retain 20180720_2112.tar.gz -- assuming this the last backup for that day, delete rest 3 files ..in that way, i will have atleast one file of backup for a day and free up some space by deleting rest of the copies for that given day

Code:

20180720_1812.tar.gz
20180720_1912.tar.gz 
20180720_2012.tar.gz 
20180720_2112.tar.gz

Any idea how can i do it conditionally. Appreciate any help

Moderator's Comments:

Please use CODE tags as required by forum rules!

How about

Code:

ls 2018* | sort -ur -k1,1 -t_ | cut -d_ -f1 | while read TS; do echo rm $(ls -r $TS* | tail -n +2); done
rm 20180721_0212.tar.gz 20180721_0112.tar.gz 20180721_0012.tar.gz
rm 20180720_2012.tar.gz 20180720_1912.tar.gz 20180720_1812.tar.gz

Remove the echo when happy with the proposed result.

Thank you Rudic, let me try. my apologies for not including Code tags.






Hi Rudic,


Its working fine. thank you...now i get little bit complicated requirement...

we have a mount point called /backup (300 GB)under which these files will be places every 3 hours...

now if disc space crosses 60 % of /backup and also for example if 90% is filled up in that 300 gb then our logic /program should stop at a point where deletion of files brought down to 60 % (starting from old date )

for example,
if /backup mount reaches 90 % , now it has to brought down to 60 % which is threshold. Now let us assume files are from

Code:

20180701_0112.tar.gz...to 20180831_2112.tar.gz..

now solution should start from 2018-07-01 and start deleting files for that day except last timestamp file per day (your proposed solution is already doing this) and it should continue only till a point where deleting till for example till 2018-07-03 and if this range is enough to disc usage to 60 % then our logic should exit..some thing like...that, next time when again disc cross beyond 60% then, when we run our command it should start from

Code:

2018-07-04

as

Code:

2018-7-01
2018-7-02,
2018-7-03 already has only 1 file per day...

Sorry if my explanation not clear or complicated requirement.. If possible please help . the idea is to not deleting files for all dates, rather limit it till disc usage comes to 60 %

I would be tempted to try a slightly simpler pipeline for what RudiC suggested:

Code:

ls -1 2[0-9][0-9][0-9][01][0-9][0-3][0-9]_[0-2][0-9][0-5][0-9].tar.gz |
awk -F_ '
$1 == last {
	print "echo rm " file
	file = $0
	next
}
{	last = $1
	file = $0
}' | sh

If the above prints a list of the rm commands you want to run, remove the echo from the script and run it again.

Note that you should also tell us what operating system and shell you're using when you start a thread in the Shell Programming and Scripting forum so we don't suggest things that can't work in your environment. If you are using a Solaris/SunOS system, change awk in the above script to /usr/xpg4/bin/awk or nawk.

If I create the files you named in post #1 in a directory and run the above script in that same directory, the output produced is:

Code:

rm 20180720_1812.tar.gz
rm 20180720_1912.tar.gz
rm 20180720_2012.tar.gz
rm 20180721_0012.tar.gz
rm 20180721_0112.tar.gz
rm 20180721_0212.tar.gz

On most systems you can omit the -1 option (that is the digit one; not the letter ell), but on some old systems the ls utility doesn't produce one name per line of output when output is directed to a pipe (as required by the standards).

Obviously, you can add a df on your source filesystem and check for the desired level of free space before or after each file is deleted or each time the date changes. Since your requirements aren't clear as to when this testing should be performed, I'll leave that as an exercise for the reader. (The output format produced by df also varies somewhat depending on what options you use and what operating system you're using. And, I'm not going to try to guess what OS you're using.)

Hi Don Cragun,

Thanks for your reply. Apologies , i shall try describe requirement better. We are using RHEL 7.4 as OS.

as for "Since your requirements aren't clear as to when this testing should be performed, I'll leave that as an exercise for the reader"

We used to get alert from network team that particular node is having high disc utilisation, at the point we manually login into that box perform this housekeeping activity (by deleting files and freeup space ) to bring down to 60 %. There is no need of program to automatically run when disc usage is high.. only thing is when we get alert that disc space is high, then when we execute solution it should delete files from start date based on file name in ascending order (for example like i mention if we have files from 20180701 to 20180831 then it has to start deleting files from date 20180701(keep only last copy for that day ,delete rest) ---> then check if discspace came down to 60 % if not --> continue with next date i.e. 20180702 (keep only last copy for that day ,delete rest)--df check if space is below 60 % --if not take next date i.e 20170703 (keep only last copy for that day ,delete rest) --> df check if its 60 % then exit the program...--> now after some days again if we get notification disc space ---> when we run program --> it should start from 20170704 start doing deletes till such date space equals 60%

apologies if its not still clear...

Talking about simplifying, why not

Code:

ls -r1 2[0-9][0-9][0-9][01][0-9][0-3][0-9]_[0-2][0-9][0-5][0-9].tar.gz | awk -F_ 'T[$1]++ {print "echo rm " $0}'  | sh

Hi onenessboy,
You can start by showing us the complete, exact output produced by the command:

Code:

df -P /backup

If the output from the above command doesn't complain about an unknown -P option, the percentage of space used on the filesystem containing /backup should be in field #5 on line #2 of the output from the above command.

If it does complain about an unknown -P option, show us the complete, exact output from the command:

Code:

df /backup

so we can figure out which field and line we need to examine to determine if you have reached your goal.

Hi RudiC,
Why not:

Code:

ls -r1 2[0-9][0-9][0-9][01][0-9][0-3][0-9]_[0-2][0-9][0-5][0-9].tar.gz | awk -F_ 'T[$1]++ {print "echo rm " $0}'  | sh

Because that will remove old files from the most recent date first. And onenessboy wants to remove old backup files from the oldest date first. And he wants to add code to exit the script when the capacity on that filesystem drops below 60% after each date change when one or more backup files were removed for any given date.