Remove excess log files past data mark


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove excess log files past data mark
# 1  
Old 08-11-2014
Remove excess log files past data mark

I have got application log files collecting up fast inside of a directory.

The logs contains many logs that look like this (were talking 1000s+):
the Log_NAME followed by Date and application specific nonsense.

Code:
(...)
application_log_2014-07-13_20-30-09 
application_log_2014-07-20_08-30-27 
(...)

Below is an excerpt of a script I use to backup excess log files before I remove all, I was wondering if anyone had any idea how I could create a script within this where I remove all scripts for last month. I think it should be possible to safely remove all the: application_log_*.... that are say older then $DT_limit I set. However I need expert opinion on this because I can not exceed the limit I set and need to be safe as folder contains other .ini and log files that must be ignored so care is needed.

Code:
##LOG FILE LOCATIONS##
##LOG FILE LOCATIONS##
##LOG FILE LOCATIONS##
export EXCESS_LOG_LOCATION="/excess_log/location"
export BACKUPDIR="/backup/log_backup"
export LOGFILE_LOCATION="/log_location"
 
## Date/Time variables

DT=`date '+%F:%H:%M'`


for file in `ls $EXCESS_LOG_LOCATION/application_log_*`
do
  mv ${file} ${BACKUPDIR}
  echo "log processed file ${file}" >> ${LOGFILE_LOCATION}
done
 
exit 0

I will post a script I am thinking of using as soon as I make progress but if anyone has an idea or has done this before let me know.


FYI - Ignore the Backup AND MV command functionality please this is just to help you see the logic I want to follow. and help you understand where I am trying to place the date check*.
# 2  
Old 08-11-2014
What operating system are you on, as many date manipulation operations can be OS specific?

Do you have a date command that supports the --date option?

Code:
$ date --date "-10 days" +'%Y/%m/%d'
2014/08/02


Last edited by Chubler_XL; 08-11-2014 at 05:34 PM..
# 3  
Old 08-11-2014
hey

I ran your command and it is supported. Ie I get the same result.

Rhel5
# 4  
Old 08-11-2014
I think something like this should get the job done:

Code:
#!/bin/bash
LDATE=$(date --date "-$DT_limit days" +'%Y-%m-%d')

for file in $EXCESS_LOG_LOCATION/application_log_*
do
    FDATE=${file#*application_log_}
    FDATE=${FDATE%%_*}
    if [[ "$FDATE" < "$LDATE" ]] 
    then
        echo "file: $file is older than $DT_limit days"
    fi
done


Last edited by Chubler_XL; 08-11-2014 at 06:19 PM..
# 5  
Old 08-12-2014
Hey , Thanks for the help.
I really appreciate it!

Last edited by Samuel12; 08-12-2014 at 02:50 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to select all files added to a directory in the past 5 mins (HP-UX)?

Hey everyone, I need to select all files that were added to a specific directory in the past 5 mins and copy them over to a different directory. I am using HP-UX OS which does not have support for amin, cmin, and mmin. B/c of this, I am creating a temp file and will use the find -newer command... (7 Replies)
Discussion started by: mattkoz
7 Replies

2. UNIX for Dummies Questions & Answers

Find command to get the modified files past 2 hours

Hello, How to get the modified/created files past 2 hours in Solaris with find command? Thank you. (7 Replies)
Discussion started by: balareddy
7 Replies

3. Shell Programming and Scripting

Can you extract (remove) lines from log files?

I use "MineOS" (a linux distro with python scripts and web ui included for managing a Minecraft Server). The author of the scripts is currently having a problem with the Minecraft server log file being spammed with certain entries. He's working on clearing up the spam. But in the meantime, I'm... (8 Replies)
Discussion started by: nbsparks
8 Replies

4. Shell Programming and Scripting

Listing files of PAST 7 days and concatenate it...

Hi All, I want to list file of LAST 7 days acc. to its modified date and then concatinate. I have following piece of code.. For concatenate cat file1 file2 >> Output (For concatinating) find . -mtime -7 -exec basename {} \; (list past files but it is including . file also) Plz... (4 Replies)
Discussion started by: vivek1489
4 Replies

5. Shell Programming and Scripting

Modify log files to get uniq data

Hello, I have a log file that has following output as below. LAP.sun5 CC LAP.sun5 CQ perl.sun5 CC perl.sun5 CQ TSLogger.sun5 CC TSLogger.sun5 CQ TSLogger.sun5 KR WAS.sun5 CC WAS.sun5 MT WAS.sun5 CQ I want to output to be in the way below, i tried using awk but could not do it. ... (12 Replies)
Discussion started by: asirohi
12 Replies

6. Shell Programming and Scripting

shell script to remove old files and write to a log file

Hi, I have a script that works on a unix box but am trying to get it working on a linux box that uses shell. I am not a programmer so this is proving harder than I imagined. I made some changes and ended up with the script below but when I run it I get the following messages. Any help would be... (4 Replies)
Discussion started by: yabai
4 Replies

7. Shell Programming and Scripting

find files from the past 7 days

Hi All, I have a file which contains the listing of another directory: >cat list.dat -rwxr-xr-x 1 test staff 10240 Oct 02 06:53 test.txtdd -rwxrwxrwx 1 test staff 0 Oct 04 07:22 test.txx -rwxrwxrwx 1 test staff 132 Sep 16 2007 test_tt.sh... (6 Replies)
Discussion started by: deepakgang
6 Replies

8. Solaris

Find 3 months old log files and remove

HI I HAVE LOG FILES IN THIS FORMAT XXX.log.20080509215745 XXX.log.20080209205745 XXX.log.20080301785745 XXX.log.20080109155745 XXX.log.20080409115745 XXX.log.20080309105745 I NEED TO FIND THE FILES FROM 3 MONTHS OLD AND REMOVE PLEASE HELP THANKS, (13 Replies)
Discussion started by: cvdev
13 Replies

9. Shell Programming and Scripting

shell script for log files data!

Hi Team, please write a shell script. It is for veritas netbackup logs. The result we currently have is a single file for each day's backups. we will keep the files in the directory and the file names are like below mentioned. example :/opt/openv/netbackup/reports/Daily/NB_success*. The No... (6 Replies)
Discussion started by: rvrao77
6 Replies

10. Shell Programming and Scripting

Script to remove log files

Can anyone help me with a shell script to remove all files (log files) created 7 days before current system date. TIA, Kiran. (1 Reply)
Discussion started by: kiranherekar
1 Replies
Login or Register to Ask a Question