Visit Our UNIX and Linux User Community


Remove excess log files past data mark


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove excess log files past data mark
# 1  
Old 08-11-2014
Remove excess log files past data mark

I have got application log files collecting up fast inside of a directory.

The logs contains many logs that look like this (were talking 1000s+):
the Log_NAME followed by Date and application specific nonsense.

Code:
(...)
application_log_2014-07-13_20-30-09 
application_log_2014-07-20_08-30-27 
(...)

Below is an excerpt of a script I use to backup excess log files before I remove all, I was wondering if anyone had any idea how I could create a script within this where I remove all scripts for last month. I think it should be possible to safely remove all the: application_log_*.... that are say older then $DT_limit I set. However I need expert opinion on this because I can not exceed the limit I set and need to be safe as folder contains other .ini and log files that must be ignored so care is needed.

Code:
##LOG FILE LOCATIONS##
##LOG FILE LOCATIONS##
##LOG FILE LOCATIONS##
export EXCESS_LOG_LOCATION="/excess_log/location"
export BACKUPDIR="/backup/log_backup"
export LOGFILE_LOCATION="/log_location"
 
## Date/Time variables

DT=`date '+%F:%H:%M'`


for file in `ls $EXCESS_LOG_LOCATION/application_log_*`
do
  mv ${file} ${BACKUPDIR}
  echo "log processed file ${file}" >> ${LOGFILE_LOCATION}
done
 
exit 0

I will post a script I am thinking of using as soon as I make progress but if anyone has an idea or has done this before let me know.


FYI - Ignore the Backup AND MV command functionality please this is just to help you see the logic I want to follow. and help you understand where I am trying to place the date check*.
# 2  
Old 08-11-2014
What operating system are you on, as many date manipulation operations can be OS specific?

Do you have a date command that supports the --date option?

Code:
$ date --date "-10 days" +'%Y/%m/%d'
2014/08/02


Last edited by Chubler_XL; 08-11-2014 at 06:34 PM..
# 3  
Old 08-11-2014
hey

I ran your command and it is supported. Ie I get the same result.

Rhel5
# 4  
Old 08-11-2014
I think something like this should get the job done:

Code:
#!/bin/bash
LDATE=$(date --date "-$DT_limit days" +'%Y-%m-%d')

for file in $EXCESS_LOG_LOCATION/application_log_*
do
    FDATE=${file#*application_log_}
    FDATE=${FDATE%%_*}
    if [[ "$FDATE" < "$LDATE" ]] 
    then
        echo "file: $file is older than $DT_limit days"
    fi
done


Last edited by Chubler_XL; 08-11-2014 at 07:19 PM..
# 5  
Old 08-12-2014
Hey , Thanks for the help.
I really appreciate it!

Last edited by Samuel12; 08-12-2014 at 03:50 PM..

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to select all files added to a directory in the past 5 mins (HP-UX)?

Hey everyone, I need to select all files that were added to a specific directory in the past 5 mins and copy them over to a different directory. I am using HP-UX OS which does not have support for amin, cmin, and mmin. B/c of this, I am creating a temp file and will use the find -newer command... (7 Replies)
Discussion started by: mattkoz
7 Replies

2. UNIX for Dummies Questions & Answers

Find command to get the modified files past 2 hours

Hello, How to get the modified/created files past 2 hours in Solaris with find command? Thank you. (7 Replies)
Discussion started by: balareddy
7 Replies

3. Shell Programming and Scripting

Can you extract (remove) lines from log files?

I use "MineOS" (a linux distro with python scripts and web ui included for managing a Minecraft Server). The author of the scripts is currently having a problem with the Minecraft server log file being spammed with certain entries. He's working on clearing up the spam. But in the meantime, I'm... (8 Replies)
Discussion started by: nbsparks
8 Replies

4. Shell Programming and Scripting

Listing files of PAST 7 days and concatenate it...

Hi All, I want to list file of LAST 7 days acc. to its modified date and then concatinate. I have following piece of code.. For concatenate cat file1 file2 >> Output (For concatinating) find . -mtime -7 -exec basename {} \; (list past files but it is including . file also) Plz... (4 Replies)
Discussion started by: vivek1489
4 Replies

5. Shell Programming and Scripting

Modify log files to get uniq data

Hello, I have a log file that has following output as below. LAP.sun5 CC LAP.sun5 CQ perl.sun5 CC perl.sun5 CQ TSLogger.sun5 CC TSLogger.sun5 CQ TSLogger.sun5 KR WAS.sun5 CC WAS.sun5 MT WAS.sun5 CQ I want to output to be in the way below, i tried using awk but could not do it. ... (12 Replies)
Discussion started by: asirohi
12 Replies

6. Shell Programming and Scripting

shell script to remove old files and write to a log file

Hi, I have a script that works on a unix box but am trying to get it working on a linux box that uses shell. I am not a programmer so this is proving harder than I imagined. I made some changes and ended up with the script below but when I run it I get the following messages. Any help would be... (4 Replies)
Discussion started by: yabai
4 Replies

7. Shell Programming and Scripting

find files from the past 7 days

Hi All, I have a file which contains the listing of another directory: >cat list.dat -rwxr-xr-x 1 test staff 10240 Oct 02 06:53 test.txtdd -rwxrwxrwx 1 test staff 0 Oct 04 07:22 test.txx -rwxrwxrwx 1 test staff 132 Sep 16 2007 test_tt.sh... (6 Replies)
Discussion started by: deepakgang
6 Replies

8. Solaris

Find 3 months old log files and remove

HI I HAVE LOG FILES IN THIS FORMAT XXX.log.20080509215745 XXX.log.20080209205745 XXX.log.20080301785745 XXX.log.20080109155745 XXX.log.20080409115745 XXX.log.20080309105745 I NEED TO FIND THE FILES FROM 3 MONTHS OLD AND REMOVE PLEASE HELP THANKS, (13 Replies)
Discussion started by: cvdev
13 Replies

9. Shell Programming and Scripting

shell script for log files data!

Hi Team, please write a shell script. It is for veritas netbackup logs. The result we currently have is a single file for each day's backups. we will keep the files in the directory and the file names are like below mentioned. example :/opt/openv/netbackup/reports/Daily/NB_success*. The No... (6 Replies)
Discussion started by: rvrao77
6 Replies

10. Shell Programming and Scripting

Script to remove log files

Can anyone help me with a shell script to remove all files (log files) created 7 days before current system date. TIA, Kiran. (1 Reply)
Discussion started by: kiranherekar
1 Replies

Featured Tech Videos