Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Shell script to copy a log file if it exceeds 5000000 bytes

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script to copy a log file if it exceeds 5000000 bytes
# 1  
Old 01-03-2011
Java Shell script to copy a log file if it exceeds 5000000 bytes
Hi,

on unix box, under /local/home/userid/logs folder, apps generated the following files

sw_warn.log
sw_error.log
eaijava.log

if there any application specific errors, the above file will keep growing.

if the file exceed 5000000 bytes, I would like to have a shell script which do the following

1) copy logs files from /local/home/userid/logs into /local/home/userid/logs/yyyy_mm_dd_timestamp.
2) Shell script should create a folder yyyy_mm_dd_timestamp.
3) Also, delete the files from /local/home/userid/logs
4) if deleting files are not recommended then delete the contents.

Thanks
# 2  
Old 01-03-2011
Question What have you tried?
You have four requests, but do not show any work or even an attempt at solving this. We like to help, not write your code.
These 3 Users Gave Thanks to joeyg For This Post:
DukeNuke2 m.d.ludwig methyl
# 3  
Old 01-03-2011
Quote:
4) if deleting files are not recommended then delete the contents.
We don't know your system.
Are the files open by an application at the time of the log file maintenance slot?
(If so, don't delete them, null them!).

Please show sample directory lists of every directory listed.


What is the maximum growth rate of the files? This determines how often to check.

Are there any disc space issues?

If you were to do this job manually on a given date, what would you type?
# 4  
Old 01-03-2011
Question
so far i have come up with the following
Code:
TODAY=`date +%d/%m/%y`
CURTIME=`date +%H:%M:%S`
 
#go to that directory
cd /local/home/userid/logs
 
#check the file size
ls -ltr sw_warn.log > sw_warn_filesize.txt
export file_size=`awk -F" " '{ print $5 }' sw_warn_filesize.txt`
 
# if greater than 5000000, then create a directory and copy the file. 
if [[ $file_size > 5000000 ]]
then
today_dir=mkdir TODAY_CURTIME
cp sw_warn.log /local/home/userid/logs/today_dir
 
#Instead of deleting the file, how to null sw_warn.log file
???

#remove this file
rm sw_warn_filesize.txt
else 
exit 0
fi

-------------------------------------------

We don't know your system.
Are the files open by an application at the time of the log file maintenance slot?

[me] yes these files are open by the apps
(If so, don't delete them, null them!).
[me] how?

Please show sample directory lists of every directory listed.
[me] the directory path is /local/home/userid/logs. under logs directory, i have three files
sw_warn.log
sw_error.log
eaijava.log


What is the maximum growth rate of the files? This determines how often to check.

[me] - i don't know the growth rate. btw, how to scheudle the shell script which keep checking the file size and run the script

Are there any disc space issues?
[me] no disc space issue.
Last edited by Franklin52; 01-04-2011 at 04:23 AM.. Reason: Please use code tags
# 5  
Old 01-03-2011
e.g. To null an open file called "logfile" (after copying it of course) using a Shell which is running as a user with sufficient permissions.
Code:
>logfile


On the subject of scheduling, use unix "cron" to schedule routine maintenance tasks. The maintenance frequency depends on the growth rate and local rules.

Most Systems Administrators schedule logfile maintenance weekly on a Sunday night.
Personally I schedule such jobs for 12:00 (lunchtime) daily which gives you a chance to read the overnight logs before your second coffee.
Imho, keeping log files down to a readable size (albeit backed with many days of history) can save you when you need information about a hot problem in a hurry.

Hmm. By this time tomorrow you will know the growth rate ... won't you?


Quote:
TODAY=`date +%d/%m/%y`
CURTIME=`date +%H:%M:%S`
It is not advisible to use characters such as colons and solidi in filenames. It can (and does) confuse multi-platform backup software as well as making subsequent processing in Shell painful. Unix will not stop you creating such filenames because they are valid. The solidus will give you the most grief becuse it is the directory delimiter in the full hierarcial pathname to a file.

This line does not work - there are no variable names preceded with a dollar sign and the "mkdir" command never runs.
Quote:
today_dir=mkdir TODAY_CURTIME
I'll leave that one for you to work out.
Last edited by methyl; 01-03-2011 at 06:40 PM.. Reason: spellin
# 6  
Old 01-03-2011
My proposed solution:
Code:
mark=$(date +%Y%m%d_%H%M%S)

cd /local/home

find */logs -maxdepth 1 \
            -type f \
            -size +5000000c \
            -name 'sw_warn.log' -o -name 'sw_error.log' -o -name 'eaijava.log'
            -print \
| while read file
do
    dir=$(dirname ${file})/${mark}

    if mkdir -p "${dir}"; then
        if cp -ip "${file} "${dir}"; then
            > "${file}"
        else
            echo "${file}": unable to copy to "${dir}" 1>&2
        fi
    else
        echo "${dir}": unable to create directory 1>&2
    fi
done

To be honest, if you are running the script once a day or week, then:
Code:
mark=$(date +%Y%m%d)

or even:
Code:
mark=$(date +%Y/%m%d)

would be more than sufficient.
But I have to admit the statement:
Quote:
Originally Posted by lookinginfo
Are there any disc space issues?
[me] no disc space issue.
bothers me. There are always disk space issues -- disks may be cheap, but no one has infinite disk space. IMHO, rotating the logfiles is "better", for it limits the total size of the logs to the number of rotations times 5M (plus or minus). Change the body of the while loop to:
Code:
    p=9

    for i in 8 7 6 5 4 3 2 1 0; do
        if [ -f "${file}.${i}" ]; then
            mv "${file}.${i}" "${file}.${p}"
        fi

        p="${i}"
    done

    if cp -ip "${file} "${file}.0"; then
        > "${file}"
    else
        echo "${file}": unable to rotate 1>&2
    fi

Run this version of the script once a week, and you will retain a minimum of ten weeks of logs.
# 7  
Old 01-04-2011
man cronolog : http://cronolog.org/
This User Gave Thanks to rdcwayx For This Post:
m.d.ludwig
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script - entered input(1-40 bytes) needs to be converted exactly 40 bytes

hello, suppose, entered input is of 1-40 bytes, i need it to be converted to 40 bytes exactly. example: if i have entered my name anywhere between 1-40 i want it to be stored with 40 bytes exactly. enter your name: donald duck (this is of 11 bytes) expected is as below - display 11... (3 Replies)
Discussion started by: shravan.300
3 Replies

2. Shell Programming and Scripting

How to remove a file in shell script if its size exceeds limit?

How can i remove a file using shell script when its size exceeds 10MB. Given that file is located in different location to the shell script where it is running? (4 Replies)
Discussion started by: vel4ever
4 Replies

3. Shell Programming and Scripting

shell script to remove all lines which exceeds a particular date & time

I have a text file which got 6th coloumn as date and 7th coloumn as time. The text contains data for last one week. I need to remove all the data whose date & time is after 03/08/2011 06:00:00 and save it on another file TEXT FILE ======== 6 dbclstr-b IXT_Web Memphis_Prod_SQL_Diff... (4 Replies)
Discussion started by: ajiwww
4 Replies

4. Shell Programming and Scripting

Shell script to copy file

Dear all, I have a database with thousands of files with the structure of name is: Filename_hour_year.abc Filename_hour_year_1.abc .............. So what I need is how to write a script that all file with contain the character "_1" will copy to "_2" For example: file name:... (7 Replies)
Discussion started by: hainguyen1402
7 Replies

5. Shell Programming and Scripting

Shell Script - Copy File at intervals

Hi, I want to copy some files from a Folder say, /usr/X at random intervals to another location. Basically, new files will be dumped at random intervals to location /usr/X and I have to copy those new files to some other location (after copying, I cannot delete those files from source... (2 Replies)
Discussion started by: angshuman_ag
2 Replies

6. Shell Programming and Scripting

Help with a shell script to modify one line and copy the next 9 to same file

Hi everyone, the problem is quite simple, yet I can't find an easy solution using awk. I need to search for a string in $3, then if I find this string, copy the line,modify $3, and copy the next 9 lines to the same file. My problem is in the copying of the lines... Finding and modifying... (5 Replies)
Discussion started by: Teroc
5 Replies

7. Shell Programming and Scripting

shell script to convert file_size from bytes to megabytes

Hi All, OS:AIX 64 bits. Requirement is to convert file_size from bytes to megabytes through shell script as below: export DBALIST="xyz@rediffmail.com" ls -ltr abcd.txt > file_size.result export file_size=`awk -F" " '{ print $5 }' file_size.result` if ] then mailx -s "File abcd.txt... (3 Replies)
Discussion started by: a1_win
3 Replies

8. UNIX for Dummies Questions & Answers

Shell script to search for text in a file and copy file

Compete noob question.... I need a script to search through a directory and find files containing text string abcde1234 for example and then copy that file with that text string to another directory help please :eek: (9 Replies)
Discussion started by: imeadows
9 Replies

9. Shell Programming and Scripting

Remove first N bytes and last N bytes from a binary file on AIX.

Hi all, Does anybody know or guide me on how to remove the first N bytes and the last N bytes from a binary file? Is there any AWK or SED or any command that I can use to achieve this? Your help is greatly appreciated!! Best Regards, Naveen. (1 Reply)
Discussion started by: naveendronavall
1 Replies

10. UNIX for Dummies Questions & Answers

Shell script to calc sum of bytes used by files

I'm looking to create a Korn Shell script that, if given a directory as an arg, will calc bytes used by all files in the given directory and display that info. If no command line arg is given the program is to calc and display the bytes used by all the files in the pwd. Example output: ... (3 Replies)
Discussion started by: kecannon
3 Replies
Login or Register to Ask a Question