UNIX for Advanced & Expert Users

Hi,

I have a log file which runs into 3 to 5 GB.

We store this typically for 6 months. When a new month starts we move the previous month into a 9 month back up log (file.9m) and delete the last month of the 9 month back up.

Iam using awk to find the data and cat to join the files like below

BEGIN { FS = "|"}

{ if (($5 == "09") && ($6 == "08")) print $0 >> "/file.1m";
}

END { print "script processed " NR}

in the above $5 is month and $6 is the year and the log file has pipe separated columns

cat /file.6m /new.log > /file.temp
mv /file.temp /new.log

the file.6m is the back up file when a new month starts and the new.log is the running log.

similarly I extract file.1m(last month) from file.6m and file.8m ( eight months without the last month from the file.9m) and join using cat as above.

This process though is NOT neat, is yielding the result.

I would like to know if there is any better of doing this. Since the file is really large sometimes i find the data jumbled up.

Appreciate your inputs.

Have you considered using logrotate tools? I dont know if it deals with such size logfile but maybe worth enquiry...

Iam not trying to use a tool.

Iam trying to develop a script myself.

Any other ideas will be appreciated.

Thanks

rather than storing the entire nine months logs in one file, have you considered one file for each month? then it would be trivial to move each month "up" at month end:

<top of head mod>

Code:

for FEXT in  8 7 6 5 4 3 2 1
do
OLDEXT=$((FEXT + 1))
OLDFILE=file.$OLDEXT
NEWFILE=file.FEXT
[ -f $NEWFILE ] && mv $NEWFILE $OLDFILE
done

which will move each file "up" by one each time it is run, knocking the last file off the end (actually overwriting it with the months before).
does this help?