Shell script for log archiving


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script for log archiving
# 1  
Old 10-28-2010
Shell script for log archiving

I have an nfs mount /logfile/project mounted on several of my application server machines. I have 5 jvms running on each machine and I have several machines. the jvms logs are created and rotated every day so it will look like
Code:
/jvm1/logs/server.log.2010-10-27
/jvm2/logs/server.log.2010-10-27
/jvm3/logs/server.log.2010-10-27
/jvm4/logs/server.log.2010-10-27
/jvm5/logs/server.log.2010-10-27
/jvm5/logs/server.log.2010-10-28

I would like to create 2 cronjobs
a) go to local directory on each machine and gzip the files everyday
b) move those files to the mountpoint /logfile/project/month-year and rename them to something like "hostname-jvm1-server.log.2010-10-27"
if we could add a check which will make sure the directory exists for each month that would be good too .

Last edited by Franklin52; 10-29-2010 at 08:52 AM.. Reason: Please use code tags
# 2  
Old 10-28-2010
Here is a script for a)

Code:
ls /jvm[1-5]/logs/*server.log* | grep -v .gz | while read file
do
   gzip $file
done

How about this bash / ksh script for b)

Code:
host=$(hostname)
host=${host%.*}
MTH=$(date +%m-%Y)
# ensure at lease 1 file exists
[ "$(echo /jvm[1-5]/logs/*.gz)" = '/jvm[1-5]/logs/*.gz' ] && exit
for file in /jvm[1-5]/logs/*.gz
do
    topdir=${file#/}
    topdir=${topdir%%/*}
    base=$(basename $file)
    mkdir -p /logfile/project/$MTH
    mv $file /logfile/project/$MTH/${host}-${topdir}-${base}
done

# 3  
Old 10-28-2010
Code:
#!/bin/bash
logdir=/jvm[0-9]*/logs
logfiles=`find ${logdir} -type f`
todat=`date +"%Y-%m-%d"`

for i in `ls ${logfiles} | grep -v ${today}`
do
	gzip ${i}
	echo "File ${i} has been gziped"
done


#!/bin/bash
logdir=/jvm[0-9]*/logs
mountpoint=/logfile/project
hostname=`hostname`
for i in `ls ${logdir}/*gz 2>/dev/null`
do
	year=`echo ${i} | grep -o [0-9]*-[0-9]*-[0-9]* | awk -F'-' '{print $1}'`
	month=`echo ${i} | grep -o [0-9]*-[0-9]*-[0-9]* | awk -F'-' '{print $2}'`
	jvm=`echo ${i} | grep -o jvm[0-9]*`
	file=`echo ${i} | sed s+/.*/++g`
	newpath=${mountpoint}/${month}-${year}
	mkdir -p ${newpath} >/dev/null 2>&1
	mv ${i} ${newpath}/${hostname}-${jvm}-${file}
	echo "${i} has been moved to ${newpath}/${hostname}-${jvm}-${file}"
done

# 4  
Old 10-29-2010
Thanks both work ..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Shell script newbie- how to generate service log from shell script

Hi, I am totally a newbie to any programming languages and I just started an entry level job in an IT company. One of my recent tasks is to create a script that is able to show the log file of linux service (i.e. ntpd service) lets say, if I run my script ./test.sh, the output should be... (3 Replies)
Discussion started by: xiaogeji
3 Replies

2. Shell Programming and Scripting

Shell Script function to use script name for log file output

Hi Team - I"m very new to Shell Scripting so I have a rather novice question. My forte is Windows Batch Scripting so I was just wondering what the Shell Script equivalent is to the DOS command %~n? %~n is a DOS variable that dispayed the script name. For instance (in DOS): REM... (11 Replies)
Discussion started by: SIMMS7400
11 Replies

3. Shell Programming and Scripting

Archiving files using shell script

Dear Team, I am looking for transferring files to and from the local and remote servers using SFTP commands. Currently the script is using the mget and mput commands to do the copying of the files. While I am trying to move the files from local to remote server, I would also like to archive... (21 Replies)
Discussion started by: Rads
21 Replies

4. Shell Programming and Scripting

Archiving or removing few data from log file in real time

Hi, I have a log file that gets updated every second. Currently the size has grown to 20+ GB. I need to have a command/script, that will try to get the actual size of the file and will remove 50% of the data that are in the log file. I don't mind removing the data as the size has grown to huge... (8 Replies)
Discussion started by: Souvik Patra
8 Replies

5. Shell Programming and Scripting

Archiving a log file on monthly basis

OS : RedHat Linux 6.2 Shell : Bash Our application write messages, warnings,..etc to the following log file . /app/cms/diagnostics/logs/cms_messages.log This file has become huge now. I want this file to be archived on monthly basis using a small shell script. ie. On the 1st day of... (1 Reply)
Discussion started by: omega3
1 Replies

6. Shell Programming and Scripting

What extra Parameters I can use for archiving log files

Hello All, I have developed a script which takes following parameter from the input file to archive log files 1)Input Path 2)File pattern(*.csv) 3)Number of days(+1) Following is the algorithm of my script Read the input file go to that path and search for particular n days older... (3 Replies)
Discussion started by: mitsyjohn
3 Replies

7. Shell Programming and Scripting

Script for Archiving data folderwise

Hi, I want to archive more than 15 days old files from /var/spool/ directory which contains 67 folders. I have done a script for a single folder which is as follows : cd /data1/ctron80/var/spool/bmaprt/ find . -mtime +15 | awk '{print "mv " $1 " /back/spool/bmaprt"}' | sh cd /back/spool/... (1 Reply)
Discussion started by: tuxian
1 Replies

8. Shell Programming and Scripting

Shell script to parsing log

Hi I Have log like this : 0 234: { 3 2: 04 EE 7 14: '20081114081' 23 1: 00 79 10: '38809' 91 15: '528111510010159' 143 29: 'Streaming/downloading service' 174 3: 'MTV' 179 43: 'rtsp://172.28/MTV2GO-Loop.sdp' 224 1: 05 ... (10 Replies)
Discussion started by: justbow
10 Replies

9. Shell Programming and Scripting

creating a simple archiving script

Im trying to create a script to archive specified directories into a specified tarball backup file. This is what i want the input to look like ex. save -i '/bin/b*' -i '/bin/ls' -o backup this is what i have #!/bin/bash #save - backup file script unset myInput unset myOutput while... (3 Replies)
Discussion started by: lensmen
3 Replies

10. UNIX for Dummies Questions & Answers

creating an archiving script

Hi all. New to unix and need a little help. I am trying to create a script to archive files or directories in to a tarball. I've played a little with scripts but dont understand how to do options. What i want to be able to do is give the command any number of inputs and an output. ex.... (2 Replies)
Discussion started by: jinxe
2 Replies
Login or Register to Ask a Question