Help with Shell Scripting


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with Shell Scripting
# 1  
Old 04-04-2011
Help with Shell Scripting

Hi Guys,

I need some help in creating a bash script that does the following:

Currently, we have a centralized log file that grows to 50MB per day. This log file is updated regularly.

Our task is to create a script that runs every 30 minutes that will read the log file and grep for a specific pattern and dumps it to another file. The next run should read the same log file but only capture the patterns that run after previous run. Then create a new file.

At the end of the day, the script should create a maximum of 48 new files.

Hope you help me.

Last edited by DukeNuke2; 04-05-2011 at 02:29 AM..
# 2  
Old 04-04-2011
There's nowhere near enough information in your post to provide any specific assistance. What's the data look like? How is a tool to know where it left off? Is the log file rotated? How often? What's the naming scheme?

Without knowing any of the particulars, the best one can do is provide pointers. My pointer would be to take a look at the log rotation tools available. Some of them (perhaps the one already in use on your system) support running a script/executable immediately after log rotation.

Regards,
Alister
# 3  
Old 04-04-2011
hi alister,

The format of the log file is this:

<dd-mm-yyyy hh:mm:ss> [INFO/ERROR]: <Log Messages> ...

Every 11:55PM, the current day's logfile is move to the archive directory and appended with the date stamp in the filename. A new log file will be created to the next days transactions.

Basically, the script will need to capture all the lines with the message "ERROR" within the 30 minute time frame.

As per your question "how will it know where it left off', that is actually my biggest problem. I have no idea how to determine at the moment where it ends. One thing I can think of is if it possible to check for lines updated 30 minutes before the system run time? Is it possible to base on the date & time that is in the logs?

Last edited by DukeNuke2; 04-05-2011 at 02:29 AM..
# 4  
Old 04-04-2011
Hi, why dont u use something easy?
Something like this:
Code:
 
firts in your cron                            
0,30 *, *, *, * /path/script.sh               
                                              
cat script.sh                                 
#!/bin/ksh                                    
                                              
LOG=$(date +file_%d%m%Y_%k%M)                 
DIRECTORY=$(date +dir_%d%m%Y)                 
cp biggest_log > $LOG                         
>biggest_log                                  
grep ERROR $LOG > $DIRECTORY/temporal.log     
cat $DIRECTORY/temporal.log >> $DIRECTORY/$LOG

This User Gave Thanks to maya_style For This Post:
# 5  
Old 04-08-2011
Well, I know that you already have an efficient answer to your problem. just in case you may wanna try something else, I am share this code that I have written, a fruit of some reading...
This script is designed for monitoring purposes, so it can be set-up to monitor multiple logs; that is (it can egrep multiple REGEXs from multiple logs and save them somewhere else).

Code:
#!/bin/sh 
COUNT=""
HNAME=$(uname -n)
SDATE=$(date +"%h-%d-%y")__$(date +"%k-%M").log
LOGCHK="/var/log/syslog:Sending%20on|avahi-daemon|Listenning%20on /var/log/daemon.log:ntpdate|dhclient"
cd /home/guto/SCRIPT/LogMonitor/
for CLOG in $LOGCHK
 do
    #setting the dynamic variables
    LOGFILE="`echo $CLOG | cut -d: -f1`"
    EXPRETION="`echo $CLOG | cut -d: -f2`"
    EXPRETION="`echo $EXPRETION |sed -e \"s/%20/ /g\"`"
    SUFX="`echo $LOGFILE |sed -e s/\\\//_/g`"
    SUFX="`echo $SUFX |sed -e \"s/\\\./_/\"`"
    #check for the first run (this segment should run only once)
    if [ "`eval echo '$COUNT'${SUFX}`" = "" ]; then
       eval \ BASE${SUFX}=`wc -l $LOGFILE |awk '{ print $1 }'`            #gets initial length of logfile
       egrep -iw "\"$EXPRETION\"" $LOGFILE > ${HNAME}__${SUFX}__${SDATE}        #greps the expretions and saves to the new file
    fi
    
    eval COUNT${SUFX}=`wc -l $LOGFILE |awk '{ print $1 }'`                        #gets the current length of logfile
    
    #this segment takes care of on going growth of the log    
    if [ `eval echo '$COUNT'${SUFX}` -gt `eval echo '$BASE'${SUFX}` ]; then
       CLINES=`eval expr '$COUNT'${SUFX} - '$BASE'${SUFX}`
       eval \ BASE${SUFX}='$COUNT'${SUFX}
       tail -$CLINES $LOGFILE |egrep -iw "\"$EXPRETION\"" > ${HNAME}__${SUFX}__${SDATE}        #greps the expretions and saves to the new file
       
    #this segment resets the count & base variables when the logfile is rotated at night.
    elif [ `eval echo '$COUNT'${SUFX}` -lt `eval echo '$BASE'${SUFX}` ]; then
       eval \ BASE${SUFX}='$COUNT'${SUFX}                                #gets initial length of logfile after the rotation
       tail -$CLINE $LOGFILE |egrep -iw "\"$EXPRETION\"" $LOGFILE > ${HNAME}__${SUFX}__${SDATE}    #greps the expretions and saves to the new file 
    fi
 done

CONFIGURATION:
1) LOGCHK - has a couple of parts separated by a column as following:
LOGCHK="/var/log/syslog:NetworkManager|sudo"
where: the first part is the log file and the second part is the words that should be picked from the log.
2) The second segment of that variable might have multiple words to search for separated by pippe '|'.
3) The second segment of that variable might also have words which have spaces in between, but in the variable configuration those spaces should be substituted with '%20'.
4) In order to monitor multiple logs, you need to set them within the same variable (just as I tested the code with two different logs), but separated by a space.

I hope that this might be useful too...to me, it's been very useful.

Last edited by Yogesh Sawant; 04-09-2011 at 11:03 AM.. Reason: added code tags
This User Gave Thanks to guto For This Post:
# 6  
Old 04-08-2011
Wow, I gotta say... I can definitely use something like this!! Definitely saves me a bit of coding. =D
# 7  
Old 04-09-2011
@ jtollefson,
It please me to know that, so enjoy it. In the event that you add anything significantly different in the future, would you be so kind to share the insight with me? I am striving to become an efficient bash programmer, so I love to analyse codes written by others - it somewhat helps in approaching and resolving possible problems.

Thank you,
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Shell script to read lines in a text file and filter user data Shell Programming and Scripting

sxsaaas (3 Replies)
Discussion started by: VikrantD
3 Replies

2. Shell Programming and Scripting

help me in Shell Scripting

Hi there please have a look at the code..i want to create Using a named pipe. Run a find in the background starting in the working directory While this is happening wait for input from the user to ask him which file to find. If the user does not enter any data in 10 seconds ask the user again.... (1 Reply)
Discussion started by: kattak1511
1 Replies

3. Shell Programming and Scripting

Shell scripting

Hi, if in a network there are lots of PCs connected with either windows or linux as operating system.Then what will be the shell script for the same and also if the PC has linux in it then we have to find if it is occupied or unoccupied. If the PC has windows in it then we have to find if it is... (6 Replies)
Discussion started by: akansha singh
6 Replies

4. UNIX for Dummies Questions & Answers

Shell Scripting

Hey I have a data in the file named as outputFile.txt. The data is in the format 123456,12345678912345,400,09/09/09,INACTIVE. I want this output without commas ie 12345612345678912345400090909INACTIVE. Please tell me what to do and clear explain all the terms, as I am new to it. (6 Replies)
Discussion started by: sampandey31
6 Replies

5. Web Development

Perl scripting or shell scripting?

i am going to study any one of the scripting languages mentioned above(shell 0r perl scripting) . Which is having more scope for a fresher? (1 Reply)
Discussion started by: Anna Hussie
1 Replies

6. What is on Your Mind?

Shell Scripting vs Perl scripting

Gents, I have been working in a Solaris/Unix environment for about 9 months. I took some linux classses online before getting the job. But, I am not very good at scripting. I want to learn how to script. Do you think that I should start with Shell scripting or Perl? I wanted to continue with... (2 Replies)
Discussion started by: Pouchie1
2 Replies

7. Android

Android Scripting Environment: Shell Scripting and Android

I just upgraded to Android 2.2 from 2.1. The GPS issue that was troublesome in 2.1 seems to have been fixed. Some of web browsing seems faster, but it could just be my connection is better today ;) Flash works in some browsers but not very good and it is too slow for Flash apps designed for... (0 Replies)
Discussion started by: Neo
0 Replies

8. What is on Your Mind?

Shell scripting vs Perl scripting

Hi all, I would like to start developping some good scripting skills. Do you think it would be best to start with shell scripting or Perl? I already got a fundation, really basics, in perl. but I am wondering what would be best to be good at first. Can you please help me determine which one to... (14 Replies)
Discussion started by: Pouchie1
14 Replies

9. Shell Programming and Scripting

Call Shell scripting from Perl Scripting.

Hi How to call a shell scripting through a Perl scripting? Actually I need some value from Shell scripting and passes in the Perl scripting. So how can i do this? (2 Replies)
Discussion started by: anupdas
2 Replies

10. Shell Programming and Scripting

difference between AIX shell scripting and Unix shell scripting.

please give the difference between AIX shell scripting and Unix shell scripting. (2 Replies)
Discussion started by: haroonec
2 Replies
Login or Register to Ask a Question