Korn/bash Script to monitor a file a check for specific data


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Korn/bash Script to monitor a file a check for specific data
# 1  
Old 02-01-2010
Korn/bash Script to monitor a file a check for specific data

Hi,


Im trying to write this script but im stuck on it, basicaly what i want to do is to write a code to verify a log file ( apache log file for example ) and for each new line with specific data , then, output this new line for another file:


full ex:

output of the server.log is ( new line each 0.5 secs )

new connection..
new connection..
new connection..
blocked....
denied >> this new line should be picked


sorry for my english, not very clear...


Thanks in advance.
# 2  
Old 02-01-2010
I've made such one but it's complex because it monitors several logfiles (auth apache/error apache/access syslog, and fail2ban) to have a snapshot of the actually banned IP's.
It makes a linecount with a grep for the interesting things at a specified rate and displays the lines in the difference count.
The problem is the logrotate because for apache/error you have also apache/error.1 and so on.
Tell us what files you want to monitor, the final usage of the script and what you've already tried.
Basics :
Code:
#!/bin/bash
OLDCOUNT=0
while :
do
    NEWS=$(grep -Fw 'denied' logfile)
    COUNT=$(echo "$NEWS" | wc -l)
    DIFF=$((COUNT-OLDCOUNT))
    if [ $DIFF -gt 0 ]
    then
        echo "$NEWS" | tail -n $DIFF
    fi
    OLDCOUNT=$COUNT
    sleep 15
done < logfile

That's the simple way (one file, no logrotate, no colour, no config file)
# 3  
Old 02-01-2010
Hi frans

thanks for the feedback

your solution is very suitable, but im facing a problem wich is i need the outputs in realtime.To be more specific, i have 25 Integration Servers wich i want to watch the log file for each new line searching for erros wich... so sleep on the script is quite bad for me. At the begining i was thinking about using something like that:


Code:
#!/bin/bash
while true;
do
tail -1 server.log | grep $ERROR
done

the problem is the memory consume of that...

Any ideas to thru that?

thanks again
# 4  
Old 02-01-2010
It's important to know how you can remotely access to the logfile (SSH...) to centralize the parsing in one script on one computer.
Hope this make sense.
I've heard of logging utilities for multiple apache servers. Syslog over TCP could be another posibility. That all depends on the architecture of the network.
# 5  
Old 02-02-2010
I will try to adapt what you wrote on what i want, when i get success i'll reply here =)
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to check for a specific number of columns in a file

Hi All I have a file which has five columns in each rows. cat file.txt a|b|c|d|e 1|2|3|4|5 a1|a2|a3|a4|a5 . . . I need to make sure that there are no less than five or more than five columns (in all the rows) by mistake. I tried this : cat file.txt | awk 'BEGIN{FS="|"};{print... (3 Replies)
Discussion started by: chatwithsaurav
3 Replies

2. UNIX for Dummies Questions & Answers

Comparing to specific line in file bash script

Hi, I have a value stored in x and I need to compare it to the numbers in every other line of a file. The file contains alternating lines of numbers and letters: aaaa1111 AAAAAAAA bbbb2222 BBBBBBBB cccc3333 CCCCCCCC I need to compare x to the numbers in every other line without the... (2 Replies)
Discussion started by: ShiGua
2 Replies

3. Shell Programming and Scripting

Korn Shell script to insert at specific line

Hi, I am trying to put together a Korn Shell script to insert at a specific line. The system we use is SunOS 5.10 I can get the line number by using:- num=`sed -n '/export ENV/=' ./tmp.file` Not getting much headway using the above variable's value to insert - export SYBASE=/opt/sybase15... (5 Replies)
Discussion started by: aj8200
5 Replies

4. Shell Programming and Scripting

Script to monitor directory size of specific users

Hi, i am new to shell scripts, i need to write a script that can monitor size of directory of specific users. Please help. Thanks, Nitin (2 Replies)
Discussion started by: nicksrulz
2 Replies

5. Shell Programming and Scripting

Shell script to find specific file name and load data

I need help as to how to write a script in Unix for the following: We have 3 servers; The mainframe will FTP them to a folder. In that folder we will need the script to look and see if the specific file name is there and load it to the correct table. Can anyone pls help me out with... (2 Replies)
Discussion started by: msrahman
2 Replies

6. Shell Programming and Scripting

How to filter required data from file using bash script?

Hi All , I have one file like below , Owner name = abu2-kxy-m29.hegen.app Item_id = AX1981, special_id = *NULL*, version = 1 Created = 09/01/2010 12:56:56 (1283389016) Enddate = 03/31/2011 00:00:00 (1301554800) From the above file I need to get the output in the below format ,i need... (3 Replies)
Discussion started by: gnanasekar_beem
3 Replies

7. Shell Programming and Scripting

Need help in writing a script to create a new text file with specific data from existing two files

Hi, I have two text files. Need to create a third text file extracting specific data from first two existing files.. Text File 1: Format contains: SQL*Loader: Release 10.2.0.1.0 - Production on Wed Aug 4 21:06:34 2010 some text ............so on...and somwhere text like: Record 1:... (1 Reply)
Discussion started by: shashi143ibm
1 Replies

8. Shell Programming and Scripting

Monitor file generation for an hour using Korn shell script

Hi, I am new to this unix scripting, I got a requirement like.. Files with *.XML extension will be generating in a /home/sample/ folder for every 15 mins. I need to monitor those files in that particular folder for every hour. If no file has been generated in that particular folder for an... (7 Replies)
Discussion started by: siri_886
7 Replies

9. AIX

how to check the existence of a file using korn shell?

we have tranferred an ear from local server to remote server using ftp.consider, we have an ear file named a.ear in remote server,again if we transfer the same file named a.ear from local server to remote server.we need the kshell to check the existence of the ear file in remote server,and if the... (3 Replies)
Discussion started by: karthikprasathk
3 Replies
Login or Register to Ask a Question