Check specific content from log file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Check specific content from log file
# 1  
Old 02-14-2017
Check specific content from log file

Hi all,

i have a logfile which is continuously being updated. I have built a script to check for a specific content and if it is found, it sends a string into a file. Here's the current script:

Code:
#!/bin/bash

logfile=/opt/jboss-eap-6.3/standalone/log/server.log

tail -fn0 $logfile | \
while read line ; do
        echo "$line" | grep -q "SocketTimeoutException invoking http://10.192.1.8:8001: connect timed out"
        if [ $? = 0 ]
        then
            echo "ConnectionTimeout_ESB-OldStack=YES" > /home/nms/Disconnection_oldstack.txt
        else
            echo "ConnectionTimeout_ESB-OldStack=NO" > /home/nms/Disconnection_oldstack.txt
        fi
done

As you can see i am continuously reading from the "server.log" file and while reading each entry, if the "grep" command find the matched string it echoes to file, else it echos to the same file but with a different string (YES or NO).

However i noticed that when the end of the day is reached, i.e @ 23:59, the file "/home/nms/Disconnection_oldstack.txt" stops updating. I noticed that the "server.log" file closes and a new one is reopened at that time, here below:

Code:
-rw-r--r--. 1 jboss jboss  29304874 Feb 10 23:59 server.log.2017-02-10.gz
-rw-r--r--. 1 jboss jboss  21277719 Feb 11 23:59 server.log.2017-02-11.gz
-rw-r--r--. 1 jboss jboss 445767175 Feb 12 23:59 server.log.2017-02-12
-rw-r--r--. 1 jboss jboss 782710795 Feb 13 23:59 server.log.2017-02-13
-rw-r--r--. 1 jboss jboss 143127996 Feb 14 09:40 server.log

i tried to add
Code:
while [ ! -f $logfile ]; do sleep 5; done

in the beginning of the script so it sleeps if file is does not exist (since i thought that at the time of the file rotation it stops since the file is not found), but the same thing happened.
Am i missing something?
Note: The output redirection to file is necessary in this case so i need to stick to that!

Thanks in advance for your feedback!
# 2  
Old 02-14-2017
Which OS and version are you using? The output from uname -a would be very helpful.

You might find that tail has a -F flag in your version. have a look in the man page and see if it is supported.


I hope that this helps,
Robin
This User Gave Thanks to rbatte1 For This Post:
# 3  
Old 02-14-2017
If I understand what is happening, the process that builds the server.log file closes the file at 23:59. It then renames the file and open a new server.log file.

I know you tried to wait a few seconds; what happens if you try to wait longer?

While the old file may close and rename, perhaps the new file does not get created until the first event to cause something to be written to it.

Unsure on what part of that sript makes it continuously read. The code snippet seems to be a one-time pass-thru.
This User Gave Thanks to joeyg For This Post:
# 4  
Old 02-14-2017
Quote:
Originally Posted by joeyg
Unsure on what part of that sript makes it continuously read. The code snippet seems to be a one-time pass-thru.
tail -f
This User Gave Thanks to Corona688 For This Post:
# 5  
Old 02-14-2017
Elaborating on what rbatte1 already alluded to above: the man page for tail (GNU coreutils) 8.25 reads:
Quote:
With --follow (-f), tail defaults to following the file descriptor, which means that even if a tail'ed file is renamed, tail will continue to track its end. This default behavior is not desirable when you really want to track the actual name of the file, not the file descriptor (e.g., log rotation). Use --follow=name in that case. That causes tail to track the named file in a way that accommodates renaming, removal and creation.
Which I guess would exactly solve your problem.
These 3 Users Gave Thanks to RudiC For This Post:
# 6  
Old 02-15-2017
Hi,

Thanks for your feedback. I modified the script to use -F instead and increased the sleep time (although this might not be really needed since -F helps in cases rotated log files.
We'll see how it goes and keep you posted.
fyi the OS is a CentOS release 6.6

Code:
#!/bin/bash

logfile=/opt/jboss-eap-6.3/standalone/log/server.log

while [ ! -f $logfile ]; do sleep 60; done

tail -F $logfile | \
while read line ; do
        echo "$line" | grep -q "SocketTimeoutException invoking http://10.192.1.8:8001: connect timed out"
        if [ $? = 0 ]
        then
            echo "ConnectionTimeout_ESB-OldStack=YES" > /home/nms/Disconnection_oldstack.txt
        else
            echo "ConnectionTimeout_ESB-OldStack=NO" > /home/nms/Disconnection_oldstack.txt
        fi
done

# 7  
Old 02-15-2017
I'm not sure if you expoect lots of messages to rush through at any time, but that could be painful calling grep for each and every one one.
If the string you are checking is exactly what you show (no leading date string or other stuff) might it be better to code like this:-
Code:
#!/bin/bash

alert_string="SocketTimeoutException invoking http://10.192.1.8:8001: connect timed out"
logfile=/opt/jboss-eap-6.3/standalone/log/server.log

while read line
do
   if [ "$line" = "$alert_string" ]
   then
      echo "ConnectionTimeout_ESB-OldStack=YES" > /home/nms/Disconnection_oldstack.
   else
      echo "ConnectionTimeout_ESB-OldStack=NO" > /home/nms/Disconnection_oldstack.txt
   fi
done < <(tail -F $logfile)

If the string you are looking for is only part of the line, then this might be better:-
Code:
#!/bin/bash

alert_string="SocketTimeoutException invoking http://10.192.1.8:8001: connect timed out"
logfile=/opt/jboss-eap-6.3/standalone/log/server.log

while read line
do
   test_line="${line%${alert_string}*}"
   if [ "$line" != "$test_line" ]
   then
      echo "ConnectionTimeout_ESB-OldStack=YES" > /home/nms/Disconnection_oldstack.
   else
      echo "ConnectionTimeout_ESB-OldStack=NO" > /home/nms/Disconnection_oldstack.txt
   fi
done < <(tail -F $logfile)

Both of these negate the need to make the external call to grep The second one uses variable substitution to cut off the string from the end of the line so if there is a leading timestamp, that will be put into variable test_line and the comparison done , which will then go to the then section. if the alert string is not in the line, then the whole line is put into test_line and the comparison will match so we go to the else section.

I did also wonder if you really want to be overwriting the output file each time your read a record too, as this could cause a heavy IO load to re-write a single line in the file many times over. If you keep the current state in a variable, you can also avoid the unnecessary IO and only re-write the file each time the state changes.

Perhaps this would do both:-
Code:
#!/bin/bash

alert_string="SocketTimeoutException invoking http://10.192.1.8:8001: connect timed out"
alert_state=""
logfile=/opt/jboss-eap-6.3/standalone/log/server.log

while read line
do
   test_line="${line%${alert_string}*}"
   if [ "$line" != "$test_line" ]
   then
      if [ "$alert_state" != "YES" ]
      then
         echo "ConnectionTimeout_ESB-OldStack=YES" > /home/nms/Disconnection_oldstack.
         alert_state="YES"
   else
      if [ "$alert_state" != "NO" ]
      then
         echo "ConnectionTimeout_ESB-OldStack=NO" > /home/nms/Disconnection_oldstack.txt
         alert_state="NO"
   fi
done < <(tail -F $logfile)

If you just want to append to the output file, put the output redirection on the whole loop, i.e. after the done and the input redirection.


I hope that this helps,
Robin

Last edited by rbatte1; 02-15-2017 at 10:31 AM.. Reason: Added the logfile definition
This User Gave Thanks to rbatte1 For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Content merging at a specific location in a file

Hi, This is a bit lengthy problem, i will try to keep explaining it simple. I have got a file say file1 that contains the following in it, ------------------------------------------------------------------------ r201463 | ngupta@gmail.com | 2012-06-19 22:02:20 +0530 (Tue, 19 Jun 2012) |... (3 Replies)
Discussion started by: Kashyap
3 Replies

2. Shell Programming and Scripting

Check content of file

Hi All, I m very new to unix...i jus want to chk the content of file. ma requirement is if file has a content then display it else dont display or something pls specify which loop shalli use either for or while?? (20 Replies)
Discussion started by: navsan
20 Replies

3. Shell Programming and Scripting

want to search some content of the file with specific to user

I have some users in one unix system and i want to search some files with specific to user and then i want to find some content inside that file so can u help me how we can implement it? File location is as below. /pools/home_unix/cmadireddy/work/models/model/ cmadireddy is user name. now... (6 Replies)
Discussion started by: lathigara
6 Replies

4. Shell Programming and Scripting

Extracting content from a file in specific format

Hi All, I have the file in this format **** Results Data **** Time or Step 1 2 20 0.000000000e+00 0s 0s 0s 1.024000000e+00 Us 0s 0s 1.100000000e+00 1s 0s 0s 1.100000001e+00 1s 0s 1s 2.024000000e+00 Us Us 1s 2.024000001e+00 ... (7 Replies)
Discussion started by: diehard
7 Replies

5. Shell Programming and Scripting

split file content into specific folders

Hi I have a large text file and I want to split its content into multiple flies. this large file contains several blocks of codes separated by a comment line for each block. this comment line represents a directory path So, when separate these blocks each into a separate file, This output... (7 Replies)
Discussion started by: turki_00
7 Replies

6. Shell Programming and Scripting

want to print the file content from the specific line

Hi All, I would like to print the content from the specific line of a file . For example... i have file abc.txt which has 100 lines of code ,from this file i would like to print the content from 20,19,18th line......like that Regards Srikanth (4 Replies)
Discussion started by: srikanthg
4 Replies

7. Shell Programming and Scripting

Extract specific content from a file

My input file: >sequence_1 ASSSSSSSSSSSDDDDDDDDDDDCCCCCCC ASDSFDFFDFDFFWERERERERFSDFESFSFD >sequence_2 ASDFDFDFFDDFFDFDSFDSFDFSDFSDFDSFASDSADSADASD ASDFFDFDFASFASFASFAFSFFSDASFASFASFAFS >sequence_3 VEDFGSDGSDGSDGSDGSDGSDGSDG dDFSDFSDFSDFSDFSDFSDFSDFSDF SDGFDGSFDGSGSDGSDGSDGSDGSDG My... (22 Replies)
Discussion started by: patrick87
22 Replies

8. Shell Programming and Scripting

Remove specific content in a file

Hi, I have a file called fl_list consists of files i have to archive. I want to create a exception parm called except_parm, so if it finds the directory it will not archive these files and remove from fl_list. $ cat fl_list /apps/dev/ihub/ready/IA003B/IA003B_Deal_Header_yyyymmdd_hhmmss.txt... (1 Reply)
Discussion started by: k9cheung
1 Replies

9. Shell Programming and Scripting

To check the content of one file in another

Hi , I have a file called "X" . the content of X are X -- abc def and i have a file called "Y" , the content of Y are Y -- erty sdss s abc sfs def I need to check if the content of file X is contained in Y.Only unix (7 Replies)
Discussion started by: giri_luck
7 Replies

10. AIX

find for specific content in file in the directory and list only file names

Hi, I am trying to find the content of file using grep and find command and list only the file names but i am getting entire file list of files in the directory find . -exec grep "test" {} \; -ls Can anyone of you correct this (2 Replies)
Discussion started by: madhu_Jagarapu
2 Replies
Login or Register to Ask a Question