Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Script to search for a pattern in 30 minutes from a log file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to search for a pattern in 30 minutes from a log file
# 1  
Old 04-29-2014
Script to search for a pattern in 30 minutes from a log file
Hello All,

I have to write a script which will search for diffrent patterns like "Struck" "Out of Memory" , etc from a log file in Linux box's.
Now I will be executing a cron job to find out the results by executing the script once in every 30 minutes.
suppose time is
14-04-29:05:31:09
So I will be searching the pattern as per below:
14-04-29:05:01:09.
Can some one help me on this please?

I can find out the date before 30 mins ie by this
Code:
date -d '-30 min'.

But what should be the set of commands to search the strings as entry would be some what similar to this in the log file:

Code:
29.04.2014 05:59:20.778 *INFO* [[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'] log.request 29/Apr/2014:05:59:20 -0700 [87146] <- 200 text/html 102ms
29.04.2014 05:59:20.778 *INFO* [[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'] log.access 127.0.0.1 - admin 29/Apr/2014:05:59:20 -0700 "GET /system/console/bundles HTTP/1.1" 200 74695 "-" "Java1.6.0_31"
29.04.2014 05:59:35.803 *INFO* [[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] log.request 29/Apr/2014:05:59:35 -0700 [87147] -> GET /system/console/bundles HTTP/1.1
29.04.2014 05:59:35.905 *INFO* [[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] log.request 29/Apr/2014:05:59:35 -0700 [87147] <- 200 text/html 102ms
.
.
.
.

thanks
Last edited by vbe; 04-29-2014 at 10:29 AM.. Reason: code tags , rm perso mail
# 2  
Old 04-29-2014
You could save the size of your log file (in bytes) for the next run:
Code:
        skip=$(stat -c%s file)
        dd if=file iflag=skip_bytes skip=$skip 2>/dev/null | grep -E "Struck|Out of Memory"

# 3  
Old 04-29-2014
Thanks Rudi for your reply,

Let me re frame my question.
What I have to do is to search for the string Struck Threads or Out of Memory from the log file.
I need to prepare a script which will be running evry 30 mins such that it will list out these in between last 30 mins.
This Log file will be of a large size as there will be many struck threads or memory issue.
So I need the logs only from the last 30 mins and not all.
I tried your commands but it dint help.
# 4  
Old 04-29-2014
OK, let me rephrase my answer (assuming that your system provides the commands and options):
Code:
grep -E "Struck|Out of Memory"; skip=$(stat -c%s file)                              # search keywords and keep file size
... wait 30 mins
dd if=file iflag=skip_bytes skip=$skip 2>/dev/null | grep -E "Struck|Out of Memory" # skip former file size, copy all lines since last time to stdout, pipe to grep searching keywords
skip=$(stat -c%s file)                                                              #save new files size

Put this in a loop. Of course you need to play around with this, adapt and test...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Check file creation Time minutes and if file older then 5 minutes execute some stuff

Hello all, Info: System RedHat 7.5 I need to create a script that based on the creation time, if the file is older then 5 minutes then execute some stuff, if not exit. I thought to get the creation time and minutes like this. CreationTime=$(stat -c %y /tmp/test.log | awk -F" " '{ print... (3 Replies)
Discussion started by: charli1
3 Replies

2. Shell Programming and Scripting

Grep a log file for the last 5 minutes of contents every 5 minutes

Hi all, System Ubuntu 16.04.3 LTS i have the following log INFO 2019-02-07 15:13:31,099 module.py:700] default: "POST /join/8550614e-3e94-4fa5-9ab2-135eefa69c1b HTTP/1.0" 500 2042 INFO 2019-02-07 15:13:31,569 module.py:700] default: "POST /join/6cb9c452-dcb1-45f3-bcca-e33f5d450105... (15 Replies)
Discussion started by: charli1
15 Replies

3. Shell Programming and Scripting

Scan of log file in Linux for entries in last 15 minutes for matching a pattern

Is there any way I can do scan of log file in Linux, where the log file entries for last 15 minutes can be searched for a particular pattern. The log file entries are in below format. 2014-01-27T23:08:53.924-0500 LDAP authentication error 2014-01-27T23:08:53.934-0500 LDAP authentication... (4 Replies)
Discussion started by: anandrudran
4 Replies

4. Shell Programming and Scripting

How to use sed to search a particular pattern in a file backward after a pattern is matched.?

Hi, I have two files file1.txt and file2.txt. Please see the attachments. In file2.txt (which actually is a diff output between two versions of file1.txt.), I extract the pattern corresponding to 1172c1172. Now ,In file1.txt I have to search for this pattern 1172c1172 and if found, I have to... (9 Replies)
Discussion started by: saurabh kumar
9 Replies

5. Shell Programming and Scripting

Search for a pattern in a String file and count the occurance of each pattern

I am trying to search a file for a patterns ERR- in a file and return a count for each of the error reported Input file is a free flowing file without any format example of output ERR-00001=5 .... ERR-01010=10 ..... ERR-99999=10 (4 Replies)
Discussion started by: swayam123
4 Replies

6. Shell Programming and Scripting

Search a pattern in a log file

I have file which gets updated every minute/second. Is it possible in shell scripting that I can search for some pattern infinitely in this file and if it finds that pattern, alert the user. A sample of log file is below. The following file is getting updated every second. I want to alert the... (2 Replies)
Discussion started by: Tuxidow
2 Replies

7. Solaris

Search date pattern in application log file

I am viewing a file in vi editor and would like to search for a date pattern. In the log, the timestamp is enclosed in parentheses ''. I am using the '/' option in vi to search for the pattern. log snippet: 000000f4 ServletWrappe I SRVE0242I: : Initialization successful. 000000f4... (3 Replies)
Discussion started by: Vangogh78
3 Replies

8. Shell Programming and Scripting

How can i search a file which has been created or modified in last five minutes

Hi Can some one please help me How can i search a file which has been created or modified in last five minutes I have used the command find . -mmin -5 and it does not work i get an error -mmin is bad option Please help Much regards Tarun (2 Replies)
Discussion started by: tarundeepdhawan
2 Replies

9. Shell Programming and Scripting

File search for pattern - script

Hi All, I have two directories as 1) mi/job -> job1.sh, job2.sh, job3.sh 2) mi/sysin -> sysin1, sysin2, sysin3,sysin4 I want to wrrite a script such that it should accept two parameters as directory paths. $ myscript mi/sysin mi/job. The script should be able to scan all files... (3 Replies)
Discussion started by: rahulrathod
3 Replies

10. Shell Programming and Scripting

Search file for pattern and grab some lines before pattern

I want to search a file for a string and then if the string is found I need the line that the string is on - but also the previous two lines from the file (that the pattern will not be found in) This is on solaris Can you help? (2 Replies)
Discussion started by: frustrated1
2 Replies
Login or Register to Ask a Question