Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Scan a file in realtime and execute certain commands on encountering 5 consecutive identical lines

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Scan a file in realtime and execute certain commands on encountering 5 consecutive identical lines
# 1  
Old 02-09-2012
Question Scan a file in realtime and execute certain commands on encountering 5 consecutive identical lines
Mysql log has something like below:

Quote:
1649560 Query select * from actions where uniqueid like 'ernie-1328697839.1233158-%-dequeue' and class='queue' and server='ernie' and time<='1328698322' and action='dequeue' limit 1
1649560 Query select * from actions where uniqueid like 'ernie-1328697839.1233158-%-dequeue' and class='queue' and server='ernie' and time<='1328698322' and action='dequeue' limit 1
1649560 Query select * from actions where uniqueid like 'ernie-1328697839.1233158-%-dequeue' and class='queue' and server='ernie' and time<='1328698322' and action='dequeue' limit 1
1649560 Query select * from actions where uniqueid like 'ernie-1328697839.1233158-%-dequeue' and class='queue' and server='ernie' and time<='1328698322' and action='dequeue' limit 1
1649560 Query select * from actions where uniqueid like 'ernie-1328697839.1233158-%-dequeue' and class='queue' and server='ernie' and time<='1328698322' and action='dequeue' limit 1
1649560 Query select * from actions where uniqueid like 'ernie-1328697839.1233158-%-dequeue' and class='queue' and server='ernie' and time<='1328698322' and action='dequeue' limit 1
1649560 Query select * from actions where uniqueid like 'ernie-1328697839.1233158-%-dequeue' and class='queue' and server='ernie' and time<='1328698322' and action='dequeue' limit 1
1649560 Query select * from actions where uniqueid like 'ernie-1328697839.1233158-%-dequeue' and class='queue' and server='ernie' and time<='1328698322' and action='dequeue' limit 1
I need a bash shell script that will do the following:

1) The script will scan the mysql.log file constantly in real time (something like tail -F mysql.log)
2) If it encounters 5 consecutive identical lines then it would invoke some commands (say echo "yes")

Please help.
Last edited by proactiveaditya; 02-09-2012 at 08:53 AM..
# 2  
Old 02-09-2012
This appears to be a continuation of:
https://www.unix.com/shell-programmin...ignP=302596963

What does "tail -F" do on your Operating System? Not valid on mine. What Operating System and version do you have?

In general it is a major programming exercise to "tail -f" a file and carry out any further processing in a pipeline. There have been posts on this board which suggest using named pipes.
# 3  
Old 02-09-2012
tail -F is same as tail -f --retry

Code:
man tail
       --retry
              keep trying to open a file even if it is inaccessible when tail starts or if it becomes inaccessible later; useful when  following  by  name,  i.e., with --follow=name

Available on GNU/Linux
# 4  
Old 02-09-2012
If you are using GNU/Linux checkout fflush() in gawk:
Code:
tail -F infile | awk 'function ff(){ if(A[p]>4) {print "yes - " p; fflush() ; A[p]=0 } } {ff(); A[$0]++} $0!=p{p=$0} END{ff()}'

*edit* Actually, this should suffice, since the END section will never happen:
Code:
tail -f infile | awk 'A[p]>4{print "yes - " p; fflush(); A[p]=0} $0!=p{p=$0} {A[p]++}'

Last edited by Scrutinizer; 02-09-2012 at 02:30 PM..
# 5  
Old 02-09-2012
What about:
Quote:
tail -F mysql.log|<THE_BELOW_AWK_SCRIPT>
Quote:
awk 'BEGIN {l="";c=0}
$0 != l { l=$0; c=0 }
l=$0 {c++; if (c>=5) {system("echo Yes")}}'
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk - If field value of consecutive records are the identical print portion of lines

I have some data that looks like this: PXD= ZW< 1,6 QR> QRJ== 1,2(5) QR> QRJ== 4,1(2) QR> QRJ== 4,2 QRB= QRB 4,2 QWM QWM 6,2 R<C ZW< 11,2 R<H= R<J= 6,1 R>H XZJ= 1,2(2) R>H XZJ= 2,6(2) R>H XZJ= 4,1(2) R>H XZJ= 6,2 RDP RDP 1,2 What I would like to do is if fields $1 and $2 are... (5 Replies)
Discussion started by: jvoot
5 Replies

2. Shell Programming and Scripting

Grep 2 consecutive lines and replace the second line in a file

I have a file lake this cat ex1.txt </DISCOUNTS> <B2B_SPECIFICATION elem="0"> <B2B_SPECIFICATION elem="0"> <DESCR>Netti 2 </DESCR> <NUMBER>D02021507505</NUMBER> </B2B_SPECIFICATION> <B2B_SPECIFICATION elem="1"> <DESCR>Puhepaketti</DESCR>... (2 Replies)
Discussion started by: Dhoni
2 Replies

3. UNIX for Dummies Questions & Answers

Delete 26 consecutive lines in a file

I have a text file that is about 90,000 lines long. How would I delete lines 64-89, 152-177, 240-265, 328-353... etc? The sections I would like to delete are 26 lines long and the number of lines between the sections I would like to delete is 62 lines. Thanks very much in advance. (6 Replies)
Discussion started by: MDeBiasse
6 Replies

4. Shell Programming and Scripting

Removing consecutive lines in a file

We have very large transaction logs that have transactions which start with a line that starts with 'Begin :' and ends with a line that starts with 'End :'. For most transactions there is valid data between those two lines. I am trying to get rid of lines that look like this: Begin :... (11 Replies)
Discussion started by: deneuve01
11 Replies

5. Shell Programming and Scripting

finding and removing 2 identical consecutive words in a text

i want to write a shell script that correct a text file.for example if i have the input file: "john has has 2 apples anne has 3 oranges oranges" i want that the output file be like this: "john has 2 apples anne has 3 oranges" i've tried to read line by line from input text file into array... (11 Replies)
Discussion started by: cocostaec
11 Replies

6. Shell Programming and Scripting

how to delete two consecutive lines from the file

Hi guys I am deleting a unique line from the file and also need to remove the line above it which is NOT unique and servers as a record separator. Here is an example: # 101 803E 823F 8240 # 102 755f 4F2A 4F2B # 290 747D 0926 0927 # 999 8123 813E ... (5 Replies)
Discussion started by: aoussenko
5 Replies

7. Shell Programming and Scripting

Find time difference between two consecutive lines in same file.

Hello I have a file in following format: IV 08:09:07 NM 08:12:01 IC 08:12:00 MN 08:14:20 NM 08:14:15 I need a script to compare time on each line with previous line and show the inconsecutive line. Ex.: 08:12:00 08:14:15 A better way... (6 Replies)
Discussion started by: vilibit
6 Replies

8. Shell Programming and Scripting

merging of 2 consecutive lines in a file for a specific pattern

Hi , I'm looking for a way to merge two lines only for a given pattern / condition. Input : abcd/dad + -49.201 2.09 -49.5 34 ewrew rewtre * fdsgfds/dsgf/sdfdsfasdd + -4.30 0.62 -49.5 45 sdfdsf cvbbv * sdfds/retret/asdsaddsa + ... (1 Reply)
Discussion started by: novice_man
1 Replies

9. UNIX for Dummies Questions & Answers

How to scan only new lines added in file?

Hi, I am planning to implement a scheduled script that will go against my log files (every hour), search for a set of key words (errors, exceptions, faults etc). The script must be intelligent enough to scan only the new lines added to the log file since it last ran. I can use grep for... (3 Replies)
Discussion started by: redlotus72
3 Replies

10. UNIX for Dummies Questions & Answers

Cutting n consecutive lines from a file...

Hi, I have this problem of separating 10 consecutive lines from a file, say starting from 21 to 30... I have used a filter like this.. head -n 30 myfile | tail -n 10 Is there a simpler way than this? (2 Replies)
Discussion started by: Vishnu
2 Replies
Login or Register to Ask a Question