Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Script to count matched string then check again from last read position

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to count matched string then check again from last read position
# 1  
Old 01-23-2017
Script to count matched string then check again from last read position
Hi all,

I am currently trying to figure out how can i capture a particular string from a log file, then search again but from the last line it read before onward.
Basically let's say that the check runs every 5 mins via cron, counting the number of matched strings "Cannot assign requested address" from the log sample below:

Code:
Nov 28 18:18:09.005100 diameterControlAgent(3426) NOTICE: {960112} Peer: 'jinny': adding permitted origin host 'AM1RTC01.NL.ASPIDERSOLUTIONS.COM'
Nov 28 18:18:09.005649 diameterControlAgent(3426) NOTICE: {960112} Peer: 'jinny': adding permitted origin host 'AM1RTC02.NL.ASPIDERSOLUTIONS.COM'
Nov 28 18:18:09.009531 diameterControlAgent(3426) NOTICE: {960113} Peer: 'jinny': adding Remote Address '<ip address>'
Nov 28 18:18:09.010373 diameterControlAgent(3426) NOTICE: {960113} Peer: 'jinny': adding Remote Address 'ip address>'
Nov 28 18:18:09.010873 diameterControlAgent(3426) NOTICE: {960114} Loading peer: 'jinny': netmaskBits: 32
Nov 28 18:18:09.011362 diameterControlAgent(3426) NOTICE: {960115} Loading peer: 'jinny': permittedInstances: 0
Nov 28 18:18:09.011981 diameterControlAgent(3426) NOTICE: {960116} Loading peer: 'jinny': reqSctpInboundStreams: 8
Nov 28 18:18:09.012590 diameterControlAgent(3426) NOTICE: {960117} Loading peer: 'jinny': reqSctpOutboundStreams: 8
Nov 28 18:18:09.014542 diameterControlAgent(3426) NOTICE: {960013} Configuration Loaded
Nov 28 18:18:09.016790 diameterControlAgent(3426) ERROR: {969108} tcp:::<ip address:port>: bind to <ip address:port> failed: Cannot assign requested address.

The next time the script runs again, the aim is to continue searching for this string from the last "line" (i.e end of file) it read previously.

Hence either using AWK or grep -c. The problem s is how to make the script intelligent enough to search from the last read line onward. The log file changes continuously.

is it possible to achieve this?

I was thinking maybe using either the tail command to store the output then match it, but am afraid that most probably there would be similar matches.
Else maybe try a word count?

Do you have ideas?
# 2  
Old 01-23-2017
If you know the last read line number, you can use sed to get the lines past that point:-
Code:
last_seen_record=12345            # .... or whatever you have determined it to be
sed -n "$last_seen_record,$"p  $logfile

Of course, the output is then from the point you specify onward, so working out the next last_see_record value will be tricky. You might be better with:-
Code:
last_seen_record=12345            # .... or whatever you have determined it to be
grep -c "" $logfile | sed -n "$last_seen_record,$"p

This will give you output prefixed with the line number of the whole file so you can work with that to save away the appropriate value for next time.

If you have large log files, you might hit a timing problem if you try to count the number of lines read as a separate operation with wc because of the delay between counting the lines and the process that you your to read the information out.

An alternate would be to use awk to both extract the data and keep a record count in a single operation and we can explore that if you think that is more appropriate, but it can be a little harder to grasp.


Can you post the output from uname -a so we know what OS and version we're working with, because there are variations we may need to consider.



Robin
# 3  
Old 01-23-2017
Hi Robin,

As for knowing the last read number, i guess i can ontain that by using the current number of lines using wc or a one liner with perl
Code:
perl -lne 'END { print $. }' $LOG_FILE

but the problem i'm trying to figure out is how to let the script know how to read from the last read line onward
Code:
#File path
LOG_FILE=/IN/service_packages/DCA/tmp/diameterControlAgent_rtcg_tmnl.log
#Current lines
last_record=`perl -lne 'END { print $. }' $LOG_FILE`
#Get count
grep -c "" $LOG_FILE | sed -n "$last_record,$"p

Of course the above is not correct. The grep will still count from the beginning of the file.

The OS here is Solaris 11. SunOS 5.11 11.2 sun4v sparc sun4v
# 4  
Old 01-23-2017
Not sure I understood correctly what you're after, but if you want to continue processing a file from where you stopped last time, this post and the reference therein might be interesting...
# 5  
Old 01-23-2017
An alternate may be to open the file with tail and keep it open.

What is your overall aim for this process? You might be able to:-
Code:
while read line
do
   process "$line"
done < <(tail -f $logfile)

I think that the < <(command) is a bash construct rather than sh or ksh so it depends if you have that or even if this is appropriate. This process will not end, but continue reading the log file when new records are written, so it's near-time (strictly not quite real-time)

Beware that the log file will remain open, so if you rotate your log files, you might need to use tail -F $logfile instead to open the replacement file.


Can you tell us what the point of this code is to be? There may be a simpler way that I'm not thinking of. It would be useful to know the OS version (output from uname -a) and shell that you are using.



Kind regards,
Robin
# 6  
Old 01-23-2017
Not sure exactly what you want to do with the output (maybe send an email ??) but the below script will search for the text and then if found send an email and then continue on from the last line searched.

Code:
#!/bin/ksh

while true
do
line_no=$(wc -l log_file | awk '{line=$1 +1; print line}')
sleep 20
tail +$line_no log_file| grep "Cannot assign requested address" > /tmp/err.txt
    if [ $? -eq 0 ]
then mailx -s "There is an error" emailaddress < /tmp/err.txt
    fi
done

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Count exact matched words

hi , i have a file test.dat which contains following data. test.dat XY|abc@xyz.com XY|abc@xyz.com ST|abc@xyz.com ST|abc@xyz.com ST|XYZ@abc.com FK|abc@xyz.com FK|STG@xyz.com FK|abc@xyz.com FK|FKG@xyz.com i want to know the count of XY,ST,FK. i.e XY = 2 , ST = 3 , FK = 4 I am... (4 Replies)
Discussion started by: itzkashi
4 Replies

2. Shell Programming and Scripting

Search for a string at a particular position and replace with blank based on position

Hi, I have a file with multiple lines(fixed width dat file). I want to search for '02' in the positions 45-46 and if available, in that lines, I need to replace value in position 359 with blank. As I am new to unix, I am not able to figure out how to do this. Can you please help me to achieve... (9 Replies)
Discussion started by: Pradhikshan
9 Replies

3. UNIX for Dummies Questions & Answers

I need a Script to read Log string and check date at the same time

I need to check 1 log file, which is logging: 2014-08-18T09:10:39+02:00 user: XXXXX START FEATURE 2014-08-18T09:10:39+02:00 user: XXXXX FINISH FEATURE I first need to check that the START FEATURE starts and finish on the same time/date for the same user, which is different each time START... (2 Replies)
Discussion started by: TheBest43
2 Replies

4. Shell Programming and Scripting

Write a word at 72nd position of a matched line in a file

Hi, I need to search a file for a pattern,replace some other word and write a word at its 72nd position. For example, My name is Mano.Im learning Unix. I want to search the file in all lines containing the word "Mano".In that matched line,replace the word "Unix" with "Java".And... (5 Replies)
Discussion started by: mano1 n
5 Replies

5. Shell Programming and Scripting

Script To read Date and count from files

Smaple 1 Date and Time: 2013-05-11 12:23:12 MST abc,1234,hi-all,45354-88888,IN,US XYZ,1234,hi-all,45354-88888,OUT,GB abc,1234,hi-all,45354-88888,IN,AS abc,1234,hi-all,45354-88888,OUT,US abc,1234,hi-all,45354-88888,IN,US Number of Records: 000005 Sample 2 HDR: 20130511... (1 Reply)
Discussion started by: Abhisrajput
1 Replies

6. Shell Programming and Scripting

sed or awk command to replace a string pattern with another string based on position of this string

here is what i want to achieve... consider a file contains below contents. the file size is large about 60mb cat dump.sql INSERT INTO `table1` (`id`, `action`, `date`, `descrip`, `lastModified`) VALUES (1,'Change','2011-05-05 00:00:00','Account Updated','2012-02-10... (10 Replies)
Discussion started by: vivek d r
10 Replies

7. Shell Programming and Scripting

how to find a position and print some string in the next and same position

I need a script for... how to find a position of column data and print some string in the next line and same position position should find based on *HEADER8* in text for ex: ord123 abs 123 987HEADER89 test234 ord124 abc 124 987HEADER88 test235 ... (1 Reply)
Discussion started by: naveenkcl
1 Replies

8. UNIX for Dummies Questions & Answers

Count of matched pattern occurance

In a file a pattern is occured many times randomly. Even it may appear more then once in the same line too. How i can get the number of times that pattern appeared in the file? let the file name is abc.txt and the pattern is "xyz". I used the following code: grep -ic "xyz" abc.txt but it is... (3 Replies)
Discussion started by: palash2k
3 Replies

9. Shell Programming and Scripting

Read string from a file,plz help me to check

#!/usr/bin/ksh exec 0<property while read newReceiveDir do if then sed -e 's//home/joshua/bodi/data/receive/{$newReceiveDir:25}/g/' mp_validate.sh >| mp_validate.sh elif then sed -e 's//home/joshua/bodi/data/temp/{$newReceiveDir:22}/g/' mp_validate.sh >| mp_validate.sh ... (1 Reply)
Discussion started by: joshuaduan
1 Replies

10. Shell Programming and Scripting

read string, check string length and cut

Hello All, Plz help me with: I have a csv file with data separated by ',' and optionally enclosed by "". I want to check each of these values to see if they exceed the specified string length, and if they do I want to cut just that value to the max length allowed and keep the csv format as it... (9 Replies)
Discussion started by: ozzy80
9 Replies
Login or Register to Ask a Question