Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

How to 'improve' this script and also 'fix' the pattern matching part?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers How to 'improve' this script and also 'fix' the pattern matching part?
# 1  
Old 10-22-2019
How to 'improve' this script and also 'fix' the pattern matching part?
Hi all,

Below is my script. It is currently working but I want some advice on maybe improving it and need some help on the pattern matching

Code:
xx.ksh:

#!/bin/ksh
#
# -------------------------------------------------------------------------------------------------
#
#Fatal NI connect error 12170.
#
#  VERSION INFORMATION:
#        TNS for Linux: Version 11.2.0.4.0 - Production
#        Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
#        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
#  Time: 18-OCT-2019 04:33:41
#  Tracing not turned on.
#  Tns error struct:
#    ns main err code: 12535
#
#TNS-12535: TNS:operation timed out
#    ns secondary err code: 12606
#    nt main err code: 0
#    nt secondary err code: 0
#    nt OS err code: 0
#  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=12.123.12.12)(PORT=49931))
#WARNING: inbound connection timed out (ORA-3136)
#Fri Oct 18 04:33:44 2019
#
# -------------------------------------------------------------------------------------------------
#

echo
echo "- Parsing $1 ..."
echo
file_to_parse=$1

cat /dev/null > tmpfile.00
cat /dev/null > tmpfile.01
cat /dev/null > tmpfile.02

grep -in "Fatal NI connect error 12170" ${file_to_parse} | awk -F":" '{ print $1"^"$1+17 }' > tmpfile.00

while read line
do
   start=`echo $line | awk -F"^" '{ print $1 }'`
   end=`echo $line | awk -F"^" '{ print $2 }'`
   sed -n "${start},${end}p" ${file_to_parse} > tmpfile.01
   line_time=`sed -n "7p" tmpfile.01 | sed 's/^ *//;s/ *$//;s/  */ /;'`
   line_client=`sed -n "17p" tmpfile.01 | sed 's/^ *//;s/ *$//;s/  */ /;'`
   line_warning=`grep "^WARNING: inbound connection timed out (ORA-3136)" tmpfile.01`

   if [[ -z "${line_warning}" ]] ; then
      line_warning="NO MATCHING WARNING of ORA-3136"
   fi

   line_detail="${line_time}^${line_client}^${line_warning}"
   echo $line_detail | tee -a tmpfile.02
done < tmpfile.00

So basically I pass a log file and need to search for a block of text.

Code:
Fatal NI connect error 12170.

  VERSION INFORMATION:
        TNS for Linux: Version 11.2.0.4.0 - Production
        Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
  Time: 18-OCT-2019 04:33:41
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12535

TNS-12535: TNS:operation timed out
    ns secondary err code: 12606
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0
  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=12.123.12.12)(PORT=49931))
WARNING: inbound connection timed out (ORA-3136)

As you can see from the script, I search for the string "Fatal NI connect error 12170." and add 17 to it, that will be the block of text that am after. Then I print this block of text to a file and grep the 3 basic information that I am after which is Time, Client address and the WARNING. The WARNING line sometimes exist and sometimes it doesn't.

So far, running the script does most of what I wanted.

Sample output below:

Code:
Time: 19-OCT-2019 11:00:18^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.46)(PORT=60771))^WARNING: inbound connection timed out (ORA-3136)
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production^Tracing not turned on.^NO MATCHING WARNING of ORA-3136
^Tns error struct:^NO MATCHING WARNING of ORA-3136
Time: 19-OCT-2019 11:08:47^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.37)(PORT=53555))^WARNING: inbound connection timed out (ORA-3136)
Time: 19-OCT-2019 12:22:21^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.36)(PORT=61857))^WARNING: inbound connection timed out (ORA-3136)
Time: 19-OCT-2019 13:51:51^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.36)(PORT=62520))^WARNING: inbound connection timed out (ORA-3136)
Time: 19-OCT-2019 15:27:38^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.46)(PORT=62541))^WARNING: inbound connection timed out (ORA-3136)
Time: 19-OCT-2019 15:59:01^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.45)(PORT=62200))^NO MATCHING WARNING of ORA-3136
Time: 19-OCT-2019 16:00:57^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.37)(PORT=55824))^NO MATCHING WARNING of ORA-3136
Time: 19-OCT-2019 16:00:57^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.37)(PORT=55828))^WARNING: inbound connection timed out (ORA-3136)
Time: 19-OCT-2019 16:02:33^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.37)(PORT=55995))^WARNING: inbound connection timed out (ORA-3136)
Time: 19-OCT-2019 16:33:00^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.45)(PORT=62409))^WARNING: inbound connection timed out (ORA-3136)
Time: 19-OCT-2019 17:29:40^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.46)(PORT=63168))^NO MATCHING WARNING of ORA-3136
Time: 19-OCT-2019 17:29:42^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.46)(PORT=63176))^NO MATCHING WARNING of ORA-3136
Time: 19-OCT-2019 17:53:52^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.45)(PORT=62812))^WARNING: inbound connection timed out (ORA-3136)
Time: 19-OCT-2019 17:55:33^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.46)(PORT=63299))^WARNING: inbound connection timed out (ORA-3136)
Time: 19-OCT-2019 18:40:15^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.45)(PORT=63152))^WARNING: inbound connection timed out (ORA-3136)
Time: 19-OCT-2019 19:09:19^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.37)(PORT=57065))^WARNING: inbound connection timed out (ORA-3136)
Time: 19-OCT-2019 19:31:03^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.37)(PORT=57160))^WARNING: inbound connection timed out (ORA-3136)
Time: 19-OCT-2019 19:39:20^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.37)(PORT=57230))^NO MATCHING WARNING of ORA-3136
Time: 19-OCT-2019 19:39:20^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.37)(PORT=57234))^WARNING: inbound connection timed out (ORA-3136)
VERSION INFORMATION:^ TNS for Linux: Version 11.2.0.4.0 - Production^NO MATCHING WARNING of ORA-3136
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production^ns main err code: 12535^NO MATCHING WARNING of ORA-3136
Time: 19-OCT-2019 20:27:18^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.36)(PORT=64251))^NO MATCHING WARNING of ORA-3136
Time: 19-OCT-2019 20:34:50^Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.123.99.37)(PORT=57593))^NO MATCHING WARNING of ORA-3136

Sometimes there are some 'malform' of some sort, the ones in RED above. And this is because sometimes the log contain something like below which I am not expecting it to have Smilie

Code:
Fatal NI connect error 12170.

Fatal NI connect error 12170.

  VERSION INFORMATION:
        TNS for Linux: Version 11.2.0.4.0 - Production
        Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production

  VERSION INFORMATION:
        TNS for Linux: Version 11.2.0.4.0 - Production
        Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
  Time: 19-OCT-2019 11:06:04
  Time: 19-OCT-2019 11:06:04
  Tracing not turned on.
  Tracing not turned on.
  Tns error struct:
  Tns error struct:
    ns main err code: 12535
    ns main err code: 12535


TNS-12535: TNS:operation timed out
TNS-12535: TNS:operation timed out
    ns secondary err code: 12606
    ns secondary err code: 12606
    nt main err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt secondary err code: 0
    nt OS err code: 0
    nt OS err code: 0
  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.111.11.37)(PORT=53542))
  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.111.11.37)(PORT=53538))
WARNING: inbound connection timed out (ORA-3136)
Sat Oct 19 11:08:47 2019

Or something like below:

Code:
Fatal NI connect error 12170.
Sat Oct 19 20:05:49 2019


***********************************************************************

  VERSION INFORMATION:
        TNS for Linux: Version 11.2.0.4.0 - Production
        Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production

Fatal NI connect error 12170.
  Time: 19-OCT-2019 20:05:49
  Tracing not turned on.

  VERSION INFORMATION:
        TNS for Linux: Version 11.2.0.4.0 - Production
        Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
  Tns error struct:
  Time: 19-OCT-2019 20:05:49
    ns main err code: 12535

  Tracing not turned on.
TNS-12535: TNS:operation timed out
  Tns error struct:
    ns secondary err code: 12606
    ns main err code: 12535
    nt main err code: 0

    nt secondary err code: 0
TNS-12535: TNS:operation timed out
    nt OS err code: 0
    ns secondary err code: 12606
  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.111.22.36)(PORT=64132))
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0
  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.111.22.36)(PORT=64136))
WARNING: inbound connection timed out (ORA-3136)
Sat Oct 19 20:10:26 2019

As a workaround to this malform, I am changing the script to check if the 7th line is not "^Time" then I am excluding this block of text. I can't find any way of matching a block of text starting from " Fatal NI connect error 12170" to "WARNING: inbound connection timed out (ORA-3136)". Unfortunately, sometimes it does not end with "WARNING: inbound connection timed out (ORA-3136)", instead the last line is "Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=11.111.22.36)(PORT=64136))" and HOST and PORT changes.

Below is the script with the 'check' for the malform. The lines in RED.

Code:
echo
echo "- Parsing $1 ..."
echo
file_to_parse=$1

cat /dev/null > tmpfile.00
cat /dev/null > tmpfile.01
cat /dev/null > tmpfile.02

grep -in "Fatal NI connect error 12170" ${file_to_parse} | awk -F":" '{ print $1"^"$1+17 }' > tmpfile.00

while read line
do
   start=`echo $line | awk -F"^" '{ print $1 }'`
   end=`echo $line | awk -F"^" '{ print $2 }'`
   sed -n "${start},${end}p" ${file_to_parse} > tmpfile.01
   line_time=`sed -n "7p" tmpfile.01 | sed 's/^ *//;s/ *$//;s/  */ /;'`

   check_line_time=`echo ${line_time} | grep "^Time"`
   if [[ -z "${check_line_time}" ]] ; then      # It's a malform then ???
      continue
   fi

   line_client=`sed -n "17p" tmpfile.01 | sed 's/^ *//;s/ *$//;s/  */ /;'`
   line_warning=`grep "^WARNING: inbound connection timed out (ORA-3136)" tmpfile.01`

   if [[ -z "${line_warning}" ]] ; then
      line_warning="NO MATCHING WARNING of ORA-3136"
   fi

   line_detail="${line_time}^${line_client}^${line_warning}"
   echo $line_detail | tee -a tmpfile.02
done < tmpfile.00

Anyway, please advise if there is any better way of doing this the way it is now.
Last edited by newbie_01; 10-22-2019 at 10:14 AM.. Reason: Updated script
This User Gave Thanks to newbie_01 For This Post:
Neo
# 2  
Old 10-22-2019
Your script seems a bit intricate. E.g. when using the > redirection, you don't need to create / truncate the file upfront. And, with that many temp files, there must be a better approach. How far would
Code:
sed -n '/Fatal NI connect error 12170./,/WARNING: inbound connection timed out (ORA-3136)/p;' file

get you for the first part of your task- in the good case?


Now, if the "WARNING" line is missing sometimes, but the "client address" line is always there, why not use that and add a condition to print the "WARNING" line individually? Is the "WARNING" line always immediately following the "Client" line?


And. looks like your logger sometimes misbehaves by mixing logs of two independent events. If you can't remedy that in the originator, you'll need additional coding on the receiving side, and that can't be done in sed, but needs tools like awk, perl, or similar. Are the log lines in relative order, i.e. the second log line consistently belongs to the second event?
Last edited by RudiC; 10-22-2019 at 10:55 AM..
These 2 Users Gave Thanks to RudiC For This Post:
Neo newbie_01
# 3  
Old 10-26-2019
Quote:
Originally Posted by RudiC
Your script seems a bit intricate. E.g. when using the > redirection, you don't need to create / truncate the file upfront. And, with that many temp files, there must be a better approach. How far would
Code:
sed -n '/Fatal NI connect error 12170./,/WARNING: inbound connection timed out (ORA-3136)/p;' file

get you for the first part of your task- in the good case?


Now, if the "WARNING" line is missing sometimes, but the "client address" line is always there, why not use that and add a condition to print the "WARNING" line individually? Is the "WARNING" line always immediately following the "Client" line?


And. looks like your logger sometimes misbehaves by mixing logs of two independent events. If you can't remedy that in the originator, you'll need additional coding on the receiving side, and that can't be done in sed, but needs tools like awk, perl, or similar. Are the log lines in relative order, i.e. the second log line consistently belongs to the second event?
Hi RudiC

Thanks for you reply. Very helpful as usual. Yeah I shouldn't need to pre-create the tmpfiles, just being paranoid I guess, I've since change it to use $$ as well as I need to run the script with several log files.

The sed syntax
Code:
sed -n '/Fatal NI connect error 12170./,/WARNING: inbound connection timed out (ORA-3136)/p;' file

sure speed things up and it indeed gets most of the pattern match that I am after.

Yes, the Client line is always there. The WARNING line is there most of the time, I am mainly wanting to include to check if the error that comes with the WARNING is always ORA-3136 or not. The HOST and PORT is always going to be different though in most cases. I tried using
Code:
sed -n '/Fatal NI connect error 12170./,/Client/p;' file

and that works fine so I'll stick to doing that.

How do I get it to print the Time, Client and WARNING only though? At the moment, I am running the same command 2x/3x, depending on whether I want to include WARNING, and then grepping for Time, Client and WARNING and then combining the 2/3 files output.

When you say
Quote:
And. looks like your logger sometimes misbehaves by mixing logs of two independent events
, are you referring to when I am doing the if/then/else check of where I expect to see the Time, Client or WARNING line to appear? If so, that is because sometimes there is some malform in the log, where it gets too much of these errors happening an the way it get printed in the log file is not in the order that I am expecting it to look like Smilie
This User Gave Thanks to newbie_01 For This Post:
Neo
# 4  
Old 10-27-2019
Try
Code:
awk '
/^ *Fatal/,/^ *Client/  {if (/Time/) printf "^%s", $0
                         if (/Client/)  {printf "^%s", $0
                                         getline
                                         if (/WARNING/) printf "^%s\n", $0
                                         else printf "^No Warning encountered\n"
                                        }
                        }
' file

Doesn't help if the log is messed up with e.g. two events being logged one on top of the other.
This User Gave Thanks to RudiC For This Post:
Neo
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Taking out part of a string by matching a pattern

Hi All, My Problem is like below. I have a file which contains just one row and contains data like PO_CREATE12457888888888889SK1234567878744551111111111SK89456321145789955455555SK8888888815788852222 i want to extract SK12345678 SK89456321 SK88888888 So basically SK and next 8... (4 Replies)
Discussion started by: Asfakul Islam
4 Replies

2. Shell Programming and Scripting

Pattern Matching in Perl script

I have a big perl script need to fix a small pattern matching inside .. I have patterns like create unique index create index The pattern matching should look for both the pattern in the same statement, The existing matching looks for only "create unique index" The exising code for this i... (6 Replies)
Discussion started by: greenworld123
6 Replies

3. Shell Programming and Scripting

Expect Script - Pattern Matching Trouble

I am still learning expect, For the below script I always get This is True as the answer. Tried to debug and does not make sense to me. Not sure, where I am doing the mistake. Need Help! - Thanks #!/usr/bin/expect -f set timeout 10 spawn -noecho bash expect { *$* } send "test -d... (3 Replies)
Discussion started by: rmsagar
3 Replies

4. UNIX for Dummies Questions & Answers

Script to list non matching files using pattern

Hi, I am trying to write a script that list down all the files that do not match the pattern My pattern will be like this "*.jpg|*.xml|*.sql". This pattern will be stored in a file. The script need to read this pattern from the file and delete the files that does not match this pattern. It... (7 Replies)
Discussion started by: meenavin
7 Replies

5. Shell Programming and Scripting

Pattern matching in shell script

Hi, I am using following command to extract string from a file. String will be after last / (slash). awk -F\ / '{print $NF}' $FILE but while appending the output in file in script, it dosent work. File created but of zero size... can anyone please help `awk -F\\\/ '{print $NF}' $FILE` >... (3 Replies)
Discussion started by: Deei
3 Replies

6. Shell Programming and Scripting

shell script pattern matching

Hi, I need to create a shell script through which i need to populate email addresses in email columns of database table in mysql. Let say if email contains yahoo, hotmail, gtalk than email addresses need to move in their respective columns. # !/bin/sh yim="example@yahoo.com"... (3 Replies)
Discussion started by: mirfan
3 Replies

7. Shell Programming and Scripting

Problem extracting just a part of a matching pattern

Hello everyone, this is my first post so please give me a hand. I apologize for my English, I'll try to be clear with my request. I need to write a script (Bash) which finds all the variables defined in the file .h of the folder and then writes the name of the files .c where these variables are... (1 Reply)
Discussion started by: paxilpaz
1 Replies

8. Shell Programming and Scripting

Pattern matching in a shell script?

I'm looking for a way to match a particular string in another string and if a match is found execute some command. I found the case statement can be used like this; case word in ) command ;; ] ... esac If my string to find is say "foo" in the string $mystring... (1 Reply)
Discussion started by: paulobrad
1 Replies

9. Shell Programming and Scripting

bash script, pattern matching + sum

Hi guys, i have been struggling to achieve the following and therefor looking for some help. I am trying to write something that will summerize the following output from my switches for daily totals. Basicly if $1 $2 $3 doesnt change, we can keep adding up $4. Probably would use a awk print end... (3 Replies)
Discussion started by: Wratholix
3 Replies

10. UNIX for Dummies Questions & Answers

vi part-pattern matching and deletion

Hi, I have a log file which shows the files which has been changed over the last week. They follow this pattern: old_file_version_number@@new_file_version_number Now I need to know how to delete from each line parts starting from @@. I would be issuing the command inside vi(m). So... (5 Replies)
Discussion started by: vino
5 Replies
Login or Register to Ask a Question