Parsing and timestamp a pattern in log | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Parsing and timestamp a pattern in log

Shell Programming and Scripting


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 03-26-2013
amazigh42 amazigh42 is offline
Registered User
 
Join Date: Feb 2013
Last Activity: 29 October 2014, 10:58 AM EDT
Posts: 77
Thanks: 11
Thanked 0 Times in 0 Posts
Parsing and timestamp a pattern in log

Hello

Thanks to Chubler_XL and MadeInGermany for their help few weeks ago.
Now, i would like modifying the script, see the next POST.

The old script works like that :
I picked any random hours
In the logs there is the stamp time of webservices, i can see the behavior or errors of webservices. Just put the approximate times in order to have a portion of the log. This could also handle errors when the script does not find the right hour. The script greps a bit of script between timestamp.


Code:
cat log_name


Code:
aaaaaaaaaaaaaa
bbbbbbbbbbbbb
cccccccccccc
[24/01/2013 09:10]
sssssssssssssss
error-jonas123
nnnnnnnnnnnnn
[24/01/2013 10:10]
uuuuuuuuuuuuuuu
jjjjjjjjjjjjjj
error-jonas123
mmmmmmmmmmmmm
[24/01/2013 10:30]
oooooooooooo
error-jonas123
qqqqqqqqqqq
[24/01/2013 10:45]
vvvvvvvvv
sssssssss
wwwwwwwwww

The result

Code:
./my script log_name
[24/01/2013 10:10]
uuuuuuuuuuuuuuu
jjjjjjjjjjjjjj
error-jonas123
mmmmmmmmmmmmm
[24/01/2013 10:30]

Of course, it would be desirable to put the dates in variables.

Code:
vi my_script


Code:
#!/bin/bash
log_name=$1
if [[ "$log_name" =~ .gz$ ]]
     then z_cat="gunzip -c"
     else z_cat=cat
fi
$z_cat $log_name |awk -F"[/ \\\][]" -v S="24/01/2013 10:10" -v E="24/01/2013 10:30" '
function dcmp(b) {
  if($4>b[3])return  1;
  if($4<b[3])return -1;
  if($3>b[2])return  1;
  if($3<b[2])return -1;
  if($2>b[1])return  1;
  if($2<b[1])return -1;
  if($5>b[4])return  1;
  if($5<b[4])return -1;
  return 0;
}
BEGIN{split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/^[[][0-9][0-9]\/[0-1][0-9]\/[[0-9][0-9][0-9][0-9] / {
   if(s&&dcmp(de)>=0) {print; exit}
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s'

---------- Post updated at 08:39 AM ---------- Previous update was at 08:32 AM ----------

Now, i would like modifying the script like this :
The script has several identical patterns like for example error-jonas123.
The script will have to pick the first pattern, then it will have to search the nearest date before. Then it will have to search the last pattern and it will have to search the nearest date after.


Code:
cat log_name


Code:
aaaaaaaaaaaaaa
bbbbbbbbbbbbb
cccccccccccc
[24/01/2013 09:10]
sssssssssssssss
error-jonas123
nnnnnnnnnnnnn
[24/01/2013 10:10]
uuuuuuuuuuuuuuu
jjjjjjjjjjjjjj
error-jonas123
mmmmmmmmmmmmm
[24/01/2013 10:30]
oooooooooooo
error-jonas123
qqqqqqqqqqq
[24/01/2013 10:45]
vvvvvvvvv
sssssssss
wwwwwwwwww


Code:
my_script log_name

The expected result

Code:
[24/01/2013 09:10]
sssssssssssssss
error-jonas123
nnnnnnnnnnnnn
[24/01/2013 10:10]
uuuuuuuuuuuuuuu
jjjjjjjjjjjjjj
error-jonas123
mmmmmmmmmmmmm
[24/01/2013 10:30]
oooooooooooo
error-jonas123
qqqqqqqqqqq
[24/01/2013 10:45]

Can you give me somes ideas to change the script.
Sponsored Links
    #2  
Old 03-26-2013
DGPickett DGPickett is offline Forum Advisor  
Registered User
 
Join Date: Oct 2010
Last Activity: 18 December 2014, 4:59 PM EST
Location: Southern NJ, USA (Nord)
Posts: 4,560
Thanks: 8
Thanked 562 Times in 537 Posts
It gets tricky when there are adjacent errors, unless you report the time in the middle twice, one at the end of the first and once at the beginning of the second. You could write a pretty simple sed script to pull all the lines from timestamp N to timestamp N+1 into the buffer, check for error and write to output or side file if any, get rid of all but the last line, and loop back to filling the buffer up to the next time stamp.

It would be easy to make the buffer load into one line before writing it out, so they can be handled more simply after.
Sponsored Links
    #3  
Old 04-02-2013
amazigh42 amazigh42 is offline
Registered User
 
Join Date: Feb 2013
Last Activity: 29 October 2014, 10:58 AM EDT
Posts: 77
Thanks: 11
Thanked 0 Times in 0 Posts
Hello,
I thought I understood the magenta pattern but not.

I have understood this line with the commande echo


Code:
$z_cat $log_name |awk -F"[/ \\\][]" -v S="24/01/2013 10:10" -v E="24/01/2013 10:30"


Code:
echo "[24/01/2013 10:10 10:51]" | awk -F"[/ \\\][]" '{ print FS ; print $2; print $3; print $4; print $5; }'
[/ \][]
24
01
2013
10:10


Code:
#!/bin/bash
log_name=$1
if [[ "$log_name" =~ .gz$ ]]
     then z_cat="gunzip -c"
     else z_cat=cat
fi
$z_cat $log_name |awk -F"[/ \\\][]" -v S="24/01/2013 10:10" -v E="24/01/2013 10:30" '
function dcmp(b) {
  if($4>b[3])return  1;
  if($4<b[3])return -1;
  if($3>b[2])return  1;
  if($3<b[2])return -1;
  if($2>b[1])return  1;
  if($2<b[1])return -1;
  if($5>b[4])return  1;
  if($5<b[4])return -1;
  return 0;
}
BEGIN{split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/^[[][0-9][0-9]\/[0-1][0-9]\/[[0-9][0-9][0-9][0-9] / {
   if(s&&dcmp(de)>=0) {print; exit}
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s'

What do these lines match ?


Code:
b[1]
b[2]
b[3]
b[4]
b[5]

Thanks in advance.
    #4  
Old 04-04-2013
amazigh42 amazigh42 is offline
Registered User
 
Join Date: Feb 2013
Last Activity: 29 October 2014, 10:58 AM EDT
Posts: 77
Thanks: 11
Thanked 0 Times in 0 Posts
Hello,

I would like to have confirmation.


Code:
$z_cat $log_name |awk -F"[/ \\\][]" -v S="24/01/2013 10:10" -v E="24/01/2013 10:30" '
function dcmp(b) {
  if($4>b[3])return  1;

In $log_name I have lines which look like to :

Code:
[24/01/2013 09:11:59,236] ERROR [pool-1-thread-3][org.objectweb.jonas.jca.process]

1- Does the red pattern b[3] match with the red pattern 2013 ?
2- And Does the magenta pattern $4 match with the magenta pattern 2013 ?
3- Else, how to debug b[3] with printf ?

Any help will be greatly appreciated because I can not move anymore.

Last edited by amazigh42; 04-04-2013 at 09:35 AM..
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
pattern parsing BeefStu Shell Programming and Scripting 4 11-09-2010 04:04 PM
Parsing a timestamp until EOF terrell Shell Programming and Scripting 6 09-22-2010 09:30 AM
Delete log file entries based on the Date/Timestamp within log file vikram3.r Shell Programming and Scripting 3 08-05-2010 05:26 AM
List all log records logged after $timestamp ? Browser_ice Shell Programming and Scripting 5 06-16-2009 06:41 AM
Spooling a log file with timestamp ukadmin UNIX for Dummies Questions & Answers 2 03-23-2004 09:06 AM



All times are GMT -4. The time now is 06:14 PM.