The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Top Forums > UNIX for Advanced & Expert Users
Extracting the required text from log files Extracting the required text from log files
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read


UNIX for Advanced & Expert Users Expert-to-Expert. Learn advanced UNIX, UNIX commands, Linux, Operating Systems, System Administration, Programming, Shell, Shell Scripts, Solaris, Linux, HP-UX, AIX, OS X, BSD.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Extracting Text krabu Shell Programming and Scripting 3 04-07-2009 03:16 AM
extracting text and reusing the text to rename file JohnDS UNIX for Dummies Questions & Answers 7 02-05-2009 03:55 AM
Extracting information from Config files /text processing oconmx Shell Programming and Scripting 3 01-21-2009 07:09 PM
help required for replacing text in vi Chandu2u Shell Programming and Scripting 6 01-26-2008 11:12 AM
Help required regarding Extracting lines from a file google_ever Shell Programming and Scripting 1 10-12-2005 07:02 AM

Reply
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Bulgarian Greek Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rating: Thread Rating: 1 votes, 4.00 average. Display Modes
  #1 (permalink)  
Old 06-20-2009
hareeshram hareeshram is offline
Registered User
  
 

Join Date: Mar 2007
Location: Chennai
Posts: 3
Extracting the required text from log files
It would be highly appreciable if any one helps me in this. I am trying to get it done through Java but I love unix and believe it can be done within minutes with couple of lines.

The input log file is a text file contains multiple entries seperated by a blank line.
Each seperated entry corresponds to upgrade process information of one file.

!ENTRY text.....<INFO> or <OKAY> <RESOURCE: /test/src/com/test1/*/test.java> 2009-06-18 13:01:01.181
!MESSAGE Requesting upgrade report for file: test.java

!ENTRY text.....<INFO> or <OKAY> <RESOURCE: /test/src/com/test1/*/test1.java> 2009-06-18 13:01:01.181
!MESSAGE information in test1.java will be upgraded.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: full path /file name> 2009-06-18 13:02:25.681
!MESSAGE Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.

Each Entry starts with "!ENTRY" as shown above and will be followed by text "com.bea.workshop.upgrade81 " and then it will be followed by
two types of tags: <OKAY> and <INFO>
And then it will be followed by tag <RESOURCE:which contains the full path of the file and then followed by time stamp as shown above.

If it is <OKAY>, then second line will be as below
!MESSAGE Requesting upgrade report for file: filename ( Which is not of much importance for my output)

Ex:
!ENTRY text.....<INFO> or <OKAY> <RESOURCE: /test/src/com/test1/*/test.java> 2009-06-18 13:01:01.181
!MESSAGE Requesting upgrade report for file: file name

If it is <INFO> then also second line will be like
!MESSAGE Requesting upgrade report for file: filename
But it will certainly be followed by another set of multiple line pairs starting with tags !SUBENTRY,!MESSAGE like below

!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: path/filename> 2009-06-18 13:02:25.681
!MESSAGE Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.


My requirement is like below:

Requirement 1

1) For all entries containing <OKAY> tag, I need to extract the file names which will be after <RESOURCE:.....and before time stamp
Please note the entries are seperated by a blank line

Requirement 2

2) For all the lines with <INFO> tag, I would like to have a text file with entries like this

Full path and the file name and then in the next line
All the text after !MESAAGE right below the line containg the tag of corresponding "!SUBENTRY 1" tags


Example Input:
==============================
!ENTRY com.bea.workshop.upgrade81 <OKAY> <RESOURCE: /fullpathr/Test.java> 2009-06-18 13:02:28.368
!MESSAGE Requesting upgrade report for file: Test.java

!ENTRY com.bea.workshop.upgrade81 <OKAY> <RESOURCE: /fullpath/Test1.jpf> 2009-06-18 13:02:28.384
!MESSAGE Requesting upgrade report for file: Test1.jpf

!ENTRY com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447
!MESSAGE Requesting upgrade report for file: Test2.jpf
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447
!MESSAGE The Java 5 annotation Jpf.Controller needs to be added.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447
!MESSAGE More annotation of Jpf.Action needs to be added.

!ENTRY com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE Requesting upgrade report for file: Test3.jpf
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE The Java 5 annotation Jpf.Controller needs to be added.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE ABC needs to be added.


Output for requirement1:
================================
Test.java
Test1.jpf

Output for requirement2:
================================
/fullpath/Test2.jpf
The Java 5 annotation Jpf.Controller needs to be added.
More annotation of Jpf.Action needs to be added.

/fullpath/Test3.jpf
The Java 5 annotation Jpf.Controller needs to be added.
Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.
ABC needs to be added.
Last edited by hareeshram; 06-20-2009 at 09:09 AM..
  #2 (permalink)  
Old 06-20-2009
scottn scottn is offline Forum Advisor  
VIP Member
  
 

Join Date: Jun 2009
Location: Zürich, CH
Posts: 1,053
This works, but probably only if your input is exactly as you described!

Code:
echo "Output from requirement 1"
echo "========================="
grep "<OKAY>.*RESOURCE" infile | sed -e "s/.*RESOURCE:.*\/\(.*\)>.*/\1/" | sort -u

echo
echo "Output from requirement 2"
echo "========================="
awk '
  /^!ENTRY.*<INFO>/ { X = 1; sub( /.*RESOURCE: /, "", $0 ); sub( />.*/, "", $0 ); print }
  (X == 1) && (/^!SUBENTRY/) { X++ }
  (X > 1) && ($1 ~ /^!MESSAGE/) { sub( /!MESSAGE /, "", $0 ); print }
  (X > 1) && ($1 ~ /^$/) { print ""; X = 0 }
' infile
 
 
Output:
Output from requirement 1
=========================
Test1.jpf
Test.java
 
Output from requirement 2
=========================
/fullpath/Test2.jpf
The Java 5 annotation Jpf.Controller needs to be added.
More annotation of Jpf.Action needs to be added.
 
/fullpath/Test3.jpf
The Java 5 annotation Jpf.Controller needs to be added.
Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.
ABC needs to be added.
Last edited by scottn; 06-20-2009 at 10:44 AM..
  #3 (permalink)  
Old 06-20-2009
radoulov's Avatar
radoulov radoulov is offline Forum Staff  
addict
  
 

Join Date: Jan 2007
Location: Варна, България / Milano, Italia
Posts: 2,860
Use gawk, nawk or /usr/xpg4/bin/awk on Solaris:

1.

Code:
awk '/<OKAY>/ { 
  sub(/>[^>]*$/, "")
  n = split($0, t, "/")
  print t[n]   
  }' infile
2.

Code:
awk '!NF { f = 0 }
/^!ENTRY.*<INFO>/ {
  sub(/>[^>]*$/, "")
  sub(/.*RESOURCE: /, "")
  print; f = 1
  }  
f && /!SUBENTRY/ { f++ }   
f > 1 && sub(/!MESSAGE /, "")
' infile
Last edited by radoulov; 06-20-2009 at 10:48 AM..
Bits Awarded / Charged to radoulov for this Post
Date User Comment Amount
06-22-2009 hareeshram Thanks for the smart solution! 200
  #4 (permalink)  
Old 06-20-2009
scottn scottn is offline Forum Advisor  
VIP Member
  
 

Join Date: Jun 2009
Location: Zürich, CH
Posts: 1,053
Cool!

You win
  #5 (permalink)  
Old 06-20-2009
radoulov's Avatar
radoulov radoulov is offline Forum Staff  
addict
  
 

Join Date: Jan 2007
Location: Варна, България / Milano, Italia
Posts: 2,860
Quote:
Originally Posted by scottn View Post
Cool!

You win
No,
I need to make the code more generic (I just modified it to remove the specific columns references).
Last edited by radoulov; 06-20-2009 at 11:08 AM.. Reason: wrong statement :), your output is correct
  #6 (permalink)  
Old 06-22-2009
hareeshram hareeshram is offline
Registered User
  
 

Join Date: Mar 2007
Location: Chennai
Posts: 3
That was so fast and accurate!
Thanks scottn and radoulov.
It worked fine for me

I have a slight change in format that I would expect for both requirements. Hope you would suggest me.

Requirement 1:

The file names should come with full paths

Requirement 2:
After the file name (with full path), whatever the messages that are being displayed (one message for one line), they are to be displayed without any duplication ( the same message in consecutive lines should be removed) per entry and all non duplicated entries should be separted by comma (rather than new line"). Unlike in the above case, both file and comma separated messages should come in the single line.

Once again many many thanks for keeping my spirit up in unix.

Example input
=========

Example Input:
==============================
!ENTRY com.bea.workshop.upgrade81 <OKAY> <RESOURCE: /fullpath/Test.java> 2009-06-18 13:02:28.368
!MESSAGE Requesting upgrade report for file: Test.java

!ENTRY com.bea.workshop.upgrade81 <OKAY> <RESOURCE: /fullpath/Test1.jpf> 2009-06-18 13:02:28.384
!MESSAGE Requesting upgrade report for file: Test1.jpf

!ENTRY com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447
!MESSAGE Requesting upgrade report for file: Test2.jpf
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447
!MESSAGE The Java 5 annotation Jpf.Controller needs to be added.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447
!MESSAGE More annotation of Jpf.Action needs to be added.

!ENTRY com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE Requesting upgrade report for file: Test3.jpf
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE The Java 5 annotation Jpf.Controller needs to be added.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE The Java 5 annotation Jpf.Controller needs to be added.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE ABC needs to be added.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE ABC needs to be added.


Output for requirement1:
================================
/fullpath/Test.java
/fullpath/Test1.jpf

Output for requirement2:
================================
/fullpath/Test2.jpf (separated by tab) The Java 5 annotation Jpf.Controller needs to be added,More annotation of Jpf.Action needs to be added.
/fullpath/Test3.jpf (separated by tab) The Java 5 annotation Jpf.Controller needs to be added,Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support,ABC needs to be added.
Reply

Bookmarks

Tags
log extract

« Samba: Install w/o Root | mutt »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Hybrid Mode Hybrid Mode
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 11:47 PM.

The UNIX and Linux Forums - Learn UNIX and UNIX Commands RSS Feeds - Contact Us - The UNIX and Linux Forums - Learn UNIX and UNIX Commands - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0