UNIX for Advanced & Expert Users

It would be highly appreciable if any one helps me in this. I am trying to get it done through Java but I love unix and believe it can be done within minutes with couple of lines.

The input log file is a text file contains multiple entries seperated by a blank line.
Each seperated entry corresponds to upgrade process information of one file.

!ENTRY text.....<INFO> or <OKAY> <RESOURCE: /test/src/com/test1/*/test.java> 2009-06-18 13:01:01.181
!MESSAGE Requesting upgrade report for file: test.java

!ENTRY text.....<INFO> or <OKAY> <RESOURCE: /test/src/com/test1/*/test1.java> 2009-06-18 13:01:01.181
!MESSAGE information in test1.java will be upgraded.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: full path /file name> 2009-06-18 13:02:25.681
!MESSAGE Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.

Each Entry starts with "!ENTRY" as shown above and will be followed by text "com.bea.workshop.upgrade81 " and then it will be followed by
two types of tags: <OKAY> and <INFO>
And then it will be followed by tag <RESOURCE:which contains the full path of the file and then followed by time stamp as shown above.

If it is <OKAY>, then second line will be as below
!MESSAGE Requesting upgrade report for file: filename ( Which is not of much importance for my output)

Ex:
!ENTRY text.....<INFO> or <OKAY> <RESOURCE: /test/src/com/test1/*/test.java> 2009-06-18 13:01:01.181
!MESSAGE Requesting upgrade report for file: file name

If it is <INFO> then also second line will be like
!MESSAGE Requesting upgrade report for file: filename
But it will certainly be followed by another set of multiple line pairs starting with tags !SUBENTRY,!MESSAGE like below

!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: path/filename> 2009-06-18 13:02:25.681
!MESSAGE Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.


My requirement is like below:

Requirement 1

1) For all entries containing <OKAY> tag, I need to extract the file names which will be after <RESOURCE:.....and before time stamp
Please note the entries are seperated by a blank line

Requirement 2

2) For all the lines with <INFO> tag, I would like to have a text file with entries like this

Full path and the file name and then in the next line
All the text after !MESAAGE right below the line containg the tag of corresponding "!SUBENTRY 1" tags


Example Input:
==============================
!ENTRY com.bea.workshop.upgrade81 <OKAY> <RESOURCE: /fullpathr/Test.java> 2009-06-18 13:02:28.368
!MESSAGE Requesting upgrade report for file: Test.java

!ENTRY com.bea.workshop.upgrade81 <OKAY> <RESOURCE: /fullpath/Test1.jpf> 2009-06-18 13:02:28.384
!MESSAGE Requesting upgrade report for file: Test1.jpf

!ENTRY com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447
!MESSAGE Requesting upgrade report for file: Test2.jpf
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447
!MESSAGE The Java 5 annotation Jpf.Controller needs to be added.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447
!MESSAGE More annotation of Jpf.Action needs to be added.

!ENTRY com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE Requesting upgrade report for file: Test3.jpf
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE The Java 5 annotation Jpf.Controller needs to be added.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE ABC needs to be added.


Output for requirement1:
================================
Test.java
Test1.jpf

Output for requirement2:
================================
/fullpath/Test2.jpf
The Java 5 annotation Jpf.Controller needs to be added.
More annotation of Jpf.Action needs to be added.

/fullpath/Test3.jpf
The Java 5 annotation Jpf.Controller needs to be added.
Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.
ABC needs to be added.

This works, but probably only if your input is exactly as you described!

Code:

echo "Output from requirement 1"
echo "========================="
grep "<OKAY>.*RESOURCE" infile | sed -e "s/.*RESOURCE:.*\/\(.*\)>.*/\1/" | sort -u

echo
echo "Output from requirement 2"
echo "========================="
awk '
  /^!ENTRY.*<INFO>/ { X = 1; sub( /.*RESOURCE: /, "", $0 ); sub( />.*/, "", $0 ); print }
  (X == 1) && (/^!SUBENTRY/) { X++ }
  (X > 1) && ($1 ~ /^!MESSAGE/) { sub( /!MESSAGE /, "", $0 ); print }
  (X > 1) && ($1 ~ /^$/) { print ""; X = 0 }
' infile
 
 
Output:
Output from requirement 1
=========================
Test1.jpf
Test.java
 
Output from requirement 2
=========================
/fullpath/Test2.jpf
The Java 5 annotation Jpf.Controller needs to be added.
More annotation of Jpf.Action needs to be added.
 
/fullpath/Test3.jpf
The Java 5 annotation Jpf.Controller needs to be added.
Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.
ABC needs to be added.

Use gawk, nawk or /usr/xpg4/bin/awk on Solaris:

1.

Code:

awk '/<OKAY>/ { 
  sub(/>[^>]*$/, "")
  n = split($0, t, "/")
  print t[n]   
  }' infile

2.

Code:

awk '!NF { f = 0 }
/^!ENTRY.*<INFO>/ {
  sub(/>[^>]*$/, "")
  sub(/.*RESOURCE: /, "")
  print; f = 1
  }  
f && /!SUBENTRY/ { f++ }   
f > 1 && sub(/!MESSAGE /, "")
' infile

Cool!

You win

No,
I need to make the code more generic (I just modified it to remove the specific columns references).

Thanks scottn and radoulov.
It worked fine for me

I have a slight change in format that I would expect for both requirements. Hope you would suggest me.

Requirement 1:

The file names should come with full paths

Requirement 2:
After the file name (with full path), whatever the messages that are being displayed (one message for one line), they are to be displayed without any duplication ( the same message in consecutive lines should be removed) per entry and all non duplicated entries should be separted by comma (rather than new line"). Unlike in the above case, both file and comma separated messages should come in the single line.

Once again many many thanks for keeping my spirit up in unix.

Example input
=========

Example Input:
==============================
!ENTRY com.bea.workshop.upgrade81 <OKAY> <RESOURCE: /fullpath/Test.java> 2009-06-18 13:02:28.368
!MESSAGE Requesting upgrade report for file: Test.java

!ENTRY com.bea.workshop.upgrade81 <OKAY> <RESOURCE: /fullpath/Test1.jpf> 2009-06-18 13:02:28.384
!MESSAGE Requesting upgrade report for file: Test1.jpf

!ENTRY com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447
!MESSAGE Requesting upgrade report for file: Test2.jpf
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447
!MESSAGE The Java 5 annotation Jpf.Controller needs to be added.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447
!MESSAGE More annotation of Jpf.Action needs to be added.

!ENTRY com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE Requesting upgrade report for file: Test3.jpf
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE The Java 5 annotation Jpf.Controller needs to be added.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE The Java 5 annotation Jpf.Controller needs to be added.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE ABC needs to be added.
!SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634
!MESSAGE ABC needs to be added.


Output for requirement1:
================================
/fullpath/Test.java
/fullpath/Test1.jpf

Output for requirement2:
================================
/fullpath/Test2.jpf (separated by tab) The Java 5 annotation Jpf.Controller needs to be added,More annotation of Jpf.Action needs to be added.
/fullpath/Test3.jpf (separated by tab) The Java 5 annotation Jpf.Controller needs to be added,Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support,ABC needs to be added.

You may try something like this:

1.

Code:

awk '/<OKAY>/ { 
  sub(/>[^>]*$/, "")
  sub(/.*RESOURCE: /,"")
  print 
  }' infile

2.

Code:

awk 'END { if (r) print h "\t" r "." }
!NF { 
  if (r) print h "\t" r "." 
  f = r = 0; split("", t) 
  }
/^!ENTRY.*<INFO>/ { 
  sub(/>[^>]*$/, ""); sub(/.*RESOURCE: /, "")
  h = $0; f = 1 
  }  
f && /!SUBENTRY/ { f++ }   
f > 1 && sub(/!MESSAGE /, "") {
  sub(/.$/, ""); t[$0]++ || r = r ? r ", " $0 : $0 
  }' infile