Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.
|
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.
|
|
google unix.com
|
|||||||
| Forums | Register | Forum Rules | Links | Albums | FAQ | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
 |
| UNIX for Advanced & Expert Users Expert-to-Expert. Learn advanced UNIX, UNIX commands, Linux, Operating Systems, System Administration, Programming, Shell, Shell Scripts, Solaris, Linux, HP-UX, AIX, OS X, BSD. |
More UNIX and Linux Forum Topics You Might Find Helpful
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Extracting Text | krabu | Shell Programming and Scripting | 3 | 04-07-2009 03:16 AM |
| extracting text and reusing the text to rename file | JohnDS | UNIX for Dummies Questions & Answers | 7 | 02-05-2009 03:55 AM |
| Extracting information from Config files /text processing | oconmx | Shell Programming and Scripting | 3 | 01-21-2009 07:09 PM |
| help required for replacing text in vi | Chandu2u | Shell Programming and Scripting | 6 | 01-26-2008 11:12 AM |
| Help required regarding Extracting lines from a file | google_ever | Shell Programming and Scripting | 1 | 10-12-2005 07:02 AM |
 |
|
|
|
LinkBack | Thread Tools | Search this Thread |
Rating: 
|
Display Modes |
06-20-2009
|
||||
|
||||
|
Extracting the required text from log files
It would be highly appreciable if any one helps me in this. I am trying to get it done through Java but I love unix and believe it can be done within minutes with couple of lines.
The input log file is a text file contains multiple entries seperated by a blank line. Each seperated entry corresponds to upgrade process information of one file. !ENTRY text.....<INFO> or <OKAY> <RESOURCE: /test/src/com/test1/*/test.java> 2009-06-18 13:01:01.181 !MESSAGE Requesting upgrade report for file: test.java !ENTRY text.....<INFO> or <OKAY> <RESOURCE: /test/src/com/test1/*/test1.java> 2009-06-18 13:01:01.181 !MESSAGE information in test1.java will be upgraded. !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: full path /file name> 2009-06-18 13:02:25.681 !MESSAGE Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support. Each Entry starts with "!ENTRY" as shown above and will be followed by text "com.bea.workshop.upgrade81 " and then it will be followed by two types of tags: <OKAY> and <INFO> And then it will be followed by tag <RESOURCE:which contains the full path of the file and then followed by time stamp as shown above. If it is <OKAY>, then second line will be as below !MESSAGE Requesting upgrade report for file: filename ( Which is not of much importance for my output) Ex: !ENTRY text.....<INFO> or <OKAY> <RESOURCE: /test/src/com/test1/*/test.java> 2009-06-18 13:01:01.181 !MESSAGE Requesting upgrade report for file: file name If it is <INFO> then also second line will be like !MESSAGE Requesting upgrade report for file: filename But it will certainly be followed by another set of multiple line pairs starting with tags !SUBENTRY,!MESSAGE like below !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: path/filename> 2009-06-18 13:02:25.681 !MESSAGE Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support. My requirement is like below: Requirement 1 1) For all entries containing <OKAY> tag, I need to extract the file names which will be after <RESOURCE:.....and before time stamp Please note the entries are seperated by a blank line Requirement 2 2) For all the lines with <INFO> tag, I would like to have a text file with entries like this Full path and the file name and then in the next line All the text after !MESAAGE right below the line containg the tag of corresponding "!SUBENTRY 1" tags Example Input: ============================== !ENTRY com.bea.workshop.upgrade81 <OKAY> <RESOURCE: /fullpathr/Test.java> 2009-06-18 13:02:28.368 !MESSAGE Requesting upgrade report for file: Test.java !ENTRY com.bea.workshop.upgrade81 <OKAY> <RESOURCE: /fullpath/Test1.jpf> 2009-06-18 13:02:28.384 !MESSAGE Requesting upgrade report for file: Test1.jpf !ENTRY com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447 !MESSAGE Requesting upgrade report for file: Test2.jpf !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447 !MESSAGE The Java 5 annotation Jpf.Controller needs to be added. !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447 !MESSAGE More annotation of Jpf.Action needs to be added. !ENTRY com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634 !MESSAGE Requesting upgrade report for file: Test3.jpf !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634 !MESSAGE The Java 5 annotation Jpf.Controller needs to be added. !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634 !MESSAGE Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support. !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634 !MESSAGE ABC needs to be added. Output for requirement1: ================================ Test.java Test1.jpf Output for requirement2: ================================ /fullpath/Test2.jpf The Java 5 annotation Jpf.Controller needs to be added. More annotation of Jpf.Action needs to be added. /fullpath/Test3.jpf The Java 5 annotation Jpf.Controller needs to be added. Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support. ABC needs to be added. Last edited by hareeshram; 06-20-2009 at 09:09 AM.. |
|
06-20-2009
|
||||
|
||||
|
This works, but probably only if your input is exactly as you described!
Code:
echo "Output from requirement 1"
echo "========================="
grep "<OKAY>.*RESOURCE" infile | sed -e "s/.*RESOURCE:.*\/\(.*\)>.*/\1/" | sort -u
echo
echo "Output from requirement 2"
echo "========================="
awk '
/^!ENTRY.*<INFO>/ { X = 1; sub( /.*RESOURCE: /, "", $0 ); sub( />.*/, "", $0 ); print }
(X == 1) && (/^!SUBENTRY/) { X++ }
(X > 1) && ($1 ~ /^!MESSAGE/) { sub( /!MESSAGE /, "", $0 ); print }
(X > 1) && ($1 ~ /^$/) { print ""; X = 0 }
' infile
Output:
Output from requirement 1
=========================
Test1.jpf
Test.java
Output from requirement 2
=========================
/fullpath/Test2.jpf
The Java 5 annotation Jpf.Controller needs to be added.
More annotation of Jpf.Action needs to be added.
/fullpath/Test3.jpf
The Java 5 annotation Jpf.Controller needs to be added.
Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support.
ABC needs to be added.
Last edited by scottn; 06-20-2009 at 10:44 AM.. |
|
06-20-2009
|
|||||
|
|||||
|
Use gawk, nawk or /usr/xpg4/bin/awk on Solaris:
1. Code:
awk '/<OKAY>/ {
sub(/>[^>]*$/, "")
n = split($0, t, "/")
print t[n]
}' infile
Code:
awk '!NF { f = 0 }
/^!ENTRY.*<INFO>/ {
sub(/>[^>]*$/, "")
sub(/.*RESOURCE: /, "")
print; f = 1
}
f && /!SUBENTRY/ { f++ }
f > 1 && sub(/!MESSAGE /, "")
' infile
Last edited by radoulov; 06-20-2009 at 10:48 AM.. |
| Bits Awarded / Charged to radoulov for this Post | |||
| Date | User | Comment | Amount |
| 06-22-2009 | hareeshram | Thanks for the smart solution! | 200 |
|
06-20-2009
|
|||||
|
|||||
|
Quote:
I need to make the code more generic (I just modified it to remove the specific columns references). Last edited by radoulov; 06-20-2009 at 11:08 AM.. Reason: wrong statement :), your output is correct |
|
06-22-2009
|
||||
|
||||
|
That was so fast and accurate!
Thanks scottn and radoulov.
It worked fine for me I have a slight change in format that I would expect for both requirements. Hope you would suggest me. Requirement 1: The file names should come with full paths Requirement 2: After the file name (with full path), whatever the messages that are being displayed (one message for one line), they are to be displayed without any duplication ( the same message in consecutive lines should be removed) per entry and all non duplicated entries should be separted by comma (rather than new line"). Unlike in the above case, both file and comma separated messages should come in the single line. Once again many many thanks for keeping my spirit up in unix. Example input ========= Example Input: ============================== !ENTRY com.bea.workshop.upgrade81 <OKAY> <RESOURCE: /fullpath/Test.java> 2009-06-18 13:02:28.368 !MESSAGE Requesting upgrade report for file: Test.java !ENTRY com.bea.workshop.upgrade81 <OKAY> <RESOURCE: /fullpath/Test1.jpf> 2009-06-18 13:02:28.384 !MESSAGE Requesting upgrade report for file: Test1.jpf !ENTRY com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447 !MESSAGE Requesting upgrade report for file: Test2.jpf !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447 !MESSAGE The Java 5 annotation Jpf.Controller needs to be added. !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test2.jpf> 2009-06-18 13:02:28.447 !MESSAGE More annotation of Jpf.Action needs to be added. !ENTRY com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634 !MESSAGE Requesting upgrade report for file: Test3.jpf !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634 !MESSAGE The Java 5 annotation Jpf.Controller needs to be added. !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634 !MESSAGE The Java 5 annotation Jpf.Controller needs to be added. !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634 !MESSAGE Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support. !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634 !MESSAGE ABC needs to be added. !SUBENTRY 1 com.bea.workshop.upgrade81 <INFO> <RESOURCE: /fullpath/Test3.jpf> 2009-06-18 13:02:28.634 !MESSAGE ABC needs to be added. Output for requirement1: ================================ /fullpath/Test.java /fullpath/Test1.jpf Output for requirement2: ================================ /fullpath/Test2.jpf (separated by tab) The Java 5 annotation Jpf.Controller needs to be added,More annotation of Jpf.Action needs to be added. /fullpath/Test3.jpf (separated by tab) The Java 5 annotation Jpf.Controller needs to be added,Will add import of org.apache.beehive.netui.pageflow.annotations.Jpf for JPF annotation support,ABC needs to be added. |
|
06-22-2009
|
|||||
|
|||||
|
You may try something like this:
1. Code:
awk '/<OKAY>/ {
sub(/>[^>]*$/, "")
sub(/.*RESOURCE: /,"")
print
}' infile
Code:
awk 'END { if (r) print h "\t" r "." }
!NF {
if (r) print h "\t" r "."
f = r = 0; split("", t)
}
/^!ENTRY.*<INFO>/ {
sub(/>[^>]*$/, ""); sub(/.*RESOURCE: /, "")
h = $0; f = 1
}
f && /!SUBENTRY/ { f++ }
f > 1 && sub(/!MESSAGE /, "") {
sub(/.$/, ""); t[$0]++ || r = r ? r ", " $0 : $0
}' infile
|
| Sponsored Links | ||
|
|
|
| Bookmarks |
| Tags |
| log extract |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
Linear Mode
