Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help with cut and parse the line

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with cut and parse the line
# 1  
Old 10-06-2010
Question Help with cut and parse the line
I have log file which contains 1000s of lines something like this.

Code:
INFO |2010-08-29 14:23:37,078|SERIAL_ID=1283109816352|ST=2010-08-2914:23:36|DP_DEVICE=11.22.33.44:4420|TYPE=TransactionLo
g|LOG_LEVEL=6|CLASS=Superior Gateway|OBJECT=/services/getdetails|PROVIDERID=dpmq://APPSEKSQManager/?RequestQueue=QU
ERYMEMOLIST;ReplyQueue=APPS.REPLY01;Sync=true;Timeout=30000;UserName=mqm|CONSUMERID=APPS|OPERATION_NAME={http://integration.spr
int.com/interfaces/getdetails/v1/getdetails.wsdl}getdetailsType/getdetailsList|MsgID=414d5120445030314150442020
2020204|TOTAL_TIME=532|DE_TIME=5|REQUEST_MESSAGE=<SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchem
a" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOA
P-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><m:wsMessageHeader xmlns:m="http://integration.xyz.com/
common/header/WSMessageHeader/v2"><m:trackingMessageHeader><m:applicationId>APPS</m:applicationId><m:applicationUserId>System<
/m:applicationUserId><m:consumerId>THS</m:consumerId><m:messageId>114972513</m:messageId><m:conversationId>114972513</m:conve
rsationId><m:timeToLive>45</m:timeToLive><m:replyCompletionCode>0</m:replyCompletionCode><m:messageDateTimeStamp>2010-08-29T1
4:23:35-05:00</m:messageDateTimeStamp></m:trackingMessageHeader></m:wsMessageHeader></SOAP-ENV:Header><SOAP-ENV:Body><tns:que
ryMemoList xmlns:tns="http://integration.xyz.com/interfaces/getdetailsList/v1/getdetailsList.xsd"><tns:info><tns:ban>56363728382939823
18</tns:ban></tns:info><tns:selectionCriteria><tns:memoDateRange><tns:fromDate>2010-05-31</tns:fromDate><tns:toDate>2010-08-2
9</tns:toDate></tns:memoDateRange><tns:memoLevel>B</tns:memoLevel></tns:selectionCriteria></tns:getdetailsList></SOAP-ENV:Body
></SOAP-ENV:Envelope>|RESPONSE_MESSAGE=...[RESPONSE MESSAGE not included under current logging level. Adjust according in log
4j.properties file based on current requirement.]

But I want to parse the only XML's to a different file excluding this part of the line

Code:
INFO |2010-08-29 14:23:37,078|SERIAL_ID=12831098152|ST=2010-08-2914:23:36|DE_DEVICE=11.22.33.44:4420|TYPE=TransactionLo
g|LOG_LEVEL=6|CLASS=Superior Gateway|OBJECT=/services/getdetails|PROVIDERID=dpmq://APPSEKSQManager/?RequestQueue=QU
ERYMEMOLIST;ReplyQueue=APPS.REPLY01;Sync=true;Timeout=30000;UserName=mqm|CONSUMERID=APPS|OPERATION_NAME={http://integration.xyz.com/interfaces/getdetails/v1/getdetails.wsdl}getdetailsType/getdetailsList|MsgID=414d5120445030314150442020
2020204|TOTAL_TIME=532|DE_TIME=5|REQUEST_MESSAGE=

&&

Code:
|RESPONSE_MESSAGE=

# 2  
Old 10-06-2010
if i understood it correctly, grep's negate option is the one you are looking for.

Code:
 
grep -v 'LINE to negate' FILE

# 3  
Old 10-06-2010
try this,

Code:
#!/usr/bin/perl

use strict;
my $flag=0;

open (FH,"<","/path/to/ur/inputfile") || die "cannot open file\n";

while (<FH>) {
if (/\|REQUEST_MESSAGE=(.*)$/) {
print $1,"\n" ;
$flag=1;
next;
}
if ( $flag==1 && $_ !~ /\|RESPONSE_MESSAGE/) {
print $_;
}
if (/(.+?)\|RESPONSE_MESSAGE/) { print $1,"\n"; $flag=0;}
}
close(FH);

# 4  
Old 10-06-2010
Code:
$ ruby -00 -ne 'p $_.split("<",2)[1]' file

# 5  
Old 10-06-2010
Code:
perl -ne 'undef $/; /^.*?REQUEST_MESSAGE=/s; print  $&' input_filename

# 6  
Old 10-06-2010
Guess your sample log should be in one line.

Code:
awk '{printf $0}' infile

so you'd like to exclude the part before first <

Code:
awk 'BEGIN{FS=OFS="<"} {$1=""}1' infile.log

# 7  
Old 10-07-2010
Yes, the entire log is only in one LINE. I did some trial like this.

Code:
awk '{print $8}'| cut -d'|'  -f16,19 serviceinfoservice.log

The output is something like this:

Code:
REQUEST_MESSAGE=<SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchem
a" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOA
P-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header><m:wsMessageHeader xmlns:m="http://integration.xyz.com/
common/header/WSMessageHeader/v2"><m:trackingMessageHeader><m:applicationId>APPS</m:applicationId><m:applicationUserId>System<
/m:applicationUserId><m:consumerId>THS</m:consumerId><m:messageId>114972513</m:messageId><m:conversationId>114972513</m:conve
rsationId><m:timeToLive>45</m:timeToLive><m:replyCompletionCode>0</m:replyCompletionCode><m:messageDateTimeStamp>2010-08-29T1
4:23:35-05:00</m:messageDateTimeStamp></m:trackingMessageHeader></m:wsMessageHeader></SOAP-ENV:Header><SOAP-ENV:Body><tns:que
ryMemoList xmlns:tns="http://integration.xyz.com/interfaces/getdetailsList/v1/getdetailsList.xsd"><tns:info><tns:ban>56363728382939823
18</tns:ban></tns:info><tns:selectionCriteria><tns:memoDateRange><tns:fromDate>2010-05-31</tns:fromDate><tns:toDate>2010-08-2
9</tns:toDate></tns:memoDateRange><tns:memoLevel>B</tns:memoLevel></tns:selectionCriteria></tns:getdetailsList></SOAP-ENV:Body
></SOAP-ENV:Envelope>|RESPONSE_MESSAGE=...[RESPONSE MESSAGE not included under current logging level. Adjust according in log
4j.properties file based on current requirement.]
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Ksh: Read line parse characters into variable and remove the line if the date is older than 50 days

I have a test file with the following format, It contains the username_date when the user was locked from the database. $ cat lockedusers.txt TEST1_21062016 TEST2_02122015 TEST3_01032016 TEST4_01042016 I'm writing a ksh script and faced with this difficult scenario for my... (11 Replies)
Discussion started by: humble_learner
11 Replies

2. Shell Programming and Scripting

Parse the next line

If you have a file like this Mike Student:1:4 Boy Student:3:4 Girl :master Then Output is: Mike Student:1:4 Boy Student:3:4 Girl :master (7 Replies)
Discussion started by: invinzin21
7 Replies

3. UNIX for Dummies Questions & Answers

Parse or cut concat variables to individual values

Hello I need to pass some environment parameters to a datastage job and am getting an error when trying to send the complete concatinated variable. I have decided to parse out just the values and send as parameters but am struggling to find the best way to do this (actually I am not very... (3 Replies)
Discussion started by: LynnC
3 Replies

4. Slackware

How should I cut this line using cut and grep?

not sure how to do it. wan't to delete it using cut and grep ince i would use it in the shell. but how must the command be? grep "64.233.181.103 wwwGoogle.com" /etc/hosts | cut -d the delimeter is just a space. can you help meplease. :D (1 Reply)
Discussion started by: garfish
1 Replies

5. Shell Programming and Scripting

To parse the line

Hi, I have a line QMNAME(qmgrname) STATUS(RUNNING) Can u jus tell me how to only get the status field ? And also the value of the status whether it is running or not running. -- Thanks (2 Replies)
Discussion started by: julie_s
2 Replies

6. UNIX for Advanced & Expert Users

how do you parse 1 line at a time of file1 ie. line(n) each line into new file

File 1 <html>ta da....unique file name I want to give file=>343...</html> <html>da ta 234 </html> <html>pa da 542 </html> and so on... File 2 343 234 542 and so on, each line in File 1 one also corresponds with each line in File 2 I have tried several grep, sed, while .. read, do,... (4 Replies)
Discussion started by: web_developer
4 Replies

7. Shell Programming and Scripting

cut a string in a textfile line per line

i need to cut the string in a textfile but each line has a specific way of cutting it (different lengths) i have a for loop that gets the string line per line, then each line has to be compared: for x in `cat tmp2.txt`; do if; then echo 'BAC' elif ... (6 Replies)
Discussion started by: izuma
6 Replies

8. Shell Programming and Scripting

SED help (remove line::parse again::add line)

Aloha! I have just over 1k of users that have permissions that they shouldn't under our system. I need to parse a provided list of usernames, check their permissions file, and strip the permissions that they are not allowed to have. If upon the permissions strip they are left with no permissions,... (6 Replies)
Discussion started by: Malumake
6 Replies

9. Shell Programming and Scripting

please help to parse the line

cp4 0 0 170.217.86.10.1421 170.217.86.8.53308 ESTABLISHED tcp4 0 0 170.217.86.10.1421 170.217.86.8.62948 ESTABLISHED tcp4 0 0 170.217.86.10.1421 170.217.86.8.62949 ESTABLISHED tcp4 0 0 170.217.86.10.1421 ... (1 Reply)
Discussion started by: ajaya
1 Replies

10. Shell Programming and Scripting

How to parse a line?

I'm currenting trying to parse the out put of the following command. iostat -xtc -r |grep cmdk0 which produces the output cmdk0,0.2,0.0,1.2,0.0,0.0,0.0,39.7,0,0,0,0,0,0,0,99 I'm then trying to get the data to look like this: rw=0.2 ws=0.0 krs=1.2 kws=0.0 wait=0.0 actv=0.0... (2 Replies)
Discussion started by: edefoe
2 Replies
Login or Register to Ask a Question