Write out specific data from log to a new file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Write out specific data from log to a new file
# 8  
Old 08-26-2013
Did you try my code? It has fixed your issue.
# 9  
Old 08-27-2013
Yes, it produces this:

Code:
2013-,72013-9 (1093:               jms_con.cpp) [140561430333184] DEBUG  - Received XML TextMessage:
<?xml version="1.0" encoding="UTF-8"?><iSig xmlns:xsi="http://www.w3.org/
2013-013-08-16 04:59:03,665 (1070:               jms_con.cpp) [140561430333184] DEBUG  - No ReplyTo queue was specified.
2013-013-08-16 04:59:03,665 (1093:               jms_con.cpp) [140561430333184] DEBUG  - Received XML TextMessage:
<?xml version="1.0" encoding="UTF-8"?><iSig xmlns:xsi="http://www.w3.org/

I need the whole xml message + INFO line with same ID.
# 10  
Old 08-27-2013
I do suggest you use solution I posted in another thread for you.

Modify this some to output to a new file what you want from the log.

-----------------------

I have divided this into multiple tasks. It may be that someone can join all this together.

1- Find all Session ID form line containing INFO and store it to file **t1**
Code:
awk -F":|," 'FNR==NR && /INFO/ {a[$6$8]++;next} END {for (i in a) print i }' xml_file >t1

2- Remove all log lines (starting with 2013) and store it in **t2**
Code:
awk  '!/^2013/' xml_file >t2

3- Print every XML bulk if it does contain one of the ID found in step 1
Code:
awk 'FNR==NR {a[$1]++;next} FNR==1 {RS="</continueProcessing>"} { for (i in a) {if ($0~i) print}}'  t1 t2

I have assumed that all XML section do end with </continueProcessing>. If that is not true, this will not work.
# 11  
Old 08-27-2013
Yeah, thats ok.

But i need the INFO lines too. Every same ID' to a new file with the xml messages.
# 12  
Old 08-27-2013
Code:
awk -F":|," 'FNR==NR && /INFO/ {a[$6$8]=$0;next} END {for (i in a) print i "|" a[i] }' xml_file >t1

Code:
awk  '!/^2013/' xml_file >t2

Code:
awk -F\| 'FNR==NR {a[$1]=$2;next} FNR==1 {RS="</continueProcessing>"} { for (i in a) {if ($0~i) print a[i] "\n" $0}}'  t1 t2 >new_xml

Code:
cat new_xml
2013-08-16 16:31:45,110 ( 122:            rogate) [98839276727]  INFO  -      UId:11453, GId:2133: new CONX started, Application Context: disconnected
 <version>1</version>
 <sessionId>114532133</sessionId>
 <networkProtocolId>CAPv2</networkProtocolId>
 <trafficType>Forwarding</trafficType>
  <messages>
   <reportNotificationAck/>
 <superviseReq>
 <requestSequenceNr>0</requestSequenceNr>
 <time>60000</time>
 <releaseAfterTimeExpires>false</releaseAfterTimeExpires>
  <playWarningTone>false</playWarningTone>
 </superviseReq>
 <eventReportReq>
 <requestSequenceNr>1</requestSequenceNr>
 <events>
<routeSelectFailure monitorMode="Interrupt"/>
<busy monitorMode="Interrupt"/>
<noAnswer monitorMode="Interrupt">
  <noAnswerTimer>180000</noAnswerTimer>
</noAnswer>
<answer monitorMode="Notify"/>
<disconnectCalling monitorMode="Interrupt"/>
<disconnectCalled monitorMode="Interrupt"/>
<abandon monitorMode="Notify"/>
</events>
</eventReportReq>
<continueProcessing>
<requestSequenceNr>2</requestSequenceNr>
<moreEventsExpected>true</moreEventsExpected>
<interruptEventReceived>true</interruptEventReceived>

This way you have the corresponding INFO line (marked red) above its XML section.

Its not easy to make sure if this is correct with only one XML section (small logfile sample), and no output example.
# 13  
Old 08-27-2013
I can provide a bigger sample of the file.

Code:
2013-08-16 05:08:56,694 ( 225:     cap_ulti_listener.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: retrieving the full application context from TCAP Context Storage.
2013-08-16 05:08:56,694 (1784:  tcap_context_storage.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: Retrieving the TCAP context.
2013-08-16 05:08:56,694 (1789:  tcap_context_storage.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: The TCAP context was FOUND in the internal cache.
2013-08-16 05:08:56,694 ( 231:     cap_ulti_listener.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: context retrieved successfully.
2013-08-16 05:08:56,695 ( 237:     cap_ulti_listener.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: going to encode Delimiter (iCAP->iSig).
2013-08-16 05:08:56,695 ( 239:     cap_ulti_listener.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: encoding of Delimiter (iCAP->iSig) was successful.
2013-08-16 05:08:56,695 ( 256:     cap_ulti_listener.cpp) [140561893431328]  INFO  - AId:57371, DId:6848: onDelimiter handler finished successfully.
2013-08-16 05:08:56,695 ( 992:            capgw_main.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: decoding of new CAP operation finished.
2013-08-16 05:08:56,695 ( 999:            capgw_main.cpp) [140561893431328]  INFO  - AId:57371, DId:6848: STAT_TCAP_PROCESSING: 0.001235.
2013-08-16 05:08:56,695 ( 767:  tcap_context_storage.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: updating the Last TCAP Access Time.
2013-08-16 05:08:56,695 ( 804:  tcap_context_storage.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: new TCAP message detected, no response time can be calculated.
2013-08-16 05:08:56,695 (1008:            capgw_main.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: TCAP Context Storage updated successfully (received TCAP message).
2013-08-16 05:08:56,695 (  59:     cap_ulti_listener.cpp) [140561893431328] DEBUG  - Returning the actual internal queue: VIRGIN_MTC_REQ.
2013-08-16 05:08:56,695 ( 907:               jms_con.cpp) [140561893431328] DEBUG  - Sending XML TextMessage:
<?xml version="1.0" encoding="UTF-8"?><iSig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.inew-cs.com/xml/isig/1.0/camelMessages.xsd"><version>1</version><sessionId>573716848</sessionId><networkProtocolId>CAPv2</networkProtocolId><trafficType>Terminating</trafficType><messages><superviseRes><invokeId>2</invokeId><usedTime>189400</usedTime><callActive>false</callActive></superviseRes><eventReportRes><invokeId>3</invokeId><disconnectCalled monitorMode="Interrupt"/><eventTimeStamp>2013-08-16T05:08:56Z</eventTimeStamp></eventReportRes></messages></iSig>
2013-08-16 05:08:56,695 ( 888:  tcap_context_storage.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: updating the Last Appl. Access Time.
2013-08-16 05:08:56,695 ( 925:  tcap_context_storage.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: new Appl. message detected, no response time can be calculated.
2013-08-16 05:08:56,695 (1027:            capgw_main.cpp) [140561893431328] DEBUG  - AId:57371, DId:6848: TCAP Context Storage updated successfully (sent iSig message).
2013-08-16 05:08:56,699 (1070:               jms_con.cpp) [140561430333184] DEBUG  - No ReplyTo queue was specified.
2013-08-16 05:08:56,699 (1093:               jms_con.cpp) [140561430333184] DEBUG  - Received XML TextMessage:
<?xml version="1.0" encoding="UTF-8"?><iSig xmlns:xsi="http://www.xxx.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.inew-cs.com/xml/isig/1.0/camelMessages.xsd">
  <version>1</version>
  <sessionId>573716848</sessionId>
  <networkProtocolId>CAPv2</networkProtocolId>
  <trafficType>Terminating</trafficType>
  <messages>
    <release>
      <requestSequenceNr>3</requestSequenceNr>
      <cause>32</cause>
    </release>
  </messages>
</iSig>
2013-08-16 05:08:56,699 (1233:            capgw_main.cpp) [140561430333184] DEBUG  - handleISigMessage - start
2013-08-16 05:08:56,699 (1569:     cap_isig_listener.cpp) [140561430333184] DEBUG  - onRelease handler.
2013-08-16 05:08:56,699 (1573:     cap_isig_listener.cpp) [140561430333184] DEBUG  - AId:57371, DId:6848: XML->iCAP:
[Release]
ApplicationId : 57371
DialogId : 6848
InvokeId : 3
Version : v2
Cause : 32

There is an INFO line with AId+DId, and somewhere a DEBUG line with the XML message, with sessionID what is same as AId+UId. I need these INFO + DEBUG line into a new file. Every same lines with the same Id's into a new file.
# 14  
Old 08-27-2013
Did you test my new post?
Still only one xml section with sessionID.
xml section does not have same format as in post #1, and this makes it hard to split this up.
You could zip the complete file, upload it, and then post example on how you like to output.

You have two INFO line with same AId and DId, who should I use?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare 2 text file with 1 column in each file and write mismatch data to 3rd file

Hi, I need to compare 2 text files with around 60000 rows and 1 column. I need to compare these and write the mismatch data to 3rd file. File1 - file2 = file3 wc -l file1.txt 58112 wc -l file2.txt 55260 head -5 file1.txt 101214200123 101214700300 101250030067 101214100500... (10 Replies)
Discussion started by: Divya Nochiyil
10 Replies

2. Shell Programming and Scripting

Write over data to new file

hi..i would ask about how to write over data to new file with BASH. so..assume my data looks like this : 11 12 13 14 15 ...and so on. It's always line by line. and that's for the first file. i want to write over those numbers into second file but by using space. so my second file should be... (5 Replies)
Discussion started by: syalala
5 Replies

3. Shell Programming and Scripting

write specific line number in file

dear all, i need your advice i have sample script like this: testing.sh for i in {1..10} do echo testing $i done but i forgot create "#!/bin/bash" in above "for" so i want output will like this testing.sh #!/bin/bash for i in {1..10} do echo testing $i done (2 Replies)
Discussion started by: zvtral
2 Replies

4. Shell Programming and Scripting

How can I write the specific content in the file through shell script

Hello, I need to do one thing that my script creates the file touch release.SPLASH_12_03_00_RC01.txt Now I want to update that file with some content e.g splashbuild::SPLASH_12_17_00_RC02.zip Thanks (1 Reply)
Discussion started by: anuragpgtgerman
1 Replies

5. Shell Programming and Scripting

how read specific line in a file and write it in a new text file?

I have list of files in a directory 'dir'. Each file is of type HTML. I need to read each file and get the string which starts with 'http' and write them in a new text file. How can i do this shell scripting? file1.html <head> <url>http://www.google.com</url> </head> file2.html <head>... (6 Replies)
Discussion started by: vel4ever
6 Replies

6. UNIX for Dummies Questions & Answers

how to write a function to get data under specific lines ?

I have a text file called (msgz ) contains data : Subscriber Data ID = 2 Customer = 99 Data ID = 4 Customer = cf99 Data ID = 5 Customer = c99 Data ID = 11 Customer = 9n9 Subscriber Data ID = 1 Customer = 9ds9 Data ID = 2 Customer = 9sad9 Data ID = 3 Customer = f99... (3 Replies)
Discussion started by: teefa
3 Replies

7. Programming

How to write data to file in C?

Hi I want to open a file and write data in the following manner. Header String 1 String 2 String 3 String 4 String 5 ... (4 Replies)
Discussion started by: AAKhan
4 Replies

8. Hardware

how to write data into a device file?

Hi, I am working in device drivers. I am new to device drivers. i have invoked chardev.c. the driver is insmoded. now i want to write something into this and i want to look what i have written. but i don't know how to write and see. please help me (0 Replies)
Discussion started by: boidi
0 Replies

9. Shell Programming and Scripting

Read Write byte range/chunk of data from specific location in file

I am new to Unix so will really appreciate if someone can guide me on this. What I want to do is: Step1: Read binary file - pick first 2 bytes, convert from hex to decimal. Read the next 3 bytes as well. 2 bytes will specify the number of bytes 'n' that I want to read and write... (1 Reply)
Discussion started by: Kbenipel
1 Replies

10. Shell Programming and Scripting

Extract data from log file from or after the specific date

Hi , I am having a script which will start a process and appends the process related logs to a log file. The log file writes logs with every line starting with date in the format of: date +"%Y %b %d %H:%M:%S". So, in the script, before I start the process, I am storing the date as DATE=`date +"%Y... (5 Replies)
Discussion started by: chiru_h
5 Replies
Login or Register to Ask a Question