Unix/Linux Go Back    


Shell Programming and Scripting Unix shell scripting - KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and shell scripts and shell scripting languages here.

Problem with python script

Shell Programming and Scripting


Closed Linux or Unix Question    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 11-19-2012
Vijeta Laad Vijeta Laad is offline
Banned
 
Join Date: Nov 2012
Last Activity: 27 November 2012, 2:10 PM EST
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
Problem with python script

I have a txt file of 8GB with same type of messages in it. for eg:
Code:
MTQ_BQUOTE, Length: 40, Timestamp: 4:00:52.064
MsgKey: symbol: XXX | reportingExchange: 11
Symbol: XXX, hash 004C5746
QS Symbol: XXX, market 1
Security Type: EQUAL (1)
Symbol Type: Equity.Share.Single.None
Session: XX_XXX (0)
Ticker Exchange: ABCD (11) => 11
Flags: x00000000
Bid: 2475, frac: 2, pure7 val: 24.75
Bid Size: 50
Ask: 0, frac: 0, pure7 val: ---
Ask Size: 0
Quote Condition: x00

There are tons of same messages with different values in the file. Can you please help me with a python script that can saperate the whole message with unique Flags value in an output file.

Last edited by jim mcnamara; 11-19-2012 at 04:20 PM..
Sponsored Links
    #2  
Old Unix and Linux 11-19-2012
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 4 September 2015, 8:59 PM EDT
Location: NM
Posts: 10,532
Thanks: 354
Thanked 882 Times in 820 Posts
Does it have to be python? There are other suitable tools.
Sponsored Links
    #3  
Old Unix and Linux 11-20-2012
Vijeta Laad Vijeta Laad is offline
Banned
 
Join Date: Nov 2012
Last Activity: 27 November 2012, 2:10 PM EST
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
UNIX?

It does not necessarily be python. I use a tool called Cygwin (UNIX like). Can it be UNIX command?
    #4  
Old Unix and Linux 11-20-2012
itkamaraj's Unix or Linux Image
itkamaraj itkamaraj is offline Forum Advisor  
^Kamaraj^
 
Join Date: Apr 2010
Last Activity: 25 June 2015, 3:24 AM EDT
Posts: 3,060
Thanks: 34
Thanked 661 Times in 640 Posts
can you explain more about " can saperate the whole message with unique Flags value in an output file."
Sponsored Links
    #5  
Old Unix and Linux 11-20-2012
Vijeta Laad Vijeta Laad is offline
Banned
 
Join Date: Nov 2012
Last Activity: 27 November 2012, 2:10 PM EST
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
Unique Flags value

Quote:
Originally Posted by itkamaraj View Post
can you explain more about " can saperate the whole message with unique Flags value in an output file."
I have a text file that has several messages, here I put 3 examples:


Code:
MTR_MKTSTAT, Length: 104, Timestamp: 3:30:00.000
MsgKey: statId: @ | type: T | basis: I | exchgId: O | modifier: T
Symbol: @XXX, hash 4F495415
QS Symbol: @XXXX, market 236
Security Type: MKTSTATS (8)
Symbol Type: Statistic.Share.Single.None
Session: US_Day (3)
Ticker Exchange: DTN (254) => 254
Flags: x00000000: 
Description: NASDAQ 3RD MKT ISSUES TICKS TOTAL 
Last: 446, frac: 0, pure7 val: 446
Open: 146, frac: 0, pure7 val: 146
High: 446, frac: 0, pure7 val: 446
Low: 146, frac: 0, pure7 val: 146
Close: 446, frac: 0, pure7 val: 446
Change: 0, frac: 0, pure7 val: --- 
Market Status: x01
Last Trade Time: 16:05:00.000
Last Trade Date: 10/4/2012
Open Time: 9:31:00.000
Close Time: 16:05:00.000
 
MTR_MKTSTAT, Length: 104, Timestamp: 3:30:00.000
MsgKey: statId: @ | type: T | basis: R | exchgId: O | modifier: T
Symbol: @TROT, hash 4F525415
QS Symbol: @TROT, market 236
Security Type: MKTSTATS (8)
Symbol Type: Statistic.Share.Single.None
Session: US_Day (3)
Ticker Exchange: DTN (254) => 254
Flags: x00000000: 
Description: NASDAQ 3RD MKT TICKS RATIO              
Last: 119, frac: 2, pure7 val: 1.19
Open: 106, frac: 2, pure7 val: 1.06
High: 124, frac: 2, pure7 val: 1.24
Low: 75, frac: 2, pure7 val: 0.75
Close: 119, frac: 2, pure7 val: 1.19
Change: 0, frac: 2, pure7 val:  --- 
Market Status: x01
Last Trade Time: 16:05:00.000
Last Trade Date: 10/4/2012
Open Time: 9:31:00.000
Close Time: 16:05:00.000
 
MTR_MKTSTAT, Length: 104, Timestamp: 3:30:01.000
MsgKey: statId: @ | type: I | basis: P | exchgId: T | modifier: T
Symbol: @XXX, hash 54504915
QS Symbol: @XXXX, market 236
Security Type: MKTSTATS (8)
Symbol Type: Statistic.Share.Single.None
Session: US_Day (3)
Ticker Exchange: DTN (254) => 254
Flags: x00000010: LAST 
Description: 
Last: 1396, frac: 0, pure7 val: 1396
Open: 5, frac: 0, pure7 val: 5
High: 1396, frac: 0, pure7 val: 1396
Low: 3, frac: 0, pure7 val: 3
Close: 0, frac: 0, pure7 val: --- 
Change: 1396, frac: 0, pure7 val: 1396
Market Status: x02
Last Trade Time: 3:30:01.000
Last Trade Date: 10/5/2012
Open Time: 0:00:00.000
Close Time: 23:59:59.000


If you look at the 9th line of each message, there is Flags with a code like x00000000. But the first 2 messages have the same code so, I want to saperate only those messages that have a unique code in a text file.

Last edited by Scrutinizer; 11-20-2012 at 02:37 PM.. Reason: Wanted to put 3 example messages but accidently put only 2. Therefore added third message.; mod: code tags
Sponsored Links
    #6  
Old Unix and Linux 11-20-2012
alister alister is offline
Registered User
 
Join Date: Dec 2009
Last Activity: 11 June 2014, 8:40 PM EDT
Posts: 3,231
Thanks: 179
Thanked 975 Times in 790 Posts
If you just want to filter out one type, and if messages are separated by at least one blank line:

Code:
awk 'index($9, "code_to_match")' RS=

Regards,
Alister
Sponsored Links
    #7  
Old Unix and Linux 11-20-2012
itkamaraj's Unix or Linux Image
itkamaraj itkamaraj is offline Forum Advisor  
^Kamaraj^
 
Join Date: Apr 2010
Last Activity: 25 June 2015, 3:24 AM EDT
Posts: 3,060
Thanks: 34
Thanked 661 Times in 640 Posts
The below code creates the file with flagname (with the appropriate message block)


Code:
 
 $ nawk '/MTR/,/Close Time/{a[i++]=$0;if($0~/Flags:/){sub(":","",$2);fname=$2}if($0~/Close Time/){for(j=0;j<=i;j++)print a[j] >> fname;i=0}}' input.txt

In Solaris, use nawk , otherwise you can use awk

Above code will create two files ( x00000000, x00000010 ) - for the sample text you posted.

Last edited by itkamaraj; 11-20-2012 at 11:36 PM..
Sponsored Links
Closed Linux or Unix Question

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Problem passing arguments to a Python script kristinu Shell Programming and Scripting 0 01-18-2011 04:58 PM
Passing variable from shell script to python script shashi792 Shell Programming and Scripting 4 12-02-2010 05:20 AM
problem with python on mac thefloydpink UNIX for Dummies Questions & Answers 0 02-11-2009 11:19 AM
Python tkinter grid problem? SRCoder! Shell Programming and Scripting 0 06-19-2004 11:49 AM



All times are GMT -4. The time now is 01:44 AM.