Problem with python script | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Problem with python script

Shell Programming and Scripting


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 11-19-2012
Vijeta Laad Vijeta Laad is offline
Banned
 
Join Date: Nov 2012
Last Activity: 27 November 2012, 2:10 PM EST
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
Problem with python script

I have a txt file of 8GB with same type of messages in it. for eg:
Code:
MTQ_BQUOTE, Length: 40, Timestamp: 4:00:52.064
MsgKey: symbol: XXX | reportingExchange: 11
Symbol: XXX, hash 004C5746
QS Symbol: XXX, market 1
Security Type: EQUAL (1)
Symbol Type: Equity.Share.Single.None
Session: XX_XXX (0)
Ticker Exchange: ABCD (11) => 11
Flags: x00000000
Bid: 2475, frac: 2, pure7 val: 24.75
Bid Size: 50
Ask: 0, frac: 0, pure7 val: ---
Ask Size: 0
Quote Condition: x00

There are tons of same messages with different values in the file. Can you please help me with a python script that can saperate the whole message with unique Flags value in an output file.

Last edited by jim mcnamara; 11-19-2012 at 04:20 PM..
Sponsored Links
    #2  
Old 11-19-2012
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 22 October 2014, 11:15 AM EDT
Location: NM
Posts: 10,247
Thanks: 281
Thanked 803 Times in 747 Posts
Does it have to be python? There are other suitable tools.
Sponsored Links
    #3  
Old 11-20-2012
Vijeta Laad Vijeta Laad is offline
Banned
 
Join Date: Nov 2012
Last Activity: 27 November 2012, 2:10 PM EST
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
UNIX?

It does not necessarily be python. I use a tool called Cygwin (UNIX like). Can it be UNIX command?
    #4  
Old 11-20-2012
itkamaraj's Avatar
itkamaraj itkamaraj is offline Forum Advisor  
^Kamaraj^
 
Join Date: Apr 2010
Last Activity: 3 October 2014, 8:50 AM EDT
Posts: 3,059
Thanks: 33
Thanked 659 Times in 638 Posts
can you explain more about " can saperate the whole message with unique Flags value in an output file."
Sponsored Links
    #5  
Old 11-20-2012
Vijeta Laad Vijeta Laad is offline
Banned
 
Join Date: Nov 2012
Last Activity: 27 November 2012, 2:10 PM EST
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
Unique Flags value

Quote:
Originally Posted by itkamaraj View Post
can you explain more about " can saperate the whole message with unique Flags value in an output file."
I have a text file that has several messages, here I put 3 examples:


Code:
MTR_MKTSTAT, Length: 104, Timestamp: 3:30:00.000
MsgKey: statId: @ | type: T | basis: I | exchgId: O | modifier: T
Symbol: @XXX, hash 4F495415
QS Symbol: @XXXX, market 236
Security Type: MKTSTATS (8)
Symbol Type: Statistic.Share.Single.None
Session: US_Day (3)
Ticker Exchange: DTN (254) => 254
Flags: x00000000: 
Description: NASDAQ 3RD MKT ISSUES TICKS TOTAL 
Last: 446, frac: 0, pure7 val: 446
Open: 146, frac: 0, pure7 val: 146
High: 446, frac: 0, pure7 val: 446
Low: 146, frac: 0, pure7 val: 146
Close: 446, frac: 0, pure7 val: 446
Change: 0, frac: 0, pure7 val: --- 
Market Status: x01
Last Trade Time: 16:05:00.000
Last Trade Date: 10/4/2012
Open Time: 9:31:00.000
Close Time: 16:05:00.000
 
MTR_MKTSTAT, Length: 104, Timestamp: 3:30:00.000
MsgKey: statId: @ | type: T | basis: R | exchgId: O | modifier: T
Symbol: @TROT, hash 4F525415
QS Symbol: @TROT, market 236
Security Type: MKTSTATS (8)
Symbol Type: Statistic.Share.Single.None
Session: US_Day (3)
Ticker Exchange: DTN (254) => 254
Flags: x00000000: 
Description: NASDAQ 3RD MKT TICKS RATIO              
Last: 119, frac: 2, pure7 val: 1.19
Open: 106, frac: 2, pure7 val: 1.06
High: 124, frac: 2, pure7 val: 1.24
Low: 75, frac: 2, pure7 val: 0.75
Close: 119, frac: 2, pure7 val: 1.19
Change: 0, frac: 2, pure7 val:  --- 
Market Status: x01
Last Trade Time: 16:05:00.000
Last Trade Date: 10/4/2012
Open Time: 9:31:00.000
Close Time: 16:05:00.000
 
MTR_MKTSTAT, Length: 104, Timestamp: 3:30:01.000
MsgKey: statId: @ | type: I | basis: P | exchgId: T | modifier: T
Symbol: @XXX, hash 54504915
QS Symbol: @XXXX, market 236
Security Type: MKTSTATS (8)
Symbol Type: Statistic.Share.Single.None
Session: US_Day (3)
Ticker Exchange: DTN (254) => 254
Flags: x00000010: LAST 
Description: 
Last: 1396, frac: 0, pure7 val: 1396
Open: 5, frac: 0, pure7 val: 5
High: 1396, frac: 0, pure7 val: 1396
Low: 3, frac: 0, pure7 val: 3
Close: 0, frac: 0, pure7 val: --- 
Change: 1396, frac: 0, pure7 val: 1396
Market Status: x02
Last Trade Time: 3:30:01.000
Last Trade Date: 10/5/2012
Open Time: 0:00:00.000
Close Time: 23:59:59.000


If you look at the 9th line of each message, there is Flags with a code like x00000000. But the first 2 messages have the same code so, I want to saperate only those messages that have a unique code in a text file.

Last edited by Scrutinizer; 11-20-2012 at 02:37 PM.. Reason: Wanted to put 3 example messages but accidently put only 2. Therefore added third message.; mod: code tags
Sponsored Links
    #6  
Old 11-20-2012
alister alister is offline
Registered User
 
Join Date: Dec 2009
Last Activity: 11 June 2014, 8:40 PM EDT
Posts: 3,231
Thanks: 179
Thanked 973 Times in 789 Posts
If you just want to filter out one type, and if messages are separated by at least one blank line:

Code:
awk 'index($9, "code_to_match")' RS=

Regards,
Alister
Sponsored Links
    #7  
Old 11-20-2012
itkamaraj's Avatar
itkamaraj itkamaraj is offline Forum Advisor  
^Kamaraj^
 
Join Date: Apr 2010
Last Activity: 3 October 2014, 8:50 AM EDT
Posts: 3,059
Thanks: 33
Thanked 659 Times in 638 Posts
The below code creates the file with flagname (with the appropriate message block)


Code:
 
 $ nawk '/MTR/,/Close Time/{a[i++]=$0;if($0~/Flags:/){sub(":","",$2);fname=$2}if($0~/Close Time/){for(j=0;j<=i;j++)print a[j] >> fname;i=0}}' input.txt

In Solaris, use nawk , otherwise you can use awk

Above code will create two files ( x00000000, x00000010 ) - for the sample text you posted.

Last edited by itkamaraj; 11-20-2012 at 11:36 PM..
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Problem passing arguments to a Python script kristinu Shell Programming and Scripting 0 01-18-2011 04:58 PM
Passing variable from shell script to python script shashi792 Shell Programming and Scripting 4 12-02-2010 05:20 AM
problem with python on mac thefloydpink UNIX for Dummies Questions & Answers 0 02-11-2009 11:19 AM
Python tkinter grid problem? SRCoder! Shell Programming and Scripting 0 06-19-2004 11:49 AM



All times are GMT -4. The time now is 01:12 AM.