Sponsored Content
Top Forums Shell Programming and Scripting Need a Python Script to filter huge files Post 302736557 by Vijeta Laad on Tuesday 27th of November 2012 01:15:12 PM
Old 11-27-2012
Need a Python Script to filter huge files

I work on various messages received from server and want to write a python script that can sort messages with unique flag values and give me the output in a text file.

I get these messages in the form of .zcap file from server and I use an internal tool to filter:

Step 1) Zcap file to get security types,
Step 2) then filter each security types file that generate various exchanges files
Steps 3) lastly filter each exchange files to get message types files.

For eg: I filtered a .zcap file to get message types as -
BOND-----CVE----- MTR_BOND, MTFD_BOND, MTQ_BQUOTE, MTQ_MBBOQUOTE
BOND-----NYSE---- MTR_BOND, MTFD_BOND, MTQ_BQUOTE
BOND-----TSE--- MTR_BOND, MTFD_BOND, MTA_RECAP, MTT_TRADE, MTT_STATUS, MTA_CLOSE

Step 4) at present, I run a UNIX command on each message type that generates multiple text files of each unique flag value messages (for eg; MTR_BOND).
$ awk '/MTQ/,/Quote Condition/{a[i++]=$0;if($0~/Flags:/){sub(":","",$2);fname=$2}if($0~/Quote Condition/){for(j=0;j<=i;j++)print a[j] > fname;i=0}}' MTA.txt

This is a very tedious job to run this command on each message type file (for eg; MTR_BOND) level, I want a python script that I run at Security Type level (for eg; BOND) .
Here I attach text file for BOND and its exchanges and message types. Please write a python script (preferably) to generate 1 text file for each message types with only 1 message of unique flag value.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell Script for searching files with date as filter

Hi , Assume today's date is 10-May-2002. I want to get a list of files which were last modified since 01-May-2002. If I run the script after 5 days, it should still list me the files modified from 01-May-2002 till today. I also plan to pass the date 01-May-2002 as an argument to the shell script... (3 Replies)
Discussion started by: kanakaraj_s
3 Replies

2. Shell Programming and Scripting

need help--script to filter specific lines from multiple txt files

Hi folks, - I have 800 txt files - those files are cisco router configs router1.txt router2.txt ... router800.txt I want to accomplish the following: - I want to have a seperate file with all the filenames that I want to process - I want a script that goes trough all those... (7 Replies)
Discussion started by: I-1
7 Replies

3. UNIX for Dummies Questions & Answers

Difference between two huge files

Hi, As per my requirement, I need to take difference between two big files(around 6.5 GB) and get the difference to a output file without any line numbers or '<' or '>' in front of each new line. As DIFF command wont work for big files, i tried to use BDIFF instead. I am getting incorrect... (13 Replies)
Discussion started by: pyaranoid
13 Replies

4. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

5. Shell Programming and Scripting

Problem running Perl Script with huge data files

Hello Everyone, I have a perl script that reads two types of data files (txt and XML). These data files are huge and large in number. I am using something like this : foreach my $t (@text) { open TEXT, $t or die "Cannot open $t for reading: $!\n"; while(my $line=<TEXT>){ ... (4 Replies)
Discussion started by: ad23
4 Replies

6. Shell Programming and Scripting

Bash script with python slicing on multiple data files

I have 2 files generated in linux that has common output and were produced across multiple hosts with the same setup/configs. These files do some simple reporting on resource allocation and user sessions. So, essentially, say, 10 hosts, with the same (2) system reporting in the files, so a... (0 Replies)
Discussion started by: jdubbz
0 Replies

7. Shell Programming and Scripting

Aggregation of Huge files

Hi Friends !! I am facing a hash total issue while performing over a set of files of huge volume: Command used: tail -n +2 <File_Name> |nawk -F"|" -v '%.2f' qq='"' '{gsub(qq,"");sa+=($156<0)?-$156:$156}END{print sa}' OFMT='%.5f' Pipe delimited file and 156 column is for hash totalling.... (14 Replies)
Discussion started by: Ravichander
14 Replies

8. UNIX for Dummies Questions & Answers

Filter records in a huge text file from a filter text file

Hi Folks, I have a text file with lots of rows with duplicates in the first column, i want to filter out records based on filter columns in a different filter text file. bash scripting is what i need. Data.txt Name OrderID Quantity Sam 123 300 Jay 342 498 Kev 78 2500 Sam 420 50 Vic 10... (3 Replies)
Discussion started by: tech_frk
3 Replies

9. Programming

Python script for extracting data using two files

Hello, I have two files. File 1 is a list of interested IDs Ex1 Ex2 Ex3File 2 is the original file with over 8000 columns and 20 millions rows and is a compressed file .gz Ex1 xx xx xx xx .... Ex2 xx xx xx xx .... Ex2 xx xx xx xx ....Now I need to extract the information for all the IDs of... (4 Replies)
Discussion started by: nans
4 Replies

10. Windows & DOS: Issues & Discussions

How to execute python script on remote with python way..?

Hi all, I am trying to run below python code for connecting remote windows machine from unix to run an python file exist on that remote windows machine.. Below is the code I am trying: #!/usr/bin/env python import wmi c = wmi.WMI("xxxxx", user="xxxx", password="xxxxxxx")... (1 Reply)
Discussion started by: onenessboy
1 Replies
rc(1M)																	    rc(1M)

NAME
rc - general purpose sequencer invoked upon entering new run level SYNOPSIS
DESCRIPTION
The shell script is the general sequencer invoked upon entering a new run level via the command (where N equals 0-6). The script is typi- cally invoked by the corresponding entry in the file as follows: is the startup and shutdown sequencer script. There is only one sequencer script and it handles all of the sequencer directories. This script sequences the scripts in the appropriate sequencer directories in alphabetical order as defined by the shell and invokes them as either startup or kill scripts. If a transition from a lower to a higher run level (i.e., init state) occurs, the start scripts for the new run level and all intermediate levels between the old and new level are executed. If a transition from a higher to a lower run level occurs, the kill scripts for the new run level and all intermediate levels between the old and new level are executed. If a start script link (e.g., in sequencer N has a stop action, the corresponding kill script should be placed in sequencer (e.g., Actions started in level N should be stopped in level This way, a system shutdown (e.g., transition from level 3 directly to level 0) will result in all subsystems being stopped. Start and Kill Scripts In many cases, a startup script will have both a start and a kill action. For example, the inetd script starts the Internet daemon in the start case, and kills that process in the stop case. Instead of two separate scripts, only one exists, which accepts both the and argu- ments and executes the correct code. In some cases, only a start action will be applicable. If this is the case, and if the action is specified, the script should produce a usage message and exit with an error. In general, scripts should look at their arguments and pro- duce error messages if bad arguments are present. When a script executes properly, it must exit with a return value of zero. If an error condition exists, the return value must be nonzero. Naming Conventions The startup and shutdown scripts (referred to as startup scripts hereafter) exist in the directory, named after the subsystem they control. For example, the script controls starting up the daemon. The contents of sequencer directories consist of symbolic links to startup scripts in These symbolic links must follow a strict naming convention, as noted in the various fields of this example: where the fields are defined as follows: The sequencer directory is numbered to reflect the run level for which its contents will be executed. In this case, start scripts in this directory will be executed upon entering run level 2 from run level 1, and kill scripts will be executed upon entering run level 2 from run level 3. The first character of a sequencer link name determines whether the script is executed as a start script (if the character is or as a kill script (if the character is A three digit number is used for sequencing scripts within the sequencer directory. Scripts are executed by type (start or kill) in alphabetical order as defined by the shell. Although it is not recommended, two scripts may share the same sequence number. The name of the startup script follows the sequence number. The startup script name must be the same name as the script to which this sequencer entry is linked. In this exam- ple, the link points to Note that short file name systems require file names of 14 or less characters. This means that the fourth field is limited to 10 or fewer characters. Scripts are executed in alphabetical order. The entire file name of the script is used for alphabetical ordering purposes. When ordering start and kill script links, note that subsystems started in any given order should be stopped in the reverse order to eliminate any dependencies between subsystems. This means that kill scripts will generally not have the same numbers as their start script counterparts. For example, if two subsystems must be started in a given order due to dependencies (e.g., followed by the kill counterparts to these scripts must be numbered so that the subsystems are stopped in the opposite order in which they were started (e.g., followed by Also keep in mind that kill scripts for a start script in directory will reside in For example, and might be start/kill counterparts. Arguments The startup/shutdown scripts should be able to recognize the following four arguments (where applicable): The argument is passed to scripts whose names start with Upon receiving the argument, the script should perform its start actions. The argument is passed to scripts whose names start with Upon receiving the argument, the script should perform its stop actions. The argument is passed to scripts whose names start with so that the script can report back a short message indicating what the start action will do. For instance, when the spooler script is invoked with a argument, it echoes This string is used by the startup routines. Scripts given just the argument will only print a message and not per- form any actions. The argument is passed to scripts whose names start with so that the script can report back a short message indicating what the stop action will do. For instance, when the spooler script is invoked with a argument, it echoes This string is used by the shutdown checklist. Scripts given just the argument will only print a message and not perform any actions. Script Output To ensure proper reporting of startup events, startup scripts are required to comply with the following guidelines for script output. o Status messages, such as must be directed to stdout. All error messages must be directed to stderr. o Script output, both stdout and stderr, is redirected to log file unless the startup checklist mode is set to the raw mode. In this case, all output goes to the console. All error messages should be echoed to stdout or stderr. o Startup scripts are not allowed to send messages directly to the console, or to start any daemons that immediately write to the console. This restriction exists because these scripts are now started by the checklist wrapper. All script output should go to either stdout or stderr, and thus be captured in a log file. Any console output will be garbled. o When a startup script returns an exit code of can display a specific message on the console prior to rebooting the system. This is achieved by creating a text file named containing the text to be displayed to the console. Note that deletes this file after displaying the message, so startup scripts need to write this file each time a specific message is required to be displayed on console prior to reboot. RETURN VALUE
The return values for startup scripts are as follows: Script exited without error. Script encountered errors. Script was skipped due to overriding control variables from files, or for other reasons, and did not actually do anything. Script will automatically reboot the system. Script exited without error and started a process in background mode. For return values greater than the action is same as return value script encountered errors. SEE ALSO
init(1M), shutdown(1M), inittab(4), rc.config(4). rc(1M)
All times are GMT -4. The time now is 06:26 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy