How to parse through a file and based on condition form another output file
I have one file say CM.txt which contains values like below.Its just a flat file
1000,A,X
1001,B,Y
1002,B,Z
...
..
total around 4 million lines of entries will be in that file.
Now i need to write another file CM1.txt which should have
1000,1
1001,2
1002,3
....
...
..
Here i am putting 1,2,3 based on condition in first file
if A + X then 1
if B+Y then 2
if B+Z then 3
These are the only three conditions that is possible.
Now can anybody suggest a way to write a Unix script which do this task a shortest possible time,since it has huge number of entries in the file.
Please give me a best solution / example that will help me .
Last edited by sivasu.india; 02-20-2008 at 11:18 AM..
I suppose awk and sed to be about the same speed, so doing the same in awk will probably gain (or loose) some seconds per run. We found that out when dealing with other problems involving very large files here that sed's and awk's work speeds are about level and way ahead of the crowd. Other probable solutions like perl, python, shell scripts, what else, ... will be considerably slower, perhaps the slowest being shell script - it is simply not built for digesting huge loads of data.
For a discussion of this have a look here for instance.
2. Security considerations
I in your place would not be so sure about what can be and what can't - strange things happen all the time. ;-)) In your case i would not rely on your knowledge that only three combinations are possible. Even if that means a slight performance degradation i would write the sed script that way:
The last clause does the following: all your "legal" lines are by now changed and end in either "1", "2" or "3". If a line now ends in something else ([^123] means: "neither 1 nor 2 nor 3") it was not changed by the 3 clauses before and and is therefore not a legal (expected) combination, which i flag with a big "ERROR", so it will easily be found. In a second step i would invest the time to validate my output:
Bottom line: it is not a problem to deal with errors as long as you are prepared for it.
Hi All,
I have a working script as below.
echo "Files loaded with $(cat /var/tmp/script.X1.out)" | mail -s "Files loaded with return code" mailid
This script takes the output from script.X1.out file and appends the text "Files loaded with return code" and sends the email. Now what I want... (5 Replies)
I collect data in a file in below format(Month Day Year Size) in RedHat Linux. Now i want to calculate the data size date wise. As i code shell script after long time, i forgot the features and syntax. Can anyone help me regard this please.
Feb 8 2014 15
Feb 10 2014 32
Feb 10 2014 32
Feb 12... (2 Replies)
Hi,
I have a flaty file from which i am fetching few columns in tablular form as below code.
Now i want to fetch the column 6 and 7 in below code only if it either of them is non zero.However below startement
awk -F, '$6==0 && $7==0{exit 1}' ${IFILE}
is not working..Not sure where is the... (36 Replies)
Hi All,
I have result log file which looks like this (below): from the content need to consolidate the result and put it in tabular form
1). Intercomponents Checking
Passed: All Server are passed.
======================================================================
2). OS version Checking... (9 Replies)
I am fairly new to programming and trying to resolve this problem. I have the file like this.
CHROM POS REF ALT 10_sample.bam 11_sample.bam 12_sample.bam 13_sample.bam 14_sample.bam 15_sample.bam 16_sample.bam
tg93 77 T C T T T T T
tg93 79 ... (4 Replies)
AWK Command parse a file based on string.
I am trying to write a shell script to parse a file based on a string and move the content of the file to another file.
Here is scenario.
File content below
Mime-Version: 1.0
Content-Type: multipart/mixed;
... (2 Replies)
Hi, Gurus,
I got a problem to resolve following issue:
I have one file file1as following:
start_dt=2010-01-01 12:00:02
start_dt=2011-01-01 09:00:02
start_dt=2009-01-01 11:00:02I have another file file2 as following:
title1, 2010-01-03 10:00:02
title2, 2011-01-04 11:00:02
title3,... (5 Replies)
Hi,
I have a comma separated file with millions of records in it.
I have a requirement to split the file based on the value in a one of the columns.
Suppose i have a text file with columns like C1, C2,C3,C4
Column C4 can hold the values either 01 or 02 03 or 04.
I nned to extract... (2 Replies)
Can any one help me to correct following script.
I have 2 directories DropZone and ProcessZone. File pattern is *VEHDESCSUM*.
Finding the 'no of files' in DropZone directory using ls *VEHDESCSUM* |wc -l
If DropZone has more than one file or 0 files then exit 1
If DropZone has one file then... (2 Replies)
Hi Friends,
Can any one help with this:
I have a huge file with the format as
A SAM 4637
B DEPT1 4758 MILAN
A SMITH 46585
B DEPT2 5385 HARRYIS
B SAMUL 63547 GEORGE
B DANIEL 899 BOISE
A FRES 736 74638
I have to read this file and write only the records that starts with "B" only
... (5 Replies)