How to parse through a file and based on condition form another output file


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users How to parse through a file and based on condition form another output file
# 1  
Old 02-20-2008
Question How to parse through a file and based on condition form another output file

I have one file say CM.txt which contains values like below.Its just a flat file

1000,A,X
1001,B,Y
1002,B,Z
...
..

total around 4 million lines of entries will be in that file.
Now i need to write another file CM1.txt which should have

1000,1
1001,2
1002,3
....
...
..

Here i am putting 1,2,3 based on condition in first file
if A + X then 1
if B+Y then 2
if B+Z then 3
These are the only three conditions that is possible.

Now can anybody suggest a way to write a Unix script which do this task a shortest possible time,since it has huge number of entries in the file.
Please give me a best solution / example that will help me .

Last edited by sivasu.india; 02-20-2008 at 11:18 AM..
# 2  
Old 02-20-2008
Question do other combinations exist?

what about
A+Y
A+Z
B+X

a 2 by 3 matrix can have six ouputs
so, does the solution need to figure six or just your intitial three possibilities?
# 3  
Old 02-20-2008
only 3 possibilities

Only three possibilities as mentioned above by me
# 4  
Old 02-20-2008
Try this:

sed -e 's/A,X/1/g' -e 's/B,Y/2/g' -e 's/B,Z/3/g' CM.txt

you can put this in a file.sed and run it like so:

sed -f sedfile file


Using sed
# 5  
Old 02-20-2008
Expanding on what nj78 said:

Code:
sed 's/A,X/1/;s/B,Y/2/;s/B,Z/3/' CM.txt > CM1.txt

1. Performance considerations

I suppose awk and sed to be about the same speed, so doing the same in awk will probably gain (or loose) some seconds per run. We found that out when dealing with other problems involving very large files here that sed's and awk's work speeds are about level and way ahead of the crowd. Other probable solutions like perl, python, shell scripts, what else, ... will be considerably slower, perhaps the slowest being shell script - it is simply not built for digesting huge loads of data.

For a discussion of this have a look here for instance.

2. Security considerations

I in your place would not be so sure about what can be and what can't - strange things happen all the time. ;-)) In your case i would not rely on your knowledge that only three combinations are possible. Even if that means a slight performance degradation i would write the sed script that way:

Code:
sed 's/A,X/1/;s/B,Y/2/;s/B,Z/3/;s/,[^123].*$/ERROR/' CM.txt > CM1.txt

The last clause does the following: all your "legal" lines are by now changed and end in either "1", "2" or "3". If a line now ends in something else ([^123] means: "neither 1 nor 2 nor 3") it was not changed by the 3 clauses before and and is therefore not a legal (expected) combination, which i flag with a big "ERROR", so it will easily be found. In a second step i would invest the time to validate my output:

Code:
if [ $(grep -c "ERROR" CM1.txt) -eq 0 ] ; then 
     print - "File passed check, everything is ok"
else
     print - "Something has gone wrong, check your file"
fi

Bottom line: it is not a problem to deal with errors as long as you are prepared for it.

I hope this helps.

bakunin
# 6  
Old 02-20-2008
Excellent addition of the error checking.
# 7  
Old 02-28-2008
Bug thnks

Its great and easy.

thanks for your reply.
SmilieSmilieSmilieSmilieSmilieSmilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Manipulate condition to send mail based on output text in file

Hi All, I have a working script as below. echo "Files loaded with $(cat /var/tmp/script.X1.out)" | mail -s "Files loaded with return code" mailid This script takes the output from script.X1.out file and appends the text "Files loaded with return code" and sends the email. Now what I want... (5 Replies)
Discussion started by: midhun3108
5 Replies

2. Shell Programming and Scripting

Adding data from a file based on some condition

I collect data in a file in below format(Month Day Year Size) in RedHat Linux. Now i want to calculate the data size date wise. As i code shell script after long time, i forgot the features and syntax. Can anyone help me regard this please. Feb 8 2014 15 Feb 10 2014 32 Feb 10 2014 32 Feb 12... (2 Replies)
Discussion started by: makauser
2 Replies

3. Shell Programming and Scripting

Getting data from a flat file based on condition

Hi, I have a flaty file from which i am fetching few columns in tablular form as below code. Now i want to fetch the column 6 and 7 in below code only if it either of them is non zero.However below startement awk -F, '$6==0 && $7==0{exit 1}' ${IFILE} is not working..Not sure where is the... (36 Replies)
Discussion started by: Vivekit82
36 Replies

4. Shell Programming and Scripting

Displaying log file pattern output in tabular form output

Hi All, I have result log file which looks like this (below): from the content need to consolidate the result and put it in tabular form 1). Intercomponents Checking Passed: All Server are passed. ====================================================================== 2). OS version Checking... (9 Replies)
Discussion started by: Optimus81
9 Replies

5. Shell Programming and Scripting

Parse tab delimited file, check condition and delete row

I am fairly new to programming and trying to resolve this problem. I have the file like this. CHROM POS REF ALT 10_sample.bam 11_sample.bam 12_sample.bam 13_sample.bam 14_sample.bam 15_sample.bam 16_sample.bam tg93 77 T C T T T T T tg93 79 ... (4 Replies)
Discussion started by: empyrean
4 Replies

6. Shell Programming and Scripting

AWK Command parse a file based on string.

AWK Command parse a file based on string. I am trying to write a shell script to parse a file based on a string and move the content of the file to another file. Here is scenario. File content below Mime-Version: 1.0 Content-Type: multipart/mixed; ... (2 Replies)
Discussion started by: aakishore
2 Replies

7. Shell Programming and Scripting

How can I change file value based on condition

Hi, Gurus, I got a problem to resolve following issue: I have one file file1as following: start_dt=2010-01-01 12:00:02 start_dt=2011-01-01 09:00:02 start_dt=2009-01-01 11:00:02I have another file file2 as following: title1, 2010-01-03 10:00:02 title2, 2011-01-04 11:00:02 title3,... (5 Replies)
Discussion started by: ken002
5 Replies

8. Shell Programming and Scripting

Spliting file based on condition

Hi, I have a comma separated file with millions of records in it. I have a requirement to split the file based on the value in a one of the columns. Suppose i have a text file with columns like C1, C2,C3,C4 Column C4 can hold the values either 01 or 02 03 or 04. I nned to extract... (2 Replies)
Discussion started by: Raamc
2 Replies

9. Shell Programming and Scripting

Moving file to directory based on condition.

Can any one help me to correct following script. I have 2 directories DropZone and ProcessZone. File pattern is *VEHDESCSUM*. Finding the 'no of files' in DropZone directory using ls *VEHDESCSUM* |wc -l If DropZone has more than one file or 0 files then exit 1 If DropZone has one file then... (2 Replies)
Discussion started by: ramanagh
2 Replies

10. Shell Programming and Scripting

Read file based on condition

Hi Friends, Can any one help with this: I have a huge file with the format as A SAM 4637 B DEPT1 4758 MILAN A SMITH 46585 B DEPT2 5385 HARRYIS B SAMUL 63547 GEORGE B DANIEL 899 BOISE A FRES 736 74638 I have to read this file and write only the records that starts with "B" only ... (5 Replies)
Discussion started by: sbasetty
5 Replies
Login or Register to Ask a Question