Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: How to parse through a file and based on condition form another output file
Top Forums UNIX for Advanced & Expert Users How to parse through a file and based on condition form another output file Post 302169147 by bakunin on Wednesday 20th of February 2008 12:19:25 PM
Old 02-20-2008
Expanding on what nj78 said:

Code:
sed 's/A,X/1/;s/B,Y/2/;s/B,Z/3/' CM.txt > CM1.txt

1. Performance considerations

I suppose awk and sed to be about the same speed, so doing the same in awk will probably gain (or loose) some seconds per run. We found that out when dealing with other problems involving very large files here that sed's and awk's work speeds are about level and way ahead of the crowd. Other probable solutions like perl, python, shell scripts, what else, ... will be considerably slower, perhaps the slowest being shell script - it is simply not built for digesting huge loads of data.

For a discussion of this have a look here for instance.

2. Security considerations

I in your place would not be so sure about what can be and what can't - strange things happen all the time. ;-)) In your case i would not rely on your knowledge that only three combinations are possible. Even if that means a slight performance degradation i would write the sed script that way:

Code:
sed 's/A,X/1/;s/B,Y/2/;s/B,Z/3/;s/,[^123].*$/ERROR/' CM.txt > CM1.txt

The last clause does the following: all your "legal" lines are by now changed and end in either "1", "2" or "3". If a line now ends in something else ([^123] means: "neither 1 nor 2 nor 3") it was not changed by the 3 clauses before and and is therefore not a legal (expected) combination, which i flag with a big "ERROR", so it will easily be found. In a second step i would invest the time to validate my output:

Code:
if [ $(grep -c "ERROR" CM1.txt) -eq 0 ] ; then 
     print - "File passed check, everything is ok"
else
     print - "Something has gone wrong, check your file"
fi

Bottom line: it is not a problem to deal with errors as long as you are prepared for it.

I hope this helps.

bakunin
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read file based on condition

Hi Friends, Can any one help with this: I have a huge file with the format as A SAM 4637 B DEPT1 4758 MILAN A SMITH 46585 B DEPT2 5385 HARRYIS B SAMUL 63547 GEORGE B DANIEL 899 BOISE A FRES 736 74638 I have to read this file and write only the records that starts with "B" only ... (5 Replies)
Discussion started by: sbasetty
5 Replies

2. Shell Programming and Scripting

Moving file to directory based on condition.

Can any one help me to correct following script. I have 2 directories DropZone and ProcessZone. File pattern is *VEHDESCSUM*. Finding the 'no of files' in DropZone directory using ls *VEHDESCSUM* |wc -l If DropZone has more than one file or 0 files then exit 1 If DropZone has one file then... (2 Replies)
Discussion started by: ramanagh
2 Replies

3. Shell Programming and Scripting

Spliting file based on condition

Hi, I have a comma separated file with millions of records in it. I have a requirement to split the file based on the value in a one of the columns. Suppose i have a text file with columns like C1, C2,C3,C4 Column C4 can hold the values either 01 or 02 03 or 04. I nned to extract... (2 Replies)
Discussion started by: Raamc
2 Replies

4. Shell Programming and Scripting

How can I change file value based on condition

Hi, Gurus, I got a problem to resolve following issue: I have one file file1as following: start_dt=2010-01-01 12:00:02 start_dt=2011-01-01 09:00:02 start_dt=2009-01-01 11:00:02I have another file file2 as following: title1, 2010-01-03 10:00:02 title2, 2011-01-04 11:00:02 title3,... (5 Replies)
Discussion started by: ken002
5 Replies

5. Shell Programming and Scripting

AWK Command parse a file based on string.

AWK Command parse a file based on string. I am trying to write a shell script to parse a file based on a string and move the content of the file to another file. Here is scenario. File content below Mime-Version: 1.0 Content-Type: multipart/mixed; ... (2 Replies)
Discussion started by: aakishore
2 Replies

6. Shell Programming and Scripting

Parse tab delimited file, check condition and delete row

I am fairly new to programming and trying to resolve this problem. I have the file like this. CHROM POS REF ALT 10_sample.bam 11_sample.bam 12_sample.bam 13_sample.bam 14_sample.bam 15_sample.bam 16_sample.bam tg93 77 T C T T T T T tg93 79 ... (4 Replies)
Discussion started by: empyrean
4 Replies

7. Shell Programming and Scripting

Displaying log file pattern output in tabular form output

Hi All, I have result log file which looks like this (below): from the content need to consolidate the result and put it in tabular form 1). Intercomponents Checking Passed: All Server are passed. ====================================================================== 2). OS version Checking... (9 Replies)
Discussion started by: Optimus81
9 Replies

8. Shell Programming and Scripting

Getting data from a flat file based on condition

Hi, I have a flaty file from which i am fetching few columns in tablular form as below code. Now i want to fetch the column 6 and 7 in below code only if it either of them is non zero.However below startement awk -F, '$6==0 && $7==0{exit 1}' ${IFILE} is not working..Not sure where is the... (36 Replies)
Discussion started by: Vivekit82
36 Replies

9. Shell Programming and Scripting

Adding data from a file based on some condition

I collect data in a file in below format(Month Day Year Size) in RedHat Linux. Now i want to calculate the data size date wise. As i code shell script after long time, i forgot the features and syntax. Can anyone help me regard this please. Feb 8 2014 15 Feb 10 2014 32 Feb 10 2014 32 Feb 12... (2 Replies)
Discussion started by: makauser
2 Replies

10. UNIX for Beginners Questions & Answers

Manipulate condition to send mail based on output text in file

Hi All, I have a working script as below. echo "Files loaded with $(cat /var/tmp/script.X1.out)" | mail -s "Files loaded with return code" mailid This script takes the output from script.X1.out file and appends the text "Files loaded with return code" and sends the email. Now what I want... (5 Replies)
Discussion started by: midhun3108
5 Replies

LEARN ABOUT REDHAT

mac2unix

dos2unix(1)						      General Commands Manual						       dos2unix(1)

NAME

       dos2unix - DOS/MAC to UNIX text file format converter

SYNOPSYS

       dos2unix [options] [-c convmode] [-o file ...] [-n infile outfile ...]

       Options:

       [-hkqV] [--help] [--keepdate] [--quiet] [--version]

DESCRIPTION

       This manual page documents dos2unix, the program that converts plain text files in DOS/MAC format to UNIX format.

OPTIONS

       The following options are available:

       -h --help
	      Print online help.

       -k --keepdate
	      Keep the date stamp of output file same as input file.

       -q --quiet
	      Quiet mode. Suppress all warning and messages.

       -V --version
	      Prints version information.

       -c --convmode convmode
	      Sets conversion mode. Simulates dos2unix under SunOS.

       -o --oldfile file ...
	      Old file mode. Convert the file and write output to it. The program default to run in this mode. Wildcard names may be used.

       -n --newfile infile outfile ...
	      New  file  mode.	Convert the infile and write output to outfile. File names must be given in pairs and wildcard names should NOT be
	      used or you WILL lost your files.

EXAMPLES

       Get input from stdin and write output to stdout.

	      dos2unix

       Convert and replace a.txt. Convert and replace b.txt.

	      dos2unix a.txt b.txt

	      dos2unix -o a.txt b.txt

       Convert and replace a.txt in ASCII conversion mode.  Convert and replace b.txt in ISO conversion mode.  Convert	c.txt  from  Mac  to  Unix
       ascii format.

	      dos2unix a.txt -c iso b.txt

	      dos2unix -c ascii a.txt -c iso b.txt

	      dos2unix -c mac a.txt b.txt

       Convert and replace a.txt while keeping original date stamp.

	      dos2unix -k a.txt

	      dos2unix -k -o a.txt

       Convert a.txt and write to e.txt.

	      dos2unix -n a.txt e.txt

       Convert a.txt and write to e.txt, keep date stamp of e.txt same as a.txt.

	      dos2unix -k -n a.txt e.txt

       Convert and replace a.txt. Convert b.txt and write to e.txt.

	      dos2unix a.txt -n b.txt e.txt

	      dos2unix -o a.txt -n b.txt e.txt

       Convert c.txt and write to e.txt. Convert and replace a.txt.  Convert and replace b.txt. Convert d.txt and write to f.txt.

	      dos2unix -n c.txt e.txt -o a.txt b.txt -n d.txt f.txt

DIAGNOSTICS

BUGS

       The program does not work properly under MSDOS in stdio processing mode.  If you know why is that so, please tell me.

AUTHORS

       Benjamin Lin - <blin@socs.uts.edu.au>

       Bernd Johannes Wuebben (mac2unix mode) <wuebben@kde.org>

MISCELLANY

       Tested environment:

	      Linux 1.2.0 with GNU C 2.5.8

	      SunOS 4.1.3 with GNU C 2.6.3

	      MS-DOS 6.20 with Borland C++ 4.02

       Suggestions and bug reports are welcome.

SEE ALSO

       unix2dos(1) mac2unix(1)

1995.03.31							   dos2unix v3.0						       dos2unix(1)
All times are GMT -4. The time now is 09:03 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy