Adding a delimiter to a text file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Adding a delimiter to a text file
# 1  
Old 03-05-2009
Adding a delimiter to a text file

Im writing a KSH script to read a simple text file and add a delimiter. Ive written the following script but it runs very slow. I initially used the cut command to substring the input record then switched to this version using awk to substring... both run too slow. Any ideas how to make this more efficient? Its taking around 10min for a 40k record file(2 meg)... very slow for such small data.

data file(input.txt):

99919xxxxxxxZ3700162111
99921xxxxxxxZ3300563111
99922xxxxxxxZ5800143111
99923xxxxxxxZ5500056111
99925xxxxxxxZ5100366111

script:

#!/bin/ksh
while read x
do
f1=$(echo "$x"|awk '{print substr($0,1,5)}')
f2=$(echo "$x"|awk '{print substr($0,6,4)}')
f3=$(echo "$x"|awk '{print substr($0,10,2)}')
f4=$(echo "$x"|awk '{print substr($0,12,3)}')
f5=$(echo "$x"|awk '{print substr($0,15,1)}')
f6=$(echo "$x"|awk '{print substr($0,16,2)}')
f7=$(echo "$x"|awk '{print substr($0,18,5)}')
f8=$(echo "$x"|awk '{print substr($0,23,1)}')
f9=$(echo "$x"|awk '{print substr($0,24,1)}')
f10=$(echo "$x"|awk '{print substr($0,25,1)}')

delimrecord="$f1|$f2|$f3|$f4|$f5|$f6|$f7|$f8|$f9|$f10"
echo $delimrecord >> output.txt
done < input.txt
# 2  
Old 03-05-2009
You invoke an external program 10 times for every record - that's why it is slow

try something like this:
Code:
awk ' {
printf("%s|%s|%s|%s|%s|%s|%s|%s|%s|%s\n",
      substr($0,1,5) ,
      substr($0,6,4) ,
      substr($0,10,2),
      substr($0,12,3),
      substr($0,15,1),
      substr($0,16,2),
      substr($0,18,5),
      substr($0,23,1),
      substr($0,24,1),
      substr($0,25,1) ) } ' infile.txt > outfile.txt

# 3  
Old 03-06-2009
Thanks Jim. Thats alot more efficient than the mess I created!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to target certain delimiter to split text file?

Hi, all. I have an input file. I would like to generate 3 types of output files. Input: LG10_PM_map_19_LEnd_1000560 LG10_PM_map_6-1_27101856 LG10_PM_map_71_REnd_20597718 LG12_PM_map_5_chr_118419232 LG13_PM_map_121_24341052 LG14_PM_1a_456799 LG1_MM_scf_5a_opt_abc_9029993 ... (5 Replies)
Discussion started by: huiyee1
5 Replies

2. Shell Programming and Scripting

Splitting records in a text file based on delimiter

A text file has 2 fields (Data, Filename) delimited by # as below, Data,Filename Row1 -> abc#Test1.xml Row2 -> xyz#Test2.xml Row3 -> ghi#Test3.xml The content in first field has to be written into a file where filename should be considered from second field. So from... (4 Replies)
Discussion started by: jayakkannan
4 Replies

3. Shell Programming and Scripting

Shell script to put delimiter for a no delimiter variable length text file

Hi, I have a No Delimiter variable length text file with following schema - Column Name Data length Firstname 5 Lastname 5 age 3 phoneno1 10 phoneno2 10 phoneno3 10 sample data - ... (16 Replies)
Discussion started by: Gaurav Martha
16 Replies

4. Shell Programming and Scripting

Adding a delimiter to a variable length file

Hi, I'm new to unix, i have a variable length file like below, 01|Test|Test1|Sample| 02|AA|BB|CC|DD| 03|AAA|BBB|CCC|DDD|EEE|RRR|TTT|SSS|YYY| I need to make this as a fixed length file. Assume that i have 10 columns in the DAT file. for ex: the first 01 record is having 4cols -... (8 Replies)
Discussion started by: Mohankumar Venu
8 Replies

5. Shell Programming and Scripting

[bash help]Adding multiple lines of text into a specific spot into a text file

I am attempting to insert multiple lines of text into a specific place in a text file based on the lines above or below it. For example, Here is a portion of a zone file. IN NS ns1.domain.tld. IN NS ns2.domain.tld. IN ... (2 Replies)
Discussion started by: cdn_humbucker
2 Replies

6. UNIX for Dummies Questions & Answers

extract fields from text file using delimiter!!

Hi All, I am new to unix scripting, please help me in solving this assignment.. I have a scenario, as follows: 1. i have a text file(read1.txt) with the following data sairam,123 kamal,122 etc.. 2. I have to write a unix... (6 Replies)
Discussion started by: G.K.K
6 Replies

7. Shell Programming and Scripting

Adding specific text and spaces to each line in a text file

Hi, I wanted to add specific text to each row in a text file containing three rows. Example: 0 8 7 6 5 5 7 8 9 0 7 9 7 8 9 0 1 2 And I want to add a 21 at the beginning of the first row, and blank spaces at the beginning of the second two rows. To get this: 21 0 8 7 6 5 5 7 8... (4 Replies)
Discussion started by: hertingm
4 Replies

8. Shell Programming and Scripting

Formatting a text file based on newline and delimiter characters

Hi Everybody, I need some help on formatting the files coming into unix box on the fly. I get a file some thing like this in a single line. ISA^M00^M ^M00^M ^M14^M006929681900 ^M01^M095449419 ... (5 Replies)
Discussion started by: ntekupal
5 Replies

9. Shell Programming and Scripting

adding delimiter to a fixed width file

Hi , I have a file : CSCH74000.00 CSCH74000.00 CSCH74100.00 CSCH74000.00 CSCH74100.00 CSCH74000.00 CSCH74000.00 CSCH74100.00 CSCH74100.00 CSCH74100.00 I have to put a delimiter( say comma) in between after 6th character: CSCH74,000.00 CSCH74,000.00 CSCH74,100.00 (2 Replies)
Discussion started by: sumeet
2 Replies

10. Shell Programming and Scripting

How to parse a text file with \034 as field and \035 as end of message delimiter?

I need some tips to write a unix korn shell script that will parse an input text file. Input text file has messages that span several lines, each field in the message is delimited by /034 and the end of message is delimited by /035. Input file looks something similar to ... (1 Reply)
Discussion started by: indianya
1 Replies
Login or Register to Ask a Question