Merging lines - Tuning the script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merging lines - Tuning the script
# 1  
Old 03-04-2008
Merging lines - Tuning the script

Hi, Anybody can tune the below script? Its taking 1hour to process 7000 lines.

addendaRecInd=7
entryDetailRecord=6
hdrDetailRecord=1
companyHdrRecord=5
companyCtrlRecord=8
trlRecord=9
recLength=94


FormatDone=false
trlCompleted=false

prevEntryCalc=false
batchCnt=0
hdrFound=false
while read line
do

# Find the Line #
grep -n "$line" $srcFile |cut -d : -f1 |read lineno
#find the Rec Type
if [ `echo $line|grep "^$companyHdrRecord"| wc -l` -ne 0 ]
then
hdrFound="true"
val=""
typeset -L94 line
hdr="$line"
typeset -R0 line
typeset -R0 batchCnt
fi
if [ `echo $line|grep "^$hdrDetailRecord"| wc -l` -ne 0 ]
then
typeset -L94 line
value=""
echo "$line" >> $destFile
line=""
typeset -R0 line
fi
if [ `echo $line|grep "^$trlRecord"| wc -l` -ne 0 -a $trlCompleted = false ]
then
echo $trlCompleted
value=""
typeset -L94 line
echo "$line" >> $destFile
line=""
typeset -R0 line
trlCompleted=true
echo $trlCompleted
fi
if [ `echo $line|grep "^$companyCtrlRecord"| wc -l` -ne 0 ]
then
if [ "$hdrFound" != "true" ]
then
echo "hdrFound" $hdrFound
echo "The last line is " $value
if [ "$value" != " " ]
then
typeset -L599 value
typeset -L6 batchCnt
echo "$value$batchCnt" >> $destFile
typeset -R0 value
typeset -R0 batchcnt
fi
typeset -L94 line
typeset -L6 batchcnt
echo "$line$batchCnt" >> $destFile
typeset -R0 line
typeset -R0 batchcnt
value=""
line=""
prevEntryCalc=false
fi
fi
if [ `echo $line|grep "^$entryDetailRecord"| wc -l` -ne 0 ]
then
# Print Header after the entry detail record has been read
if [ "$hdrFound" != "false" ]
then
batchCnt=`expr $batchCnt + 1 `
typeset -L94 hdr
typeset -L6 batchCnt
echo "$hdr$batchCnt" >> $destFile
typeset -R0 hdr
hdrFound="false"
hdr=""
fi
if [ "$prevEntryCalc" != "false" ]
then
typeset -L599 value
typeset -L6 batchCnt
echo "$value$batchCnt" >> $destFile
typeset -R0 value
typeset -R0 batchcnt
value=""
fi
line=`echo "$line"`
value=`echo "$line"`
prevEntryCalc="true"
fi
if [ `echo $line|grep "^$addendaRecInd" | wc -l` -ne 0 ]
then
typeset -L101 line
len=`echo $line|wc -c`
#echo "The Length of addenda record is "$len
value="$value$line"
typeset -R0 line
fi
done < $srcFile

Last edited by senthil_is; 03-04-2008 at 09:40 PM.. Reason: adding some more tips
# 2  
Old 03-04-2008
The example of the file format is i am using # for comment

101 10000217311 # Header always starts with 1
5200Paydonee 11 # Batch always Header Stats with 5
622100002173011 # Data always starts with 6
705CTB 4 0112 # If lin strts with 7, it has to be merged with previous line. and also the batch header counter has to be padded at end of line.
705Pymt Ref 4991
622100002173066
705CTB 19 5622
705532119P 386
622100002173011
705CTB 20 338
705 44888
822000000112998 # batch trailer always starts with 8
900001100021000 # Trailer of the file always starts with 9


Lines start with 7 has to be merged with the previous lines start with 6 and the line length should be 605.

Last edited by senthil_is; 03-04-2008 at 09:48 PM.. Reason: adding some more
# 3  
Old 03-05-2008
Hi,

You'd better tell us your requirement. Maybe some one can figure out some different solution, then you can compare them.

I think nobody like to read the complex scripte line by line.

Smilie
# 4  
Old 03-05-2008
Thank you very much summer_cherry. Thanks for another awk reply also.

My requirement is :

i have file which has the following format

header
BatchHeader-1
Details1
700000023
Details2
75858585
72323232
BatchTrailer-1
BatchHeader-2
Details1
700000023
Details2
75858585
72323232
BatchTrailer-2
Trailer

I want the output like below.

header
BatchHeader-1
Details1700000023 1 # this 1 means that detail belongs to batchheader-1
Details27585858572323232 1
BatchTrailer-1
BatchHeader-2
Details1700000023 2 # this 2 means that detail belongs to batchheader-2
Details27585858572323232 2
BatchTrailer-2
Trailer
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merging two lines into one (awk)

Hi, I am attempting to merge the following lines which run over two lines using awk. INITIAL OUTPUT 2019 Sep 28 10:47:24.695 hkaet9612 last message repeated 1 time 2019 Sep 28 10:47:24.695 hkaet9612 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interfa ce Ethernet1/45 is down (Interface removed)... (10 Replies)
Discussion started by: sand1234
10 Replies

2. Shell Programming and Scripting

Merging multiple lines to columns with awk, while inserting commas for missing lines

Hello all, I have a large csv file where there are four types of rows I need to merge into one row per person, where there is a column for each possible code / type of row, even if that code/row isn't there for that person. In the csv, a person may be listed from one to four times... (9 Replies)
Discussion started by: RalphNY
9 Replies

3. Shell Programming and Scripting

Merging the lines of a file

Hello, I have a file with few lines starting with a digit (1-5 only ) followed by a dot (.). Remaining all the lines to be merged with its previous numbered lines. Merging must be done with a space. E.g., Source file: 3. abc def xyz 5. pqr mno def 4. jkl uvw 7. ghi 1. abc xyz 6. mno... (4 Replies)
Discussion started by: magnus29
4 Replies

4. Shell Programming and Scripting

Merging 2 lines together

I have a small problem, which due to my lack of knowledge, has left me unable to decipher some of the solutions that I looked at on these forums. So below is a piece of text, which I ran via cat -vet, which comes from within a program file. I have many such programs to process and repeatable,... (4 Replies)
Discussion started by: skarnm
4 Replies

5. Shell Programming and Scripting

Merging lines

Thanks it worked for me. I have one more question on top of that. We had few records which were splitted in 2 lines instead of one. Now i identified those lines. The file is too big to open via vi and edit it. How can i do it without opening the file. Suppose, I want line number 1001 & 1002 to... (2 Replies)
Discussion started by: Gangadhar Reddy
2 Replies

6. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

7. Shell Programming and Scripting

Merging lines

Hi folks. Could somebody help me write a script or command that will look through a file and for every line that doesn't contain a certain value, merge it with the one above? For example, the file contains: SCOTLAND|123|ABC|yes SCOTLAND|456|DEF|yes SCOTLAND|78 9|GHI|yes ... (3 Replies)
Discussion started by: MDM
3 Replies

8. Shell Programming and Scripting

Shell script for merging lines in a loop

Dear All, I need a script to merge lines of an input file in a loop, please guide me for the script or one liner(awk, sed, tr, shell, perl). I/P File------------------------- APaul,,,,SDH,,23,,,PPH,,2 ,,,,KKH,,19,,,MMH,,12, ,,,,CCH,,22,,,MNH,,19, ,,,,TCH,,55,,,NNH,,67,... (3 Replies)
Discussion started by: ashis.tewari
3 Replies

9. Web Development

MySQL Tuning Tools with mysqltuner.pl and tuning-primer.sh

We have been tuning MySQL lately and I ran accoss two useful tools that you might be interested in: mysqltuner.pl tuning-primer.sh Both of these scripts are quite useful for MySQL tuning. Here is some sample output of mysqltuner.pl >> MySQLTuner 0.9.8 - Major Hayden... (3 Replies)
Discussion started by: Neo
3 Replies

10. UNIX for Dummies Questions & Answers

Merging lines into one

Hello. I would be very pleased if sb. help me to solve my problem. I've got a file with many non blank lines and I want to merge all lines into one not destroy the informations on them. I've tryed it with split and paste, tr, sed , but everything I've done has been wrong. I know about crazy... (8 Replies)
Discussion started by: Foxgard
8 Replies
Login or Register to Ask a Question