03-04-2008
Merging lines - Tuning the script
Hi, Anybody can tune the below script? Its taking 1hour to process 7000 lines.
addendaRecInd=7
entryDetailRecord=6
hdrDetailRecord=1
companyHdrRecord=5
companyCtrlRecord=8
trlRecord=9
recLength=94
FormatDone=false
trlCompleted=false
prevEntryCalc=false
batchCnt=0
hdrFound=false
while read line
do
# Find the Line #
grep -n "$line" $srcFile |cut -d : -f1 |read lineno
#find the Rec Type
if [ `echo $line|grep "^$companyHdrRecord"| wc -l` -ne 0 ]
then
hdrFound="true"
val=""
typeset -L94 line
hdr="$line"
typeset -R0 line
typeset -R0 batchCnt
fi
if [ `echo $line|grep "^$hdrDetailRecord"| wc -l` -ne 0 ]
then
typeset -L94 line
value=""
echo "$line" >> $destFile
line=""
typeset -R0 line
fi
if [ `echo $line|grep "^$trlRecord"| wc -l` -ne 0 -a $trlCompleted = false ]
then
echo $trlCompleted
value=""
typeset -L94 line
echo "$line" >> $destFile
line=""
typeset -R0 line
trlCompleted=true
echo $trlCompleted
fi
if [ `echo $line|grep "^$companyCtrlRecord"| wc -l` -ne 0 ]
then
if [ "$hdrFound" != "true" ]
then
echo "hdrFound" $hdrFound
echo "The last line is " $value
if [ "$value" != " " ]
then
typeset -L599 value
typeset -L6 batchCnt
echo "$value$batchCnt" >> $destFile
typeset -R0 value
typeset -R0 batchcnt
fi
typeset -L94 line
typeset -L6 batchcnt
echo "$line$batchCnt" >> $destFile
typeset -R0 line
typeset -R0 batchcnt
value=""
line=""
prevEntryCalc=false
fi
fi
if [ `echo $line|grep "^$entryDetailRecord"| wc -l` -ne 0 ]
then
# Print Header after the entry detail record has been read
if [ "$hdrFound" != "false" ]
then
batchCnt=`expr $batchCnt + 1 `
typeset -L94 hdr
typeset -L6 batchCnt
echo "$hdr$batchCnt" >> $destFile
typeset -R0 hdr
hdrFound="false"
hdr=""
fi
if [ "$prevEntryCalc" != "false" ]
then
typeset -L599 value
typeset -L6 batchCnt
echo "$value$batchCnt" >> $destFile
typeset -R0 value
typeset -R0 batchcnt
value=""
fi
line=`echo "$line"`
value=`echo "$line"`
prevEntryCalc="true"
fi
if [ `echo $line|grep "^$addendaRecInd" | wc -l` -ne 0 ]
then
typeset -L101 line
len=`echo $line|wc -c`
#echo "The Length of addenda record is "$len
value="$value$line"
typeset -R0 line
fi
done < $srcFile
Last edited by senthil_is; 03-04-2008 at 09:40 PM..
Reason: adding some more tips
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hello. I would be very pleased if sb. help me to solve my problem. I've got a file with many non blank lines and I want to merge all lines into one not destroy the informations on them. I've tryed it with split and paste, tr, sed , but everything I've done has been wrong. I know about crazy... (8 Replies)
Discussion started by: Foxgard
8 Replies
2. Web Development
We have been tuning MySQL lately and I ran accoss two useful tools that you might be interested in:
mysqltuner.pl
tuning-primer.sh
Both of these scripts are quite useful for MySQL tuning. Here is some sample output of mysqltuner.pl
>> MySQLTuner 0.9.8 - Major Hayden... (3 Replies)
Discussion started by: Neo
3 Replies
3. Shell Programming and Scripting
Dear All,
I need a script to merge lines of an input file in a loop, please guide me for the script or one liner(awk, sed, tr, shell, perl).
I/P File-------------------------
APaul,,,,SDH,,23,,,PPH,,2
,,,,KKH,,19,,,MMH,,12,
,,,,CCH,,22,,,MNH,,19,
,,,,TCH,,55,,,NNH,,67,... (3 Replies)
Discussion started by: ashis.tewari
3 Replies
4. Shell Programming and Scripting
Hi folks.
Could somebody help me write a script or command that will look through a file and for every line that doesn't contain a certain value, merge it with the one above?
For example, the file contains:
SCOTLAND|123|ABC|yes
SCOTLAND|456|DEF|yes
SCOTLAND|78
9|GHI|yes
... (3 Replies)
Discussion started by: MDM
3 Replies
5. Shell Programming and Scripting
Hi everyone,
I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines).
Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies
6. Shell Programming and Scripting
Thanks it worked for me. I have one more question on top of that. We had few records which were splitted in 2 lines instead of one. Now i identified those lines. The file is too big to open via vi and edit it. How can i do it without opening the file.
Suppose, I want line number 1001 & 1002 to... (2 Replies)
Discussion started by: Gangadhar Reddy
2 Replies
7. Shell Programming and Scripting
I have a small problem, which due to my lack of knowledge, has left me unable to decipher some of the solutions that I looked at on these forums.
So below is a piece of text, which I ran via cat -vet, which comes from within a program file. I have many such programs to process and repeatable,... (4 Replies)
Discussion started by: skarnm
4 Replies
8. Shell Programming and Scripting
Hello,
I have a file with few lines starting with a digit (1-5 only ) followed by a dot (.). Remaining all the lines to be merged with its previous numbered lines. Merging must be done with a space.
E.g.,
Source file:
3. abc def
xyz
5. pqr mno
def
4. jkl uvw
7. ghi
1. abc xyz
6. mno... (4 Replies)
Discussion started by: magnus29
4 Replies
9. Shell Programming and Scripting
Hello all,
I have a large csv file where there are four types of rows I need to merge into one row per person, where there is a column for each possible code / type of row, even if that code/row isn't there for that person.
In the csv, a person may be listed from one to four times... (9 Replies)
Discussion started by: RalphNY
9 Replies
10. UNIX for Beginners Questions & Answers
Hi,
I am attempting to merge the following lines which run over two lines using awk.
INITIAL OUTPUT
2019 Sep 28 10:47:24.695 hkaet9612 last message repeated 1 time
2019 Sep 28 10:47:24.695 hkaet9612 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interfa
ce Ethernet1/45 is down (Interface removed)... (10 Replies)
Discussion started by: sand1234
10 Replies
LEARN ABOUT MOJAVE
typeset
typeset(1) User Commands typeset(1)
NAME
typeset, whence - shell built-in functions to set/get attributes and values for shell variables and functions
SYNOPSIS
typeset [ +- HLRZfilrtux [n]] [ name [ = value]]...
whence [-pv] name...
DESCRIPTION
typeset sets attributes and values for shell variables and functions. When typeset is invoked inside a function, a new instance of the
variables name is created. The variables value and type are restored when the function completes. The following list of attributes may be
specified:
-H This flag provides UNIX to host-name file mapping on non-UNIX machines.
-L Left justify and remove leading blanks from value. If n is non-zero it defines the width of the field; otherwise, it is determined
by the width of the value of first assignment. When the variable is assigned to, it is filled on the right with blanks or trun-
cated, if necessary, to fit into the field. Leading zeros are removed if the -Z flag is also set. The -R flag is turned off.
-R Right justify and fill with leading blanks. If n is non-zero it defines the width of the field, otherwise it is determined by the
width of the value of first assignment. The field is left filled with blanks or truncated from the end if the variable is reas-
signed. The -L flag is turned off.
-Z Right justify and fill with leading zeros if the first non-blank character is a digit and the -L flag has not been set. If n is
non-zero it defines the width of the field; otherwise, it is determined by the width of the value of first assignment.
-f The names refer to function names rather than variable names. No assignments can be made and the only other valid flags are -t, -u
and -x. The flag -t turns on execution tracing for this function. The flag -u causes this function to be marked undefined. The
FPATH variable will be searched to find the function definition when the function is referenced. The flag -x allows the func-
tion definition to remain in effect across shell procedures invoked by name.
-i Parameter is an integer. This makes arithmetic faster. If n is non-zero it defines the output arithmetic base; otherwise, the
first assignment determines the output base.
-l All upper-case characters are converted to lower-case. The upper-case flag, -u is turned off.
-r The given names are marked readonly and these names cannot be changed by subsequent assignment.
-t Tags the variables. Tags are user definable and have no special meaning to the shell.
-u All lower-case characters are converted to upper-case characters. The lower-case flag, -l is turned off.
-x The given names are marked for automatic export to the environment of subsequently-executed commands.
The -i attribute can not be specified along with -R, -L, -Z, or -f.
Using + rather than - causes these flags to be turned off. If no name arguments are given but flags are specified, a list of names (and
optionally the values) of the variables which have these flags set is printed. (Using + rather than - keeps the values from being
printed.) If no names and flags are given, the names and attributes of all variables are printed.
For each name, whence indicates how it would be interpreted if used as a command name.
The -v flag produces a more verbose report.
The -p flag does a path search for name even if name is an alias, a function, or a reserved word.
On this man page, ksh(1) commands that are preceded by one or two * (asterisks) are treated specially in the following ways:
1. Variable assignment lists preceding the command remain in effect when the command completes.
2. I/O redirections are processed after variable assignments.
3. Errors cause a script that contains them to abort.
4. Words, following a command preceded by ** that are in the format of a variable assignment, are expanded with the same rules as a vari-
able assignment. This means that tilde substitution is performed after the = sign and word splitting and file name generation are not
performed.
ATTRIBUTES
See attributes(5) for descriptions of the following attributes:
+-----------------------------+-----------------------------+
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
+-----------------------------+-----------------------------+
|Availability |SUNWcsu |
+-----------------------------+-----------------------------+
SEE ALSO
ksh(1), set(1), sh(1), attributes(5)
SunOS 5.10 1 Feb 1995 typeset(1)