Sponsored Content
Top Forums Shell Programming and Scripting Merging lines - Tuning the script Post 302172832 by senthil_is on Tuesday 4th of March 2008 08:39:03 PM
Old 03-04-2008
Merging lines - Tuning the script

Hi, Anybody can tune the below script? Its taking 1hour to process 7000 lines.

addendaRecInd=7
entryDetailRecord=6
hdrDetailRecord=1
companyHdrRecord=5
companyCtrlRecord=8
trlRecord=9
recLength=94


FormatDone=false
trlCompleted=false

prevEntryCalc=false
batchCnt=0
hdrFound=false
while read line
do

# Find the Line #
grep -n "$line" $srcFile |cut -d : -f1 |read lineno
#find the Rec Type
if [ `echo $line|grep "^$companyHdrRecord"| wc -l` -ne 0 ]
then
hdrFound="true"
val=""
typeset -L94 line
hdr="$line"
typeset -R0 line
typeset -R0 batchCnt
fi
if [ `echo $line|grep "^$hdrDetailRecord"| wc -l` -ne 0 ]
then
typeset -L94 line
value=""
echo "$line" >> $destFile
line=""
typeset -R0 line
fi
if [ `echo $line|grep "^$trlRecord"| wc -l` -ne 0 -a $trlCompleted = false ]
then
echo $trlCompleted
value=""
typeset -L94 line
echo "$line" >> $destFile
line=""
typeset -R0 line
trlCompleted=true
echo $trlCompleted
fi
if [ `echo $line|grep "^$companyCtrlRecord"| wc -l` -ne 0 ]
then
if [ "$hdrFound" != "true" ]
then
echo "hdrFound" $hdrFound
echo "The last line is " $value
if [ "$value" != " " ]
then
typeset -L599 value
typeset -L6 batchCnt
echo "$value$batchCnt" >> $destFile
typeset -R0 value
typeset -R0 batchcnt
fi
typeset -L94 line
typeset -L6 batchcnt
echo "$line$batchCnt" >> $destFile
typeset -R0 line
typeset -R0 batchcnt
value=""
line=""
prevEntryCalc=false
fi
fi
if [ `echo $line|grep "^$entryDetailRecord"| wc -l` -ne 0 ]
then
# Print Header after the entry detail record has been read
if [ "$hdrFound" != "false" ]
then
batchCnt=`expr $batchCnt + 1 `
typeset -L94 hdr
typeset -L6 batchCnt
echo "$hdr$batchCnt" >> $destFile
typeset -R0 hdr
hdrFound="false"
hdr=""
fi
if [ "$prevEntryCalc" != "false" ]
then
typeset -L599 value
typeset -L6 batchCnt
echo "$value$batchCnt" >> $destFile
typeset -R0 value
typeset -R0 batchcnt
value=""
fi
line=`echo "$line"`
value=`echo "$line"`
prevEntryCalc="true"
fi
if [ `echo $line|grep "^$addendaRecInd" | wc -l` -ne 0 ]
then
typeset -L101 line
len=`echo $line|wc -c`
#echo "The Length of addenda record is "$len
value="$value$line"
typeset -R0 line
fi
done < $srcFile

Last edited by senthil_is; 03-04-2008 at 09:40 PM.. Reason: adding some more tips
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Merging lines into one

Hello. I would be very pleased if sb. help me to solve my problem. I've got a file with many non blank lines and I want to merge all lines into one not destroy the informations on them. I've tryed it with split and paste, tr, sed , but everything I've done has been wrong. I know about crazy... (8 Replies)
Discussion started by: Foxgard
8 Replies

2. Web Development

MySQL Tuning Tools with mysqltuner.pl and tuning-primer.sh

We have been tuning MySQL lately and I ran accoss two useful tools that you might be interested in: mysqltuner.pl tuning-primer.sh Both of these scripts are quite useful for MySQL tuning. Here is some sample output of mysqltuner.pl >> MySQLTuner 0.9.8 - Major Hayden... (3 Replies)
Discussion started by: Neo
3 Replies

3. Shell Programming and Scripting

Shell script for merging lines in a loop

Dear All, I need a script to merge lines of an input file in a loop, please guide me for the script or one liner(awk, sed, tr, shell, perl). I/P File------------------------- APaul,,,,SDH,,23,,,PPH,,2 ,,,,KKH,,19,,,MMH,,12, ,,,,CCH,,22,,,MNH,,19, ,,,,TCH,,55,,,NNH,,67,... (3 Replies)
Discussion started by: ashis.tewari
3 Replies

4. Shell Programming and Scripting

Merging lines

Hi folks. Could somebody help me write a script or command that will look through a file and for every line that doesn't contain a certain value, merge it with the one above? For example, the file contains: SCOTLAND|123|ABC|yes SCOTLAND|456|DEF|yes SCOTLAND|78 9|GHI|yes ... (3 Replies)
Discussion started by: MDM
3 Replies

5. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

6. Shell Programming and Scripting

Merging lines

Thanks it worked for me. I have one more question on top of that. We had few records which were splitted in 2 lines instead of one. Now i identified those lines. The file is too big to open via vi and edit it. How can i do it without opening the file. Suppose, I want line number 1001 & 1002 to... (2 Replies)
Discussion started by: Gangadhar Reddy
2 Replies

7. Shell Programming and Scripting

Merging 2 lines together

I have a small problem, which due to my lack of knowledge, has left me unable to decipher some of the solutions that I looked at on these forums. So below is a piece of text, which I ran via cat -vet, which comes from within a program file. I have many such programs to process and repeatable,... (4 Replies)
Discussion started by: skarnm
4 Replies

8. Shell Programming and Scripting

Merging the lines of a file

Hello, I have a file with few lines starting with a digit (1-5 only ) followed by a dot (.). Remaining all the lines to be merged with its previous numbered lines. Merging must be done with a space. E.g., Source file: 3. abc def xyz 5. pqr mno def 4. jkl uvw 7. ghi 1. abc xyz 6. mno... (4 Replies)
Discussion started by: magnus29
4 Replies

9. Shell Programming and Scripting

Merging multiple lines to columns with awk, while inserting commas for missing lines

Hello all, I have a large csv file where there are four types of rows I need to merge into one row per person, where there is a column for each possible code / type of row, even if that code/row isn't there for that person. In the csv, a person may be listed from one to four times... (9 Replies)
Discussion started by: RalphNY
9 Replies

10. UNIX for Beginners Questions & Answers

Merging two lines into one (awk)

Hi, I am attempting to merge the following lines which run over two lines using awk. INITIAL OUTPUT 2019 Sep 28 10:47:24.695 hkaet9612 last message repeated 1 time 2019 Sep 28 10:47:24.695 hkaet9612 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interfa ce Ethernet1/45 is down (Interface removed)... (10 Replies)
Discussion started by: sand1234
10 Replies
typeset(1)                                                         User Commands                                                        typeset(1)

NAME
typeset, whence - shell built-in functions to set/get attributes and values for shell variables and functions SYNOPSIS
typeset [ +- HLRZfilrtux [n]] [ name [ = value]]... whence [-pv] name... DESCRIPTION
typeset sets attributes and values for shell variables and functions. When typeset is invoked inside a function, a new instance of the variables name is created. The variables value and type are restored when the function completes. The following list of attributes may be specified: -H This flag provides UNIX to host-name file mapping on non-UNIX machines. -L Left justify and remove leading blanks from value. If n is non-zero it defines the width of the field; otherwise, it is determined by the width of the value of first assignment. When the variable is assigned to, it is filled on the right with blanks or trun- cated, if necessary, to fit into the field. Leading zeros are removed if the -Z flag is also set. The -R flag is turned off. -R Right justify and fill with leading blanks. If n is non-zero it defines the width of the field, otherwise it is determined by the width of the value of first assignment. The field is left filled with blanks or truncated from the end if the variable is reas- signed. The -L flag is turned off. -Z Right justify and fill with leading zeros if the first non-blank character is a digit and the -L flag has not been set. If n is non-zero it defines the width of the field; otherwise, it is determined by the width of the value of first assignment. -f The names refer to function names rather than variable names. No assignments can be made and the only other valid flags are -t, -u and -x. The flag -t turns on execution tracing for this function. The flag -u causes this function to be marked undefined. The FPATH variable will be searched to find the function definition when the function is referenced. The flag -x allows the func- tion definition to remain in effect across shell procedures invoked by name. -i Parameter is an integer. This makes arithmetic faster. If n is non-zero it defines the output arithmetic base; otherwise, the first assignment determines the output base. -l All upper-case characters are converted to lower-case. The upper-case flag, -u is turned off. -r The given names are marked readonly and these names cannot be changed by subsequent assignment. -t Tags the variables. Tags are user definable and have no special meaning to the shell. -u All lower-case characters are converted to upper-case characters. The lower-case flag, -l is turned off. -x The given names are marked for automatic export to the environment of subsequently-executed commands. The -i attribute can not be specified along with -R, -L, -Z, or -f. Using + rather than - causes these flags to be turned off. If no name arguments are given but flags are specified, a list of names (and optionally the values) of the variables which have these flags set is printed. (Using + rather than - keeps the values from being printed.) If no names and flags are given, the names and attributes of all variables are printed. For each name, whence indicates how it would be interpreted if used as a command name. The -v flag produces a more verbose report. The -p flag does a path search for name even if name is an alias, a function, or a reserved word. On this man page, ksh(1) commands that are preceded by one or two * (asterisks) are treated specially in the following ways: 1. Variable assignment lists preceding the command remain in effect when the command completes. 2. I/O redirections are processed after variable assignments. 3. Errors cause a script that contains them to abort. 4. Words, following a command preceded by ** that are in the format of a variable assignment, are expanded with the same rules as a vari- able assignment. This means that tilde substitution is performed after the = sign and word splitting and file name generation are not performed. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ SEE ALSO
ksh(1), set(1), sh(1), attributes(5) SunOS 5.10 1 Feb 1995 typeset(1)
All times are GMT -4. The time now is 05:56 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy