Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Performance issue to read line by line Post 302972381 by balu1729 on Wednesday 4th of May 2016 03:54:22 PM
Old 05-04-2016
Performance issue to read line by line

Hi All- we have performance issue in unix to read line by line.
I am looking at processing all the records.

description: Our script will read data from a flat file, it will pickup first four character and based on the value it will set up variables accordingly and appended the final output to another flat file as show below.

Concern: script is working fine but we have a performance issue to read line by line, we are looking at something like it will ready all the lines at a time and dynamically identify the first four character and accordingly set up the individual variables and finally append the value.

Please find attached script, sample input data file.


actual script:
---------------------------------------------------------------------------------------
Code:
#!/bin/ksh
#set -x


process_each_record()
{

  record=$1
  ###### extract first four characters  ##############
  record_type=`echo $record | sed 's/\(^....\).*/\1/'`

case $record_type  in

  1111)  a1=100; a2=0; a3=0; a4=0; a5=0; a6=0; a7=0;a8=0; a9=0;
	echo "${line_no}${a1}${a2}${a3}${a4}${a5}${a6}${a7}${a8}${a9}${record}" >> test1_all_data.log;
	;;
  1112) a2=` expr ${a2} + 1 `; a3=` expr ${a3} + 2 `;
	echo "${line_no}${a1}${a2}${a3}${a4}${a5}${a6}${a7}${a8}${a9}${record}" >> test1_all_data.log;
	;;
  1113) a7=` expr ${a7} + 1 `; a5=` expr ${a5} + 3 `;
	echo "${line_no}${a1}${a2}${a3}${a4}${a5}${a6}${a7}${a8}${a9}${record}" >> test1_all_data.log;
	;;
  1114) a4=` expr ${a4} + 3 `; a6=` expr ${a6} + 4 `; 
	echo "${line_no}${a1}${a2}${a3}${a4}${a5}${a6}${a7}${a8}${a9}${record}" >> test1_all_data.log;
	;;
  1115) a7=` expr ${a7} + 1 `; a9=` expr ${a9} + 3 `;
	echo "${line_no}${a1}${a2}${a3}${a4}${a5}${a6}${a7}${a8}${a9}${record}" >> test1_all_data.log;
	;;
  1116) a8=` expr ${a8} + 1 `; a5=` expr ${a5} + 1 `;
	echo "${line_no}${a1}${a2}${a3}${a4}${a5}${a6}${a7}${a8}${a9}${record}" >> test1_all_data.log;
	;;
  2221) a6=0; a7=0;a8=0; a9=0;
	echo "${line_no}${a1}${a2}${a3}${a4}${a5}${a6}${a7}${a8}${a9}${record}" >> test1_all_data.log;
	;;
  2222) a3=` expr ${a3} + 1 `; a7=` expr ${a7} + 3 `;
	echo "${line_no}${a1}${a2}${a3}${a4}${a5}${a6}${a7}${a8}${a9}${record}" >> test1_all_data.log;
	;;
  3333) a8=` expr ${a8} + 1 `; a9=` expr ${a9} + 5 `;
	echo "${line_no}${a1}${a2}${a3}${a4}${a5}${a6}${a7}${a8}${a9}${record}" >> test1_all_data.log;
	;;
  5555) a1=` expr ${a1} + 1 `; a2=` expr ${a2} + 3 `; a3=` expr ${a3} + 1 `; a4=` expr ${a4} + 1 `;
	echo "${line_no}${a1}${a2}${a3}${a4}${a5}${a6}${a7}${a8}${a9}${record}" >> test1_all_data.log;
	;;

     *) echo "${line_no}${a1}${a2}${a3}${a4}${a5}${a6}${a7}${a8}${a9}${record}" >> test1_all_data.log;
	;;

  esac


}



######## define variables #####
typeset -Z10 a1
typeset -Z7  a2
typeset -Z3  a3
typeset -Z6  a4
typeset -Z2  a5
typeset -Z7  a6
typeset -Z9  a7
typeset -Z5  a8
typeset -Z2  a8
typeset -Z4  a9
typeset -Z10  line_no

a1=0; a2=0; a3=0; a4=0; a5=0; a6=0; a7=0;a8=0; a9=0;line_no=0;


if [ -f test1_all_data.log ]
then
	rm test1_all_data.log
fi


cat test2.tlog | while read  line1
do
  line_no=` expr ${line_no} + 1 `
  process_each_record  ${line1}
done

---------------------------------------------------------------------------------------

Last edited by Don Cragun; 05-04-2016 at 06:11 PM.. Reason: Add CODE tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

bash: read file line by line (lines have '\0') - not full line has read???

I am using the while-loop to read a file. The file has lines with null-terminated strings (words, actually.) What I have by that reading - just a first word up to '\0'! I need to have whole string up to 'new line' - (LF, 10#10, 16#A) What I am doing wrong? #make file 'grb' with... (6 Replies)
Discussion started by: alex_5161
6 Replies

2. Shell Programming and Scripting

While loop read line Issue

Hi I am using while loop, below, to read lines from a very large file, around 400,000 rows. The script works fine until around line 300k but then starts giving incorrect result. I have tried running the script with a smaller data set and it works fine. I made sure to include the line where... (2 Replies)
Discussion started by: saurabhkumar198
2 Replies

3. Shell Programming and Scripting

Multi Line 'While Read' command issue when using sh -c

Hi, I'm trying to run the following command using sh -c ie sh -c "while read EachLine do rm -f $EachLine ; done < file_list.lst;" It doesn't seem to do anything. When I run this at the command line, it does remove the files contained in the list so i know the command works ie... (4 Replies)
Discussion started by: chrispward
4 Replies

4. Shell Programming and Scripting

while read LINE issue

Hi, This is the script and the error I am receiving Can anyone please suggest ? For the exmaple below assume we are using vg01 #!/bin/ksh echo "##### Max Mount Count Fixer #####" echo "Please insert Volume Group name to check" read VG lvs |grep $VG | awk {'print $1'} > /tmp/audit.log ... (2 Replies)
Discussion started by: galuzan
2 Replies

5. Shell Programming and Scripting

how to read the contents of two files line by line and compare the line by line?

Hi All, I'm trying to figure out which are the trusted-ips and which are not using a script file.. I have a file named 'ip-list.txt' which contains some ip addresses and another file named 'trusted-ip-list.txt' which also contains some ip addresses. I want to read a line from... (4 Replies)
Discussion started by: mjavalkar
4 Replies

6. Shell Programming and Scripting

Need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line...

Hello, I need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line... An example of entries in the file would be: SRVXPAPI001 ERRO JUN24 07:28:34 1775 REASON= 0000, PROCID= #E506 #1065: TPCIPPR, INDEX= 003F ... (8 Replies)
Discussion started by: Ferocci
8 Replies

7. Shell Programming and Scripting

How to read file line by line and compare subset of 1st line with 2nd?

Hi all, I have a log file say Test.log that gets updated continuously and it has data in pipe separated format. A sample log file would look like: <date1>|<data1>|<url1>|<result1> <date2>|<data2>|<url2>|<result2> <date3>|<data3>|<url3>|<result3> <date4>|<data4>|<url4>|<result4> What I... (3 Replies)
Discussion started by: pat_pramod
3 Replies

8. Shell Programming and Scripting

Read line, issue with leading - and {}'s

Heyas With my forum search term 'issue with leading dash' i found 2 closed threads which sadly didnt help me. Also me was to eager to add the script, that i didnt properly test, and just now figured this issue. So i have this code: if ] then while read line do line="${line/-/'\-'}"... (7 Replies)
Discussion started by: sea
7 Replies

9. Shell Programming and Scripting

[BASH] read 'line' issue with leading tabs and virtual line breaks

Heyas I'm trying to read/display a file its content and put borders around it (tui-cat / tui-cat -t(ypwriter). The typewriter-part is a 'bonus' but still has its own flaws, but thats for later. So in some way, i'm trying to rewrite cat using bash and other commands. But sadly it fails on... (2 Replies)
Discussion started by: sea
2 Replies

10. Shell Programming and Scripting

Performance issue - to read line by line

All- We have a performance issue in reading a file line by line. Please find attached scripts for the same. Currently it is taking some 45 min to parse "512444" lines. Could you please have a look at it and provide any suggestions to improve the performance. Thanks, Balu ... (12 Replies)
Discussion started by: balu1729
12 Replies
All times are GMT -4. The time now is 06:48 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy