Performance issue in UNIX while generating .dat file from large text file

KRAMA · #1 (**permalink**) 04-20-2009

Hello Gurus,

We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this .

Problem Definition:
/Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below portion of code. The below portion of codes reads an input file and writes them into an .dat file. The performance issue arises when there is huge volume of data in the input file.
For example: For data volume having 200,000 records is taking 38 mins to get append/write into the .dat file which increases the complete load process timings. We need to increase the performance of this proces by reducing the time its taking to append/write the records.
/*****************************************

Portion of Code from Shell Script:
/**************************************************************************************************** *******************************************
m_arr_ctr=1
cat ${m_recv_dir}/${m_glb_d92_nm}${m_glb_file_seq} |while read d92_line
do
m_brch_cd=`echo "${d92_line}" |cut -c166-168`
# This is the case when we reach the last line '*/', we just skip that line
if [ "${m_brch_cd}" = "" ]
then
continue
fi
if [ "${m_brch_cd}" = "400" ]
then
m_jv_cd=`echo "${d92_line}" |cut -c190-192`
else
m_jv_cd=${m_brch_cd}
fi
if [ ! -s tmp_d92${m_brch_cd}z${m_jv_cd} ]
then
echo "TMP" > tmp_d92${m_brch_cd}z${m_jv_cd}
m_a_d92_list[$m_arr_ctr]=tmp_d92${m_brch_cd}z${m_jv_cd}
m_a_d92_files[$m_arr_ctr]=${m_recv_dir}/gd${m_brch_cd}x${m_jv_cd}${m_glb_rate_cd}.dat
m_arr_ctr=`expr $m_arr_ctr + 1`
m_touched="N"
else
m_touched="Y"
fi
if [ m_touched = "N" ]
then
echo "${d92_line}" > ${m_recv_dir}/gd${m_brch_cd}${m_jv_cd}${m_glb_rate_cd}.dat
else
echo "${d92_line}" >> ${m_recv_dir}/gd${m_brch_cd}${m_jv_cd}${m_glb_rate_cd}.dat
fi

done
for m_file_name in `echo ${m_a_d92_files[*]}`
do
if [[ `grep "*/" ${m_file_name} | wc -l` = 0 ]]
then
echo "*/" >> ${m_file_name}
fi
done
for m_file_name in `echo ${m_a_d92_list[*]}`
do
rm -f $m_file_name
done
/************************************

Please provide your valuable suggestions. Also is there any way by using SED command for appending the output in fast way?

cfajohnson · #2 (**permalink**) 04-20-2009

*

Quote:

Originally Posted by KRAMA

We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this .

Problem Definition:
/Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below portion of code. The below portion of codes reads an input file and writes them into an .dat file. The performance issue arises when there is huge volume of data in the input file.
For example: For data volume having 200,000 records is taking 38 mins to get append/write into the .dat file which increases the complete load process timings. We need to increase the performance of this proces by reducing the time its taking to append/write the records.

With a file that size, you should really be using awk.

Quote:

/*****************************************

Portion of Code from Shell Script:

Please put code inside [code] tags.

Quote:

/****************************************

Code:

m_arr_ctr=1
cat ${m_recv_dir}/${m_glb_d92_nm}${m_glb_file_seq} |while read d92_line

That cat is an unnecessary external command, but since it is only run once, eliminating it wll make very little difference.

Part of the slowness is due to calling multiple external commands (many of which are unnecessary: there's no need for expr as the shell can do its own arithmetic) for every line.

Quote:

Code:

do
m_brch_cd=`echo "${d92_line}" |cut -c166-168`

What shell are you using? If it's bash or ksh93, you can replace the call to cut:

Code:
m_brch_cd=${d92_line:165:3}

KRAMA · #3 (**permalink**) 04-20-2009

Hi johnson,

thanks for your advise. I will try to implement your suggestion and will look in the performance. Also the shell used here is ksh.

cfajohnson · #4 (**permalink**) 04-20-2009

Which version of ksh?

KRAMA · #5 (**permalink**) 04-21-2009

Hi John,

The ksh version is 88f. Also i implemented the comand which you gave but the one having removing cut (i.e m_brch_cd=${d92_line:165:3} ) did not worked as you said it will work for ksh93 . And rest of the command did not improved the perfoprmance much . (it improved performance by 1-2 mins). Can you please help me with the suggestion of using AWK. I am very new to AWK .

cfajohnson · #6 (**permalink**) 04-21-2009

Quote:

Originally Posted by KRAMA

Can you please help me with the suggestion of using AWK. I am very new to AWK ...

Please describe exactly what the script needs to do.

What files does it use for input? What is the format of those files?

What is the format of the output?

KRAMA · #7 (**permalink**) 04-30-2009

Can any one help me on this...

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Need to extract 7 characters immediately after text '19' from a large file.	parshant_bvcoe	Shell Programming and Scripting	7	01-21-2009 08:10 AM
How can i read a non text file in unix - ELF-64 executable object file - IA64	alexcol	UNIX for Advanced & Expert Users	8	11-07-2008 08:56 AM
Scaling Up Text Classification for Large File Systems	iBot	UNIX and Linux RSS News	0	06-23-2008 06:20 AM
how to edit large file in unix	balireddy_77	Shell Programming and Scripting	3	12-14-2006 07:40 AM
Unix File System performance with large directories	dive	UNIX for Dummies Questions & Answers	3	03-12-2004 05:31 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread: