Shell Programming and Scripting

I have a file like the format

0000023716272400000237162724555778180300000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000001099001099425%00109925%IPY 2007013120070227Y99991231N YYX SRT0001010125%0000503318882HANDH800 R
0000023716282300000237162823555765956000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000001099001099425%00109925%IPY 2007013100010101Y99991231N YYX SRT0001010125%0000503418302HANDH800 R
0000030805382100000308053821555002938300000010000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000001699001699Q39%00169939%IP 2000093020000905N00010101N NYXNNNSRT0001010139%0000200601622PMPP6007 Y Z

here i copy and past my code

#############################################
# Parse File
#############################################
cut -c2-14 /var/www/html/final_java/IngramHourlyFile/stock@ingram.dat|awk '{print "where isbn=|"$1"|"}'|tr "|" "'">/var/www/html/final_java/IngramHourlyFile/isbn.dat
cut -c39-45 /var/www/html/final_java/IngramHourlyFile/stock@ingram.dat|awk '{print "UPDATE item_details SET ib_N_qty="$1}'>/var/www/html/final_java/IngramHourlyFile/OHQTY_N.dat
cut -c46-52 /var/www/html/final_java/IngramHourlyFile/stock@ingram.dat|awk '{print "ib_E_qty="$1}'>/var/www/html/final_java/IngramHourlyFile/OHQTY_E.dat
cut -c53-59 /var/www/html/final_java/IngramHourlyFile/stock@ingram.dat|awk '{print "ib_D_qty="$1}'>/var/www/html/final_java/IngramHourlyFile/OHQTY_D.dat
cut -c60-66 /var/www/html/final_java/IngramHourlyFile/stock@ingram.dat|awk '{print "ib_C_qty="$1}'>/var/www/html/final_java/IngramHourlyFile/OHQTY_C.dat
cut -c151-156 /var/www/html/final_java/IngramHourlyFile/stock@ingram.dat|awk '{print "iblastupdate=NOW(),ib_qty=(ib_E_qty+ib_N_qty+ib_D_qty+ib_C_qty),ib_listprice="$1"/100"}'>/var/www/html/final_java/IngramHourlyFile/PRICE.dat
cut -c164-166 /var/www/html/final_java/IngramHourlyFile/stock@ingram.dat|awk '{print "ib_discode=|"$1"|"}'|tr "|" "'">/var/www/html/final_java/IngramHourlyFile/DISCODE.dat

############################################
# Change File permission
############################################

chmod 777 /var/www/html/final_java/IngramHourlyFile/isbn.dat
chmod 777 /var/www/html/final_java/IngramHourlyFile/OHQTY_N.dat
chmod 777 /var/www/html/final_java/IngramHourlyFile/OHQTY_E.dat
chmod 777 /var/www/html/final_java/IngramHourlyFile/OHQTY_D.dat
chmod 777 /var/www/html/final_java/IngramHourlyFile/OHQTY_C.dat
chmod 777 /var/www/html/final_java/IngramHourlyFile/PRICE.dat
chmod 777 /var/www/html/final_java/IngramHourlyFile/DISCODE.dat

paste -d"," /var/www/html/final_java/IngramHourlyFile/OHQTY_N.dat /var/www/html/final_java/IngramHourlyFile/OHQTY_E.dat /var/www/html/final_java/IngramHourlyFile/OHQTY_D.dat /var/www/html/final_java/IngramHourlyFile/OHQTY_C.dat /var/www/html/final_java/IngramHourlyFile/PRICE.dat /var/www/html/final_java/IngramHourlyFile/DISCODE.dat>/var/www/html/final_java/IngramHourlyFile/tmpfinal_isbn.dat

chmod 777 /var/www/html/final_java/IngramHourlyFile/tmpfinal_isbn.dat

paste -d" " /var/www/html/final_java/IngramHourlyFile/tmpfinal_isbn.dat /var/www/html/final_java/IngramHourlyFile/isbn.dat>/var/www/html/final_java/IngramHourlyFile/tfinal_isbn.dat

chmod 777 /var/www/html/final_java/IngramHourlyFile/tfinal_isbn.dat

paste -d";" /var/www/html/final_java/IngramHourlyFile/tfinal_isbn.dat /var/www/html/final_java/IngramHourlyFile/single_quote.dat>/var/www/html/final_java/IngramHourlyFile/final_isbn.dat

chmod 777 /var/www/html/final_java/IngramHourlyFile/final_isbn.dat

It takes too long time because the file size is too big.I have to do it with in 15 min.

please help me.

thanks!
vaskar

not sure if this will increase performance by a significant amuont but you could always copy those files to the /tmp dir and read from there. also, a suggestion for you. when you have a dir that is common in your code, i would write it once to help others and you read your script a little easier:

Code:

#############################################
# Parse File
#############################################
dir=/var/www/html/final_java/IngramHourlyFile

cut -c2-14 $dir/stock@ingram.dat|awk '{print "where isbn=|"$1"|"}'|tr "|" "'">$dir/isbn.dat
cut -c39-45 $dir/stock@ingram.dat|awk '{print "UPDATE item_details SET ib_N_qty="$1}'>$dir/OHQTY_N.dat
cut -c46-52 $dir/stock@ingram.dat|awk '{print "ib_E_qty="$1}'>$dir/OHQTY_E.dat
cut -c53-59 $dir/stock@ingram.dat|awk '{print "ib_D_qty="$1}'>$dir/OHQTY_D.dat
cut -c60-66 $dir/stock@ingram.dat|awk '{print "ib_C_qty="$1}'>$dir/OHQTY_C.dat
cut -c151-156 $dir/stock@ingram.dat|awk '{print "iblastupdate=NOW(),ib_qty=(ib_E_qty+ib_N_qty+ib_D_qty+ib_C_qty),ib_listprice="$1"/100"}'>$dir/PRICE.dat
cut -c164-166 $dir/stock@ingram.dat|awk '{print "ib_discode=|"$1"|"}'|tr "|" "'">$dir/DISCODE.dat

This is a good example of Useless Use of cut, tr , chmod, paste and temp files when you can use only awk.

Code:

#cat awk.code
{
a="UPDATE item_details SET ib_N_qty="substr($0,39,7)
b="ib_E_qty="substr($0,46,7)
c="ib_D_qty="substr($0,53,7)
d="ib_C_qty="substr($0,60,7)
e="iblastupdate=NOW(),ib_qty=(ib_E_qty+ib_N_qty+ib_D_qty+ib_C_qty),ib_listprice="substr($0,151,6)"/100"
f="ib_discode=\47"substr($0,164,3)"\47"
g="where isbn=\47" substr($0,2,13)"\47"
print a","b","c","d","e","f" "g";"
}

And now run the code.

Code:

awk -f awk.code data.file > sql.file

Thanks! it is working....