I have been trying off an on to figure out how your code works and have been unable to do so. I assume that this is the contents of $config_file_path/crt_extracts_branch_gswid_cntl:

Code:

##Files Structure on which position we do get Obligor/GSW ID for each file :
crt_extracts_branch_gsw_cntl file data
#################################################
#Carrs Extract File                                         GSWID Column Possition
#################################################
Grermany_Source_1.txt						3
Grermany_Source_2.txt						3
Grermany_Source_3.txt						6			
Grermany_Source_4.txt						1
Grermany_Source_5.txt						1
Grermany_Source_6.txt						4

So after running the commands in your script:

Code:

        set -A FILENAME   $( cat $config_file_path/crt_extracts_branch_gswid_cntl | grep -v "^#" | awk '{ print $1 }')
        set -A COLPOS     $( cat $config_file_path/crt_extracts_branch_gswid_cntl | grep -v "^#" | awk '{ print $4 }')
        set -A CTLCOLPOS  $( cat $config_file_path/crt_extracts_branch_gswid_cntl | grep -v "^#" | awk '{ print $3 }')

FILENAME is an array containing crt_extracts_branch_gsw_cntl and Grermany_Source_1.txt through Grermany_Source_6.txt, COLPOS is an array containing nothing, and CTLCOLPOS is an array containing the string data. Moving on from there nothing makes any sense to me. It would seem that you would be better off running awk once to extract the three fields you want and read the output in a while loop instead of creating three arrays to be processed later.

I'm sorry, but if I can't understand how this code is working, I don't know how to make it work faster.

Obviously you could get rid of the three calls to cat and the three calls to grep in the above pipelines and just use:

Code:

        arrayname=($(awk '!/^#/{print $n}))

for various values of arrayname and n to speed up the code above, and you could change:

Code:

	SOURCE_DIR=$( echo "/tgrrappkg3/DataStage/CONSUMER_BI/datap5/WSLGER")

to the simpler and faster:

Code:

	SOURCE_DIR="/tgrrappkg3/DataStage/CONSUMER_BI/datap5/WSLGER"

but these things are only done once; so they won't have a big effect on the speed of the inner loop.

So, since your code is working for you as is, I don't think I can help much.

You have lots of command substitutions that could be replaced by simple (and faster) parameter expansions. You have a command construction segment and an eval that seem like they could be replaced with a simple command invocation. But I hesitate to make any suggestions here because I can't figure out what is going on with the data you've shown us.

We might have a better chance of understanding what you were doing if you would show us the actual contents of the files named in your script and label those files with their names (as used in your script). AND, please use CODE tags for you sample input files so we can see what is actually in the files versus what may just be descriptive text before or after the contents of those files.