Shell Programming and Scripting

So was awk, and it is much easier to learn, and awk scripts are much easier to understand.

As far as I can see this section of code reads all the output data files to find out if they contain a '*/' and then appends a '*/' if there isn't one present.

Earlier in the script we apparently ignored the last line '*/' in the input stream (not proven that that bit of code works).

Providing that '*/' was properly ignored in the input stream (an area of the script which could be improved by using grep -v \^'*/' instead of the very first cat) it is impossible for a '*/' to appear in any of the output files. We can therefore halve the run time by not re-reading the output data before appending the '*/'.

Code:

for m_file_name in `echo ${m_a_d92_files[*]}`
do
                echo "*/" >> ${m_file_name}
done

Untested.

Using echo is unnecessary and will break the script if any member of m_a_d92_files[*] contains whitespace.

Code:

for m_file_name in "${m_a_d92_files[@]}"

cafjohnson.
Agreed. The script has many areas which could be improved. Apparently the script works with the files provided, but takes too long.
I looked at whether the "card dealing" method for splitting the data could be improved without using a high level language, but there is insufficient information about the data type distribution and no rules stated about the processing order of the data. As far as I can see the core script is slow because it appends to multiple output files.

The script is slow because you are using the shell on a very large file; that is exacerbated by a number of inefficient constructs and poorly written code.

If I knew exactly what you are trying to do, I could suggest an awk script.

pls post few records of the input file to help you .