The goal of The UNIX and Linux Forums is to help you learn how to do "stuff" on your own; not to write programs for you. I gave you a sample script to get you started, and from your message #9 in this thread it sounded like you were well on your way to getting a working solution. (And posting a 384Kb zipped archive that expands to over 1Mb without a clear indication of the desired outcome of processing it takes more space and time that most volunteers are willing to donate.)

From what you have shown here in message #10, you are learning quickly. I will make a few more comments that may help you speed this up a little bit: First, in the pipeline:

Code:

df -h | ls *.out.txt | sort -t_ -k$KEY_FIELD,$KEY_FIELD'n' | awk -F_ -v f=$KEY_FIELD -v c=$FILE_COUNT 'NR > c {exit} {printf("%s", $0)}'

what would happen if you remove the code shown in red? The ls utility doesn't read from standard input, so it seems that the df command in this pipeline should make no difference in the output of this pipeline. (It will just make the pipeline run slower.)

Second you seem to go to a lot of effort to store the output of this pipeline in an array and then spend a lot of time trying to extract individual file names from the array. It looks like the array will only have one element because the printf in your awk command doesn't put a space between the names of the files it prints. If you would change the printf statement from:

Code:

printf("%s", $0)

to:

Code:

printf(" %s", $0)

you could reference filenames in the array more simply by using ${FILELIST[0]} through ${FILELIST[$((FILE_COUNT-1))]}.

But, why have an array at all. Why not just process the files one at a time as they come out of awk? As an example, what would happen if you replaced:

Code:

   # sort the list of filenames and output the top number "n" as specified in argument $3
   FILE_LIST=( $(df -h | ls *.out.txt | sort -t_ -k$KEY_FIELD,$KEY_FIELD'n' | awk -F_ -v f=$KEY_FIELD -v c=$FILE_COUNT 'NR > c {exit} {printf("%s", $0)}') )

   # loop up to file count to parse output and copy files that were found by sort
   for (( LOOP_CT=1; LOOP_CT<=$FILE_COUNT; LOOP_CT++ ))
   do

      # parse output string on .out.txt to locate individual files
      FILE_TEMP=`echo $FILE_LIST | awk -v N=$LOOP_CT 'BEGIN {FS=".out.txt"} {print $N}'`
      # restore file extension
      FILE_NAME=$FILE_TEMP'.out.txt'

      echo $FILE_NAME

      # copy file and corresponding ini weight set to continue
      # copy file to continue
      cp -p './'$FILE_NAME './'$FOLD'_continue/'$SET_TYPE'/'$FILE_NAME

      #  find random ini set number
      RAND_SET=`echo $FILE_NAME | awk 'BEGIN {FS="_"} {print $5}'`
      # copy random ini weight file to continue
      cp -p '../rnd_ini/'$FOLD'/ri_'$RAND_SET'_'*'.wts'  './'$FOLD'_continue/'$SET_TYPE'/'

   done

with the much simpler:

Code:

      ls *.out.txt | sort -t_ -k$KEY_FIELD,${KEY_FIELD}n |
      awk -F_ -v c="$FILE_COUNT" '
        NR > c {exit}
        {print $0, $5}' |
      while read FILE_NAME RAND_SET
      do
        # copy files that were found by sort
        echo "file_name: $FILE_NAME rand_set: $RAND_SET"

        # copy file and corresponding ini weight set to continue
        # copy file to continue
        cp -p './'$FILE_NAME './'$FOLD'_continue/'$SET_TYPE'/'$FILE_NAME

        # copy random ini weight file to continue
        cp -p '../rnd_ini/'$FOLD'/ri_'$RAND_SET'_'*'.wts'  './'$FOLD'_continue/'$SET_TYPE'/'
      done

Note that there is no array here, there is only one invocation of awk (instead of n+1 invocations to process n files), and RAND_SET is pulled from the file name at the file name at a time when we already have the fields in the file name split out (so we only have to split the name once). You can also get rid of some unneeded temporary variables since OUTPUT was not (and still is not) referenced after being set, and FILE_TEMP is no longer used.