Looking at your code more closely, it seems that it is even worse than I thought. You are loading all of the data from all of the files into awk on each run, but just printing a fraction of the results. And, you aren't being consistent in your specified pathnames (sometimes using pathnames relative to the current directory and sometimes using the directory specified by $BASEPATH. The following should run MUCH faster for you:

Code:

#!/bin/bash
BASEPATH="/path/to/your/data"	# Must be an absolute pathname
		# Source files must be in directory $BASEPATH/dataset
FILENAME="result"	# Store results in $BASEPATH/processed/$FILENAME

cd "$BASEPATH"
date '+START PROCESSING DATE: %d/%m/%y - TIME: %H:%M:%S'
awk '
FNR == 1 {	
	fn++
}
{	keys[key = $1 OFS $2]
	data[key, fn] = $3
}
END {	for(key in keys) {
		sum = 0
		printf("%s", key)
		for(i = 1; i <= fn; i++) {
			printf("%s%d", OFS, data[key, i])
			sum += data[key, i]
		}
		print OFS sum
	}
}' "dataset/"* > "processed/$FILENAME"
date '+END PROCESSING DATE: %d/%m/%y - TIME: %H:%M:%S'

or, if you need the output sorted, change the next to the last line of the script to:

Code:

}' "dataset/"* | sort -k2,2 -o "processed/$FILENAME"

which with your 3 sample input files located in the directory $BASEPATH/dataset stores the following data in the file named $BASEPATH/processed/$FILENAME:

Code:

a sample_1 200 10 1 211
a sample_3 10 67 0 77
a sample_4 0 0 20 20
a.b sample_2 10 0 10 20

(although the output order may vary with different versions of awk) or, if you pipe the output through sort:

Code:

a sample_1 200 10 1 211
a.b sample_2 10 0 10 20
a sample_3 10 67 0 77
a sample_4 0 0 20 20

I didn't see any need to clear the screen before outputting two lines of data, but you can add the clear back into the top of the script if you want to.

As always, if you are using this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.