First, you should (a) install the ATLAS and scalapak libraries, and make sure these are on each node. Second, you need to install one of the MPI packages (OpenMPI, LAMMPI, MPICH, etc); the run-times need to be on each node, and the compiler libraries and tools need to be on one node. Third, you need to recompile for MPI and ATLAS. I believe linpack uses a configure script in which you tell it to use MPI or something like that. Fourth, for these benchmarks, you should disable Linux's swap; this ensures the linpack doesn't start swapping and killing performance. (Do this with sysctl vm.swappiness=0" and after "=1") (If it runs out of memory, the problem size is too large, and the process fails.)

Next, start out with a simple test to make sure your hpl + mpi setup is working. You'll need a dummy config file like this:

Code:

Our cluster benchmark
My university lab
HPL.out
6
1
400
1
50
1
1     
4
3
-1
1       # of panel fact
0
1       # of recursive stopping criterium
2
1       # of panels in recursion
2
1       # of recursive panel fact.
0
1       # of Bcasts
1
1       # of Lookahead depths
0
2
60
0
0
0       
8       # alignment of double

It should run to completion and give you some reasonable output (the last number of the output is GFlops). With these Xeon's, your theoretical peak is 3 (nodes) * 4 (CPUs) * 4 (cores) * 3.4 (GHz) * N (Floating point operations / cycle) = 163 * N. (See the Intel spec sheet for your processor to determine N).

Once you have that working, you're ready for tuning the HPL suite: run a series of tests, each with a different configuration. One configuration file does this. The linpack program permutes all possible combinations of parameters within the file, and runs one test on each permutation. A quide to this format can be found here, but here's what I suggest you start with:

Code:

Our cluster benchmark
My university lab
HPL.out
6
7
100 200 400 800 1600 3200 6400 
5
50 100 150 200 250
1
4
12 4 3 1
1 3 4 12
-1
3       # of panel fact
0 1 2   PFACTs (0=left, 1=Crout, 2=Right)
4       # of recursive stopping criterium
1 2 4 8 NBMINs (>= 1)
3       # of panels in recursion
2 3 4   NDIVs
3       # of recursive panel fact.
0 1 2   RFACTs (0=left, 1=Crout, 2=Right)
5       # of Bcasts
0 1 2 3 4 5
3       # Lookaheads
0 1 2
2       # SWAP type
60      # SWAP=2 threshold
0
0
1
8

Now, this will will take a long time. Make sure you pipe the output with " | tee hpout.dat" to make sure you capture the output and can see it and it gets saved to disk.

After this, look for the top 8 or 16 results, and refine the config file to use only the parameters that produced these results.

NOW you can start performance tuning the cluster. Most critically, you will want to (a) tune the TCP/IP kernel parameters, (b) disable all non-essential Linux processes on all nodes, and (c) tune the switch parameters for the cluster ports -- ie, disable auto-negotiate and maybe tune the messaging queues (some switches use different types of service and have small queues for each one; you want one large queue for all TOS).