Benchmarking a Beowulf Cluster


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
Special Forums UNIX and Linux Applications High Performance Computing Benchmarking a Beowulf Cluster
# 1  
Benchmarking a Beowulf Cluster

Hi guys. I am trying to test my universities cluster using the Intel Linpack Benchmarking software. Let me say from the get-go that i am a Linux novice, and only recently learnt some Linux commands to have a play around, so if you can please keep the language simple Smilie
I have run a Linpack test as can be seen from the below terminal copy and paste. However, the test that i have currently run, i believe is just testing the avg GFLOPS for the current node i am on. What if i want to test the strength of a cluster of 3 nodes all working together? Would anyone happen to know what i would need to type in the command line in order to run a test on 3 nodes that i have logged into (all clustered by the university already). The nodes are all Xeon 64's, with 4 Gigs of ram each. Would really appreciate any help.
Code:
[******@beowulf linpack]$ ./xlinpack_xeon64
 Input data or print help ? Type [data]/help :
Number of equations to solve (problem size): 20000
Leading dimension of array: 20000
Number of trials to run: 4
Data alignment value (in Kbytes): 4
Current date/time: Wed Apr 29 13:50:57 2009
 CPU frequency:    3.400 GHz
Number of CPUs: 4
Number of threads: 4
Parameters are set to:
 Number of tests                             : 1
Number of equations to solve (problem size) : 20000
Leading dimension of array                  : 20000
Number of trials to run                     : 4
Data alignment value (in Kbytes)            : 4
 Maximum memory requested that can be used = 3200404096, at the size = 20000
============= Timing linear equation system solver =================
 Size   LDA    Align. Time(s)    GFlops   Residual      Residual(norm)
Error: info returned = 1
20000  20000  4      237.211    22.4869  4.455000e-10 3.943651e-02
Error: info returned = 1
20000  20000  4      236.686    22.5368  4.455000e-10 3.943651e-02
Error: info returned = 1
20000  20000  4      235.285    22.6710  4.455000e-10 3.943651e-02
Error: info returned = 1
20000  20000  4      237.404    22.4686  4.455000e-10 3.943651e-02
 Performance Summary (GFlops)
 Size   LDA    Align.  Average  Maximal
20000  20000  4       22.5408  22.6710
 End of tests


Last edited by Neo; 05-11-2009 at 06:15 PM.. Reason: code tags
# 2  
First, you should (a) install the ATLAS and scalapak libraries, and make sure these are on each node. Second, you need to install one of the MPI packages (OpenMPI, LAMMPI, MPICH, etc); the run-times need to be on each node, and the compiler libraries and tools need to be on one node. Third, you need to recompile for MPI and ATLAS. I believe linpack uses a configure script in which you tell it to use MPI or something like that. Fourth, for these benchmarks, you should disable Linux's swap; this ensures the linpack doesn't start swapping and killing performance. (Do this with sysctl vm.swappiness=0" and after "=1") (If it runs out of memory, the problem size is too large, and the process fails.)

Next, start out with a simple test to make sure your hpl + mpi setup is working. You'll need a dummy config file like this:
Code:
Our cluster benchmark
My university lab
HPL.out
6
1
400
1
50
1
1     
4
3
-1
1       # of panel fact
0
1       # of recursive stopping criterium
2
1       # of panels in recursion
2
1       # of recursive panel fact.
0
1       # of Bcasts
1
1       # of Lookahead depths
0
2
60
0
0
0       
8       # alignment of double

It should run to completion and give you some reasonable output (the last number of the output is GFlops). With these Xeon's, your theoretical peak is 3 (nodes) * 4 (CPUs) * 4 (cores) * 3.4 (GHz) * N (Floating point operations / cycle) = 163 * N. (See the Intel spec sheet for your processor to determine N).

Once you have that working, you're ready for tuning the HPL suite: run a series of tests, each with a different configuration. One configuration file does this. The linpack program permutes all possible combinations of parameters within the file, and runs one test on each permutation. A quide to this format can be found here, but here's what I suggest you start with:
Code:
Our cluster benchmark
My university lab
HPL.out
6
7
100 200 400 800 1600 3200 6400 
5
50 100 150 200 250
1
4
12 4 3 1
1 3 4 12
-1
3       # of panel fact
0 1 2   PFACTs (0=left, 1=Crout, 2=Right)
4       # of recursive stopping criterium
1 2 4 8 NBMINs (>= 1)
3       # of panels in recursion
2 3 4   NDIVs
3       # of recursive panel fact.
0 1 2   RFACTs (0=left, 1=Crout, 2=Right)
5       # of Bcasts
0 1 2 3 4 5
3       # Lookaheads
0 1 2
2       # SWAP type
60      # SWAP=2 threshold
0
0
1
8

Now, this will will take a long time. Make sure you pipe the output with " | tee hpout.dat" to make sure you capture the output and can see it and it gets saved to disk.

After this, look for the top 8 or 16 results, and refine the config file to use only the parameters that produced these results.

NOW you can start performance tuning the cluster. Most critically, you will want to (a) tune the TCP/IP kernel parameters, (b) disable all non-essential Linux processes on all nodes, and (c) tune the switch parameters for the cluster ports -- ie, disable auto-negotiate and maybe tune the messaging queues (some switches use different types of service and have small queues for each one; you want one large queue for all TOS).
# 3  
Thanks for the response otheus.

Everything seems to be working except the tuning of the HPL.dat. I keep getting processor errors such as:

HPL ERROR from process # 0, on line 419 of function HPL_pdinfo:
>>> Need at least 8 processes for these tests <<<

HPL ERROR from process # 0, on line 621 of function HPL_pdinfo:
>>> Illegal input in file HPL.dat. Exiting ... <

That is trying to run it on 8 cores across 2 nodes. I have also tried your HPL.dat you provided, and i get a similar error except it says Need at least 12 processes.

Do you know what causes these errors. I have a hosts file in the same directory with the names of the two nodes which i wish to run the tests on.

At the command line i am typing:

mpirun -np 8 -machinefile hosts xhpl_em64t

where hosts file has the names:
machine1
machine2

With each machine being a 3ghz QX6850 Core 2 Extreme (Quad Core), 4GB RAM.

The dat file being uses for two nodes is:

Code:
HPLinpack benchmark input file
Innovative Computing Laboratory, University of Tennessee
HPL.out      output file name (if any) 
8            device out (6=stdout,7=stderr,file)
1            # of problems sizes (N)
29184         Ns
1            # of NBs
128           NBs
0            PMAP process mapping (0=Row-,1=Column-major)
1            # of process grids (P x Q)
2            Ps
4            Qs
16.0         threshold
1            # of panel fact
2            PFACTs (0=left, 1=Crout, 2=Right)
1            # of recursive stopping criterium
4            NBMINs (>= 1)
1            # of panels in recursion
2            NDIVs
1            # of recursive panel fact.
1            RFACTs (0=left, 1=Crout, 2=Right)
1            # of broadcast
1            BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
1            # of lookahead depth
1            DEPTHs (>=0)
2            SWAP (0=bin-exch,1=long,2=mix)
64           swapping threshold
0            L1 in (0=transposed,1=no-transposed) form
0            U  in (0=transposed,1=no-transposed) form
1            Equilibration (0=no,1=yes)
8            memory alignment in double (> 0)
##### This line (no. 32) is ignored (it serves as a separator). ######
0                               Number of additional problem sizes for PTRANS
1200 10000 30000                values of N
0                               number of additional blocking sizes for PTRANS
40 9 8 13 13 20 16 32 64        values of NB



Hoping someone could please help.


Thanks.

Last edited by Neo; 05-11-2009 at 06:16 PM.. Reason: code tags
# 4  
Quote:
That is trying to run it on 8 cores across 2 nodes. I have also tried your HPL.dat you provided, and i get a similar error except it says Need at least 12 processes.

Do you know what causes these errors. I have a hosts file in the same directory with the names of the two nodes which i wish to run the tests on.

At the command line i am typing:

mpirun -np 8 -machinefile hosts xhpl_em64t

where hosts file has the names:
machine1
machine2
Several points:
  • The number of processor-cores you run this thing on must match the product of P and Q. So if P is 2 and Q is 4, you will need 8 cores; no more, no less
  • If you provide MPICH with -np 8 and you specify a machine file, it expects at least that number of hosts in the machine file. If a host has multiple processor-cores (in your case, yes of course), you enter the hostname for each core. So if machine1 has 8 processor cores, your machinefile should include 8 lines of "machine1".
# 5  
Otheus, thank you so much for your responses. I can't wait to test that out when i get to university on Monday. I think my mistake is that i have not put down a name in the machines file for each core.

Hopefully this should work. Sorry for the double post, and will post back to let you know how it goes.
# 6  
Quote:
Originally Posted by otheus
Several points:
  • The number of processor-cores you run this thing on must match the product of P and Q. So if P is 2 and Q is 4, you will need 8 cores; no more, no less
  • If you provide MPICH with -np 8 and you specify a machine file, it expects at least that number of hosts in the machine file. If a host has multiple processor-cores (in your case, yes of course), you enter the hostname for each core. So if machine1 has 8 processor cores, your machinefile should include 8 lines of "machine1".
Hi Otheus, i have tried what you have stated, and i am still getting the error:

HPL ERROR from process # 0, on line 621 of function HPL_pdinfo:
>>> Illegal input in file HPL.dat. Exiting ... <<<

I am using 2 nodes including the head node, so 2 in total. Each of these nodes is a quad core system. So my machine file has this in it:
machine1
machine1
machine1
machine1
machine2
machine2
machine2
machine2

The command execution line i am typing in is:

mpirun -np 8 -machinefile hosts xhpl_em64t

p*q = 8 from my HPL.dat file, where p = 2 , and q = 4.

Yet still i am getting that error. Would you happen to know what else could be wrong?

Thanks.
# 7  
You are getting a different error.
Code:
HPL ERROR from process # 0, on line 621 of function HPL_pdinfo:
>>> Illegal input in file HPL.dat. Exiting ... <<<

Unfortunately, I don't have access to the source code to say what the error is. The HPL.dat file I posted was a suggested sample based on my reading of internet documentation. It could be that documentation is out-of-date or perhaps newer or perhaps just wrong. Check the README and/or sample HPL.dat files that came with you HPL and go through them line-by-line vis-a-vis your own.
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #718
Difficulty: Medium
Alan Minsky invented the first head-mounted graphical display in 1933.
True or False?

9 More Discussions You Might Find Interesting

1. Solaris

Sun cluster 4.0 - zone cluster failover doubt

Hello experts - I am planning to install a Sun cluster 4.0 zone cluster fail-over. few basic doubts. (1) Where should i install the cluster s/w binaries ?. ( global zone or the container zone where i am planning to install the zone fail-over) (2) Or should i perform the installation on... (0 Replies)
Discussion started by: NVA
0 Replies

2. UNIX and Linux Applications

Benchmarking and performance analyzing in OS

Is/Are there an/some application/applications , package/packages for benchmarking or system performance measuring which are there for almost all Linux releases and distributions? (2 Replies)
Discussion started by: nixhead
2 Replies

3. UNIX for Dummies Questions & Answers

benchmarking application

Where i get a open source benchmark program using pthread library for benchmarking our multicore system for the first stage.I need the source code too for that application ,because in later stage we need to develop our application so that i need to study pthread more. please anybody guide me . (0 Replies)
Discussion started by: sujith4u87
0 Replies

4. UNIX for Advanced & Expert Users

Benchmarking a new Solaris, with four different clients

Good morning, for the impatient: I have a new backup-server and need to monitor, what the machine can do, what's the best way of finding that out? I will tell the story right from the beginning, so you have a clue about what's going on: I have a setup of three machines: A new... (6 Replies)
Discussion started by: PatrickBaer
6 Replies

5. High Performance Computing

Tuning HPL.dat for Beowulf Cluster [Linpack]

Hi guys, I am having some issues tuning the HPL.dat file for the Linpack benchmark test across 2 nodes. I am very new to this with minimal Linux experience, however i am trying my luck. The specs for the two nodes are: 3GHZ QX6850 CORE 2 EXTREME (QUAD CORE) 4GB RAM I have been typing these... (1 Reply)
Discussion started by: mercthunder
1 Replies

6. High Performance Computing

MySQL Cluster - Designing, Evaluating and Benchmarking (reg. req'd)

Registration is required. In this white paper learn the fundamentals of how to design and select the proper components for a successful MySQL Cluster evaluation. Explore hardware, networking and software requirements. Work through basic functional testing and evaluation best practices. More... (0 Replies)
Discussion started by: Linux Bot
0 Replies

7. High Performance Computing

Building a Solaris Cluster Express cluster in a VirtualBox on OpenSolaris

Provides a description of how to set up a Solaris Cluster Express cluster in a VirtualBox on OpenSolaris. More... (0 Replies)
Discussion started by: Linux Bot
0 Replies

8. HP-UX

HP-Unix Hardware benchmarking

Hi everyone, I'm working on one HP-Unix application which we have to port completely onto Windows xp. Before that I have to compare performance of two different machines. My HP-Unix is running on HP-C8000 workstation and windows XP machine is intel Xeon. Now the problem is to evaluate the... (0 Replies)
Discussion started by: dgatkal
0 Replies

9. UNIX Benchmarks

Server and Workstation benchmarking

This is from my server AMD K6 133MHz 64Mb RAM 4GB HDD (Maxtor - ATA33) 2x10Mb NIC 1Mb Intel Graphic Card BYTE UNIX Benchmarks (Version 3.11) System -- FreeBSD sergiu.tarnita.net 5.0-RELEASE FreeBSD 5.0-RELEASE #2: Thu Mar 17 15:49:16 EET 2005... (0 Replies)
Discussion started by: Sergiu-IT
0 Replies

Featured Tech Videos