Q1: What percentage of the operations are floating point? Do you need double-precision? (Usually the answer is yes).

What's the expected ratio between computation time and communication time between the pieces. Medium ratio: do some computation, then send intermediate results to all neighbors, then do some more computation. Low ratio: compute, send a result, wait for a message, compute, send a result, and so on. High ratio: the CPUs crunch, crunch, crunch, then finally send results to a central task which does a final computation.

This is important in deciding what kind of network capacity you will need.
[/quote]?

How about reliability? Commodity nodes means high rate of disk failures and/or node failures. Can you bear with frequent filesystem downtime? Or will you need high availability on this filesystem?

Does your budget include life operating costs? Does your server room have specifications for lb/ft^2 ? One institution I worked at discovered that the building was designed for a certain amount of weight density -- even in the server room. It turns out that putting more than about 8 computer racks in the room exceeded this density! So we had the room, but adding more racks might make the floor unstable, especially given that this building was in a geographically active area (about 1 4+ quake every 2 to 3 years).