Some of your questions are so vague that it is hard to make any informed suggestions. How would you respond if you got a request from someone to tell them how to choose the best vehicle? (Who is going to be driving it? How many passengers do you need to carry? How much weight do you need to be able to tow? How much secured cargo space do you need? What are the weather conditions where it will be driven? What type of terrain does it need to traverse? ...)

I know very little about about ME and nothing about Ansys CFD. Are you trying to build a cluster to support hundreds of users submitting thousands of jobs? Are you trying to build a cluster than can break a single huge job into thousands of threads and run all of those threads simultaneously? Do you have any experience writing thread-safe code?

Can you use only open-source software? Of course you can! You can write all of the code you need and make it available for everyone to use as they see fit.

Does open-source software already exist for all of the code you want to run? How can we guess at that from what you've told us? We have no idea what all of the code you want to run needs to do.

If you don't know the difference between a heterogeneous cluster and a homogeneous cluster, you probably don't have the background needed to design the cluster you want. Please consider hiring an architect with experience setting up and running an HPC data center who you can sit down with and discuss budget, capabilities, computing projects to be run, users to be supported, software to be run, software to be written, etc., etc., etc. Setting up an HPC data center is a very complex, expensive undertaking.