Quote:
Originally Posted by
gmark99
In my case, "my_job" does some database queries and decides to call a function that will result in a cross-processor message and then wait for a result to be returned.
I would still benefit from a lot of jobs in parallel if they're just waiting for results from another machine, right?
Well, how many queries can your database server
really run at the same time? There's no point exceeding that. 5,000 processes sitting around waiting for the database server are no faster than 5 processes sitting around waiting for the database server.
Also, what exactly does your query do? Might it create lots of temporary tables, consume lots of memory, ask for things on opposite ends of the disk, lock tables so everyone else has to wait, etc? If you push any of your server's limits, be they bandwidth, disk, or CPU, you lose.
Spamming hundreds of network connections causes its own bottlenecks, too.
Quote:
Am I missing anything, or am I on the right track?
I think you should look into having one job doing n queries, instead n jobs doing 1 query. There's a lot of overhead in making and breaking connections over and over. Besides, hammering any kind of server like that is just rude. If you can do that, try 2 jobs doing n/2 queries and see if that's an improvement or not.