Complicated job distribution


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Complicated job distribution
# 1  
Old 01-18-2010
Complicated job distribution

Hello,

This is going to sound convoluted, however, this is how I need to do things.

I have three awesome machines and a couple hundred jobs. EACH of these jobs takes approximately 4 hours to complete. I want to give jobs to each of these machines, and their respective CPUs in an intelligent way. I basically would like to emulate a cluster, however, I cannot actually use a cluster (we don't want to hook up the machines to our already existing cluster, and don't also want to install a cluster on these machines...as this is really an emergency,hopefully limited one time, manner of usage).

Basically, what I would like to script is:
From one machine: assign jobs to the available CPUs on that machine, ALSO, from the one machine, assign jobs to the CPUs available on the other machines.

It needs to:
1) Apply no more than one job per CPU, and use only a total of 10 CPUs on each machine.
2) Not execute more jobs to a CPU until the job which is working away on it complete.

..as you can see, I basically need to emulate a cluster and it's queueing, deploying machinery...without installing a cluster. All of these machines are on the same network and share the same file system. Is this possible at all?

Thanks.
# 2  
Old 01-19-2010
IMO this is largely OS dependent, so which one are you using?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Complicated string

I have a data file example 10302|77747373|442422442|290209|244242|"234|2352"|92482892 It has about 5000 rows the same way in that field. Needs to look like this.... I need to remove the quotes but the more difficult thing is to remove the pipe between the quotes because there is a pipe... (6 Replies)
Discussion started by: xgringo
6 Replies

2. Solaris

Solaris? Is it really this complicated?

Hello Everybody, I am a mid-level Linux administrator trying to get a hold of Solaris. So far, my experience has been painful. I think it's confusing and it's just a big mess... To be more specific, I have noticed that whenever I install a package... it never followed a specific pattern.... (4 Replies)
Discussion started by: EssenceNY
4 Replies

3. Shell Programming and Scripting

complicated alias command

hi guys i m making one alias which will set variable , invoke sqlplus and also set prompt of sqlplus,,i have made successfully upto invoking sqlplus in unix but cant pass command in sqlplus here is the command alias sett='export ORACLE_SID=devdb2;sqlplus system/system@test' now this... (3 Replies)
Discussion started by: tapia
3 Replies

4. Linux

Complicated Join command!

Hi I have a serious issue when trying to join to files so I have two files, one for meals and one for people Meal1:Turkey:Potato Chips:Twinkie:Coke:5.95 Meal2:Ham & Cheese:Doritos:Cookie:Sprite:6.49 Meal3:Vegetarian:Cheese Crackers:Brownie:Pepsi:5.75 Meal4:Tuna:Cheese Puffs:Eclair:Diet... (1 Reply)
Discussion started by: ehshi1992
1 Replies

5. Shell Programming and Scripting

complicated search within file

Hi, I have following problem. I have a file with time stamps and some data describing what happened between time stamps. Something like this: 10:00 meeting with K meeting with L 11:00 lunch 12:00 work with K 13:00 From this file I have to get a file with... (7 Replies)
Discussion started by: mmike
7 Replies

6. Shell Programming and Scripting

Seems Complicated but would b happy if its possible

Hey guys, I m here again with (may b) a silly query n sorry for that, I am new to UNIX n all so don't kno much @ it. Thro' a script I am generating an output (It is a TXT file) After generating it, I get it to windows n mails it by dragging n dropping it in lotus mail new mail window. Is there... (6 Replies)
Discussion started by: anushree.a
6 Replies

7. Shell Programming and Scripting

Sort complicated two fields

Hi experts, I am trying sort command with my data but still not getting the expected results. For example, I have 5 fields data here c,18:12:45,c,c,c d,12:34:34,d,d,d a,13:50:10,a,a,a b,13:50:50,b,b,b a,13:50:50,a,a,a b,14:10:01,b,b,b c,10:12:45,c,c,c I want to get ... (3 Replies)
Discussion started by: lalelle
3 Replies

8. UNIX for Dummies Questions & Answers

complicated(?) grep

Hi all, I need help to figure out this grep. I would like to search all files in a directory for the occurrence of abc AND xyz. I started with this: grep -i abc * | grep xyz But this only returns lines which contain both values on the same line. So I am thinking that I need to do... (5 Replies)
Discussion started by: khearnen
5 Replies

9. Shell Programming and Scripting

More complicated log parsing

Hey Guys, I am trying to grep within a file to find and output certain parts of lines to other file(s). The output files need to have a dynamic file name based on a field in the main log. The problem is that every line of the log is not the same, and often not even similar. To explain... (25 Replies)
Discussion started by: sjug
25 Replies

10. Shell Programming and Scripting

Very complicated script..

I have a script that I need to create tha involves moving files and renaming them(see previous post) Are there any websites with user made shell scripts? (5 Replies)
Discussion started by: rocinante
5 Replies
Login or Register to Ask a Question