Multiprocessing Help Post: 302724863

Sponsored Content

Top Forums UNIX for Dummies Questions & Answers Multiprocessing Help Post 302724863 by bakunin on Thursday 1st of November 2012 11:38:23 AM

11-01-2012

Registered User

Quote:

Originally Posted by Tyler_92

My question is this: The computer I'm using has 12 cores, each with two virtual cores, so I am working with 24 virtual processors. Is there a way to either 1) assign different jobs to different processors, or 2) get all of the processors to collaborate on one job? Does this happen automatically?

Ha - i never thought my work with massively parallel computers would ever be of help here!

In principle, this is possible. The whole point of having several CPUs instead of one is to work on several tasks at the same time - in parallel.

There are some restrictions to this, though: first, as my venerable colleague Corona688 already told you, 24 virtual processors are not 24 real processors. But even setting this point aside, there are two classes of problems:

Suppose i give you a number and ask you to multiply it by 3. It will take you a certain time to compute the result. Now, if i give 50 of these numbers to 50 people, the time to come up with the answer will be more or less the same (save for some slack), because every one of them will compute one result only. This is a problem which "scales" very good.

Now let us alter the problem a bit: i give you a number and ask you to multiply it by 3, the result again by 3, and so forth, 50 times. This time 50 people wouldn't help at all because every result is depending on the previous result and therefore computation of the final result will take the same time, regardless of how many people will work on it. This is a very bad scaling problem.

Real world problems are somewhere in between these extremes and usually contain parts which scale good and parts which scale less good. The art of programming massively parallel computers is to identify the good scaling parts and optimize them to the utmost possible.

OK, so far for theory, which will not directly solve your problem. It answers the second part of your question, though: if all the processors can be brought to work together on one analysis depends - it depends on how well the program doing the analysis was written with SMP in mind. As we probably don't know anything about your program you will be left to simply test it. Try it on a two-processor system (1 processor for the OS to eliminate effects from this side) and on your 12-processor system and look if there are any processing time differences.

I am no OSX expert, but as far as i know setting processor affinity is not supported. It might work to use OpenGrid or similar products, but honestly, i have no idea if this even runs under OSX. Maybe someone else can fill the gaps here. You might consider transferring the analysis to a Linux-Workstation, where such tools indeed exist.

I hope this helps.

bakunin

bakunin

View Public Profile for bakunin

Find all posts by bakunin

4 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Multiprocessing under Linux

I'm writing C programs to be executed on a multi-processor UNIX (GNU/Linux, kernel 2.6.11) Do I need to add a special kind of code to somewhere or run a special utility to execute the program file to be executed by all processors? Or is it handled automatically by kernel?

2. Programming

Multiprocessing multipointers

I have a complex problem..... I have to search files on directory "text files" then search on all of them for a word or sentence....the user inter my problem is,,,, if I want to create a child for each file...and point a file by pointer to search...and I don't know how much files i have in...

3. Shell Programming and Scripting

Multiprocessing in Python

Hi there, I have a code that can take in any function with two arguements and do processing. However, I would like to implement a feature whether it can limit a number of process running concurrently so as not take up too much resources. I have tried researching for pool.map however I am unable...

4. Shell Programming and Scripting

<< Threading inside multiprocessing using queues >>

Hi All, I am trying to achieve threading inside each process of multiprocessing. I have 2 queues one for multiprocess (process) & another inside each process. when i execute it got hung after below output. My goal here is to go through p_source queue & for each process picks up all t_source...

LEARN ABOUT DEBIAN

ncftpbatch

ncftpbatch(1)						      General Commands Manual						     ncftpbatch(1)

NAME

       ncftpbatch - Individual batch FTP job processor

SYNOPSIS

       ncftpbatch -d

       ncftpbatch -l

       ncftpbatch -D

OPTIONS

   Command line flags:
       -d      Begin  background  processing  of FTP jobs in the current user's $HOME/.ncftp/spool directory.  This returns immediately, because a
	       daemon process is spawned and ran in the background.

       -l      Lists the contents of the user's job queue.

       -D      This is like -d, except that the process does not become a daemon.

DESCRIPTION

       This program is responsible for processing background FTP requests.  It is normally only run by ncftp and not manually by  a  human  being,
       however you can run it to manually process the FTP job queue.

       The  jobs are spool files written to a user's $HOME/.ncftp/spool directory and have a special format and file-naming convention (which con-
       tains when the job is to be run).  ncftp runs this program when it needs to, but if the ncftpbatch daemon dies unexpectedly the	jobs  that
       are left in the queue will not be processed until another instance of ncftpbatch is run.

       ncftpget  and  ncftpput can also be used to submit jobs for batch processing, using those utilities' -b command-line flag.  If desired, you
       can also manually create the spool files although this procedure is not documented here (see the manual	page  for  ncftpspooler  for  more
       information on how to do that).

DIAGNOSTICS

       ncftpbatch  writes  to its own log file, the $HOME/.ncftp/spool/log file.  This file should be examined to determine if any ncftpbatch pro-
       cesses are actively working on jobs.

AUTHOR

       Mike Gleason, NcFTP Software (http://www.ncftp.com).

SEE ALSO

       ncftp(1), ncftpput(1), ncftpget(1).

ncftpbatch							  NcFTP Software						     ncftpbatch(1)