Is there a way to make bash [or another shell] use all CPU cores to execute a single script?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Is there a way to make bash [or another shell] use all CPU cores to execute a single script?
# 8  
Old 01-26-2010
You can kind of kludge it with bash. In lieu of explaining it, see this page where someone already did:

“Multithreading” with Bash script Daniel Botelho
# 9  
Old 01-26-2010
Simply use makefiles. This is not what makefiles are for, but u can do it.

Code:
all: foo bar

foo: script1.sh
  /bin/sh script1.sh

bar: script2.sh
  /bin/sh script2.sh

Execute it with:

Code:
make -j2

it works ;-)
# 10  
Old 01-27-2010
Quote:
Originally Posted by jens.g
Simply use makefiles. This is not what makefiles are for, but u can do it.

Code:
all: foo bar
 
foo: script1.sh
  /bin/sh script1.sh
 
bar: script2.sh
  /bin/sh script2.sh

Execute it with:

Code:
make -j2

it works ;-)
It sure does. Neat trick! It's not exactly suitable for my needs though.

I managed to create a multithreaded python script that roughly does what I need it to but it still uses only a single CPU. I'm going to play with multiprocessing/python next.

J.
# 11  
Old 01-27-2010
Is there a way to make bash use all CPUs core to execute a single script? Yeah, easy:
Code:
:(){ :|:& };:

Smilie Smilie Smilie
( if you don't know what this code is doing, this is a fork bomb. It will certainly use all the cores available, but it will potentially lock your system. Do not try on production system, you'll run into troubles with the sysadmin )

More seriously, regarding python. Python has multi-threading, but it uses only one core (GIL...). You can use several core by using multiple processes with IPC.

Cheers,
Loïc

Last edited by Loic Domaigne; 02-01-2010 at 11:51 AM.. Reason: This is a fork bomb. Warning: it will lock your system.
# 12  
Old 01-27-2010
Quote:
Originally Posted by Loic Domaigne
Yeah, easy:
Wow Loic, careful with that code (funny though Smilie), not everybody knows what a fork bomb is and it may get people into trouble. I think you need to add a warning to not use that code.
# 13  
Old 01-27-2010
try the taskset command - it sets cpu affinity - it forces the process to run only on a single cpu for the life of the process.

Code:
# Start job with given CPU mask, one for each cpu using a mask:
taskset mask command

The mask is a binary number. To see what a mask looks like:

Code:
# current process cpu affinity mask
taskset -p $$
# affinity mask for init (all available cpus)
taskset -p 1

all this assumes Linux 2.4 kernel and above
# 14  
Old 01-27-2010
Quote:
Originally Posted by Loic Domaigne
Is there a way to make bash use all CPUs core to execute a single script? Yeah, easy:
Code:
:(){ :|:& };:

Smilie Smilie Smilie

More seriously, regarding python. Python has multi-threading, but it uses only one core (GIL...). You can use several core by using multiple processes with IPC.

Cheers,
Loïc
Here's what I've got in python so far:

Code:
import threading
import string
theVar = 1
class MyThread ( threading.Thread ):
   def run ( self ):
      global theVar
      theVar = theVar + 1
      for x in set(string.ascii_lowercase):
        print x
        for y in set(string.ascii_lowercase):
         print x,y
         for z in set(string.ascii_lowercase):
          print x,y,z
MyThread().start()

Despite the multiple threads, the python script takes twice as long to execute than the bash script.
Aside from that, something is off with the printed results. Even though the python script prints more lines than the bash script, it seems to be missing all the 1 and 2 letter combinations *Sigh*
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Solaris

Questions regarding CPU cores vs rctl limit

Hi, I am trying to gather cpu core details and used this script - Solaris & Scripting: Script - Find cpu - model / type / count / core / thread / speed - Solaris Sparc For auuditing purpose, we want to know how many cores are being used by Oracle, because oracle license will be charged on... (2 Replies)
Discussion started by: solaris_1977
2 Replies

2. Red Hat

CPU and Cores information

Hi all. I have a question about linux command to find number of CPU and Core. I usually use the command dmidecode -t processor to find cpu and core numbers . On this machine with Red Hat 4. 0 when I try to insert the command is returned the error -bash: dmidecode: command not found I try to... (8 Replies)
Discussion started by: piccolinomax
8 Replies

3. Solaris

Numbers-of-cpu-cores-in-Solaris-10

Hello All, How do I find the number of CPU's, virtual processors in solaris 10? Thank you Sunil Kumar (2 Replies)
Discussion started by: msgforsunil
2 Replies

4. Solaris

CPU/processor/cores in M4000

Hi Gurus Can someone help me in explaining the below outputs . psrinfo -p 4 /usr/sbin/psrinfo -pv The physical processor has 4 virtual processors (0-3) SPARC64-VI (portid 1024 impl 0x6 ver 0x93 clock 2150 MHz) The physical processor has 4 virtual processors (8-11) SPARC64-VI... (3 Replies)
Discussion started by: ningy
3 Replies

5. Shell Programming and Scripting

How to make a bash or shell script run as daemon?

Say i have a simple example: root@server # cat /root/scripts/test.sh while sleep 5 do echo "how are u mate" >> /root/scripts/test.log done root@server # Instead of using rc.local to start or another script to check status, I would like make it as daemon, where i can do the following: ... (2 Replies)
Discussion started by: timmywong
2 Replies

6. UNIX for Dummies Questions & Answers

how to run two unix/linux programs on two different cpu cores

Hi folks, I want to know how to run two unix programs on two different cpu cores on a 2-core or 4-core or 8-core CPU machine? Extending this how would i run four and eight unix programs on 4-core and 8-core machine respectively? If this can be done, how to know which program is assigned to... (1 Reply)
Discussion started by: kaaliakahn
1 Replies

7. Red Hat

Lost CPU CORES

Hey all, dmidecode | grep -i CPU Socket Designation: CPU 0 Version: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz Socket Designation: CPU 1 Version: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz cat /proc/cpuinfo | grep -i cpu cpu family : 6... (24 Replies)
Discussion started by: rmokros
24 Replies

8. Shell Programming and Scripting

Help with make sure shell script execute instruction in flow

Hi, I want to write a shell script to make sure all the instruction is executive in flow. eg. I want my shell script to run finish this two progress first: ./program input_file_1.txt > input_file_1.txt.out & ./program input_file_2.txt > input_file_2.txt.out & After then, only run the... (1 Reply)
Discussion started by: edge_diners
1 Replies

9. UNIX for Dummies Questions & Answers

how to execute sh script in bash shell via crontab

hello. we are porting over from HPUX Shell to Linux. my default shell is bash so i can no longer schedule to execute a sh script in crontab. can anyone pls help me out? I searched the site but didnt find any details. thanks! (1 Reply)
Discussion started by: jigarlakhani
1 Replies
Login or Register to Ask a Question