Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Python GNU parallel single command on multiple cores

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Python GNU parallel single command on multiple cores
# 1  
Old 08-29-2016
Python GNU parallel single command on multiple cores
Hello,

I have a 4 core machine.

Here is my initial script

Code:
cd /work/
python script.py input.txt output.txt 1 2 3

This script runs for 1.5hrs.

So I read across the web and figured out that you can use GNU parallel to submit multiple jobs using parallel. But I am not sure if I can run single command on multiple processors. Still, I went ahead and used the following command


Code:
cd /work/
parallel python script.py input.txt output.txt 1 2 3

Here is the error

Code:
parallel: Warning: Input is read from the terminal. Only experts do this on purpose. Press CTRL-D to exit.

# 2  
Old 08-29-2016
Quote:
Originally Posted by jacobs.smith
Code:
cd /work/
parallel python script.py input.txt output.txt 1 2 3

Here is the error
Code:
parallel: Warning: Input is read from the terminal. Only experts do this on purpose. Press CTRL-D to exit.

Hello Jacobs.smith,

Not sure about your complete code/requirement(as you haven't shown us) but seems like it may be code is executing a process into a sub-shell then there it is expecting a variable to be passed rather than it gets passed while running the script.

Thanks,
R. Singh
# 3  
Old 08-29-2016
But the script is designed to take inputs from the terminal itself.

Do you know of a way where I can run a python command on multiple processors instead of one?

Thank you for your time R.Singh.
# 4  
Old 08-29-2016
Can you modify the python script and use the multiprocessing module.
# 5  
Old 08-29-2016
All I can access it is for execution only.

I cannot edit it.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Get count of multiple word in single command

Hello Experts, I have a log file that contains 4 different type of exception : 1- Exception 2- Fatal 3- Error 4- Exec My requirement is to find count of each type of exception, i tried using combination of -E and -C but that doesn't seems to be working : grep -ec 'Exception' -ec... (4 Replies)
Discussion started by: mukulverma2408
4 Replies

2. UNIX for Dummies Questions & Answers

Sftp multiple files in single command

Hi All, I would like to sftp 2 files with a single command. I tried the below options, sftp suer@test13:"/u01/home/oracle/SetDb.sh /u01/home/oracle/.profile" ./ But what actually happens is Fetching /u01/home/oracle/SetDb.sh to /u01/home/oracle/.profile /u01/home/oracle/SetDb.sh ... (3 Replies)
Discussion started by: sid2013
3 Replies

3. UNIX for Dummies Questions & Answers

How to run multiple command in a single line?

Normally i would do this- cd abc ls -ltr I wish to run above command in a single line, like this- cd abc | ls -ltr But above command doesn't works, it simply runs the second command, ignoring the 1st one. :confused: (4 Replies)
Discussion started by: boy18nj
4 Replies

4. UNIX for Dummies Questions & Answers

Gnu parallel and awk syntax quotes

I have many files that has the following structure. CHR SNP BP A1 TEST NMISS OR SE L95 U95 STAT P 1 rs2980319 766985 A ADD 4948 1.068 0.08356 0.9065 1.258 0.7853 0.4323 1 ... (4 Replies)
Discussion started by: genehunter
4 Replies

5. Shell Programming and Scripting

Empty out multiple files with a single command?

I have a log directory: /logs/foo.log /logs/bar.log /logs/err.out I'm trying to find a way to > /logs/*.log > /logs/*.out to blank them out, but of course, that doesn't work. Any suggestions? (4 Replies)
Discussion started by: Validatorian
4 Replies

6. UNIX for Dummies Questions & Answers

Grep multiple strings in multiple files using single command

Hi, I will use below command for grep single string ("osuser" is search string) ex: find . -type f | xarg grep -il osuser but i have one more string "v$session" here i want to grep in which file these two strings are present. any help is appreciated, Thanks in advance. Gagan (2 Replies)
Discussion started by: gagan4599
2 Replies

7. Shell Programming and Scripting

Is there a way to make bash [or another shell] use all CPU cores to execute a single script?

I wrote a very simple script that matches combinations of alphabetic characters (1-5). I want to use it to test CPU speeds of different hardware/platforms. The problem is that on multi-core/processor systems, only one CPU is being utilized to execute the script. Is there a way to change that?... (16 Replies)
Discussion started by: ph0enix
16 Replies

8. Shell Programming and Scripting

Multiple pipes toward a single awk command

Hello, I would like to pipe two variables into awk, but I don't know how to do. Each variable, "a" and "b", are in fact a list of data. They are not files. So to get awk to work with it I am using: echo $a | awk 'FNR==NR{print $1}FNR!=NR{print $4}' The above works, but when I am... (5 Replies)
Discussion started by: jolecanard
5 Replies

9. UNIX for Dummies Questions & Answers

How to run multiple command in single command?

Dear Unix Guru, I have several directories as below /home/user/ dir1 dir2 dir3 Each directory has different size. I want to print each directory size (Solaris command du -hs .) Can you please guide me how to achieve this? Thanks Bala (2 Replies)
Discussion started by: baluchen
2 Replies

10. Shell Programming and Scripting

how to rename multiple files with a single command

Hi I have following list of files at a path: 01.AR.asset 01.AR.index 01.AR.asset.vf 01.AR.asset.xv I want to rename all these files as follows: 73.AR.asset.Z 73.AR.index.Z 73.AR.asset.vf.Z 73.AR.asset.xv.Z Can any body give me a single command to acheive the above results. ... (5 Replies)
Discussion started by: tayyabq8
5 Replies
Login or Register to Ask a Question