UNIX loop question


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers UNIX loop question
# 1  
Old 01-07-2015
UNIX loop question

Dear all,

I have a question regarding unix loops. I want to run 100 commands using file1.txt-file100.txt in parallel. I use the code below and it works well. But now I want to run first 20 commands first using file1.txt-file20.txt in parallel, then when they are completed, run the next 20 commands in parallel, then the next 20 commands ... and so on. But I don't know how to modify the code to do that. Does anyone know how to write the code? Thank you for your time and help in advance.

Code:
for snplist in $(seq 1 100); do plink --bfile mydata  --extract file${snplist}.txt --make-bed --out file${snplist}& done

thanks,
Lin

Moderator's Comments:
Mod Comment edit by bakunin: please use CODE-tags for code, nothing else. Thank you.

Last edited by bakunin; 01-07-2015 at 12:18 PM..
# 2  
Old 01-07-2015
Quote:
Originally Posted by forevertl
But now I want to run first 20 commands first using file1.txt-file20.txt in parallel, then when they are completed, run the next 20 commands in parallel, then the next 20 commands ... and so on.
I sense a slight ambiguity here and i want to clarify it before anything else:

do you want to wait for the completion of all the 20 commands before you start the next batch of twenty or do you want to start a new one any time one finishes so that always 20 jobs are running in parallel? Both is possible, but done differently.

Btw., the construction with "for" and "seq" is prone to problems and quite depcrecated, as modern shells all have to possibility to do integer math themselves. The following should work in any modern shell and do the same as your for-loop, just more robustly:

Code:
typeset -i snplist=1

while [ $snplist -le 100 ] ; do
     <your command(s) here>
     (( snplist = snplist + 1 ))
done

This User Gave Thanks to bakunin For This Post:
# 3  
Old 01-07-2015
thank you for your reply. either way is fine for me. whichever has code easier to understand for beginners is fine for me. Since each command will take similar time, so waiting all 20 command are done, then start next 20 commands is fine for me.
thank you for your time and help!
Lin
# 4  
Old 01-07-2015
Assuming that you have no other background processes, you can use the wait command to ensure all the 20 that you have just started have completed, a bit like this:-
Code:
for offset in 0 20 40 60 80
do
   i=1
   while [ $i -le 20 ]
   do
      ((value=$offset+$i))
      <run a single command here using $value> &
      ((i=$i+1))
   done
   wait
done

All untested, but it might help.


Robin
# 5  
Old 01-07-2015
If your goal is to run all 100 jobs but not more than 20 in parallel,
then check out my solution in this previous thread
Adapt it for your needs, e.g. maxjobs=20
This User Gave Thanks to MadeInGermany For This Post:
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

If loop question

Hi I have script if ; then echo "Found Node" else echo "Sorry this is not a node" I need to add conditions "nn" ,"rm", and "hh" in the same line along with "hd" .Can this be done or do I need to use elif (1 Reply)
Discussion started by: Dublin4320
1 Replies

2. UNIX for Dummies Questions & Answers

Question about for loop

Hi, I have code like below disk_list=$(ls /root/file1) for disk in $disk_list do pvcreate $i done I know what pvcreate command does, but I do not understand what this $i do here. can someone please explain. (2 Replies)
Discussion started by: stew
2 Replies

3. UNIX for Dummies Questions & Answers

Loop question

Hi, I would really appreciate some help with manipulating a file. Here is my input file: SNP_1 : 1 SNP_11 : 1 SNP_2 -0.12 1 SNP_12 -0.3472 1 SNP_3 -0.708 1 SNP_13 0.5037 1 SNP_4 1.492 0.5 SNP_14 -0.0685 1 SNP_5 1.27 0.5 SNP_15 0.547 ... (3 Replies)
Discussion started by: zajtat
3 Replies

4. Shell Programming and Scripting

While loop Question

Hi, I have a requirement where I have to check for 10 files in Unix location. If all of the files present in that directory then i have execute another process. can you help me resolving this issue. sample location /home/usr abc.txt cde.txt aaaa.txt lll.txt ooo.txt if all the... (3 Replies)
Discussion started by: manasvi24
3 Replies

5. Shell Programming and Scripting

For Loop Question

I am struggling with the for loop. I have a file name heros.txt and I would like to go through a list in file where.txt and see if I can find the name from where inside heros. One of the problems that I am having is I dont understand how to setup the for loop to find the list to search.:wall: ... (6 Replies)
Discussion started by: captaindoogles
6 Replies

6. Shell Programming and Scripting

BASH loop inside a loop question

Hi all Sorry for the basic question, but i am writing a shell script to get around a slightly flaky binary that ships with one of our servers. This particular utility randomly generates the correct information and could work first time or may work on the 12th or 100th attempt etc !.... (4 Replies)
Discussion started by: rethink
4 Replies

7. Shell Programming and Scripting

loop question

HI ALL How to read first 10 line from a file using any method (5 Replies)
Discussion started by: aliahsan81
5 Replies

8. Shell Programming and Scripting

For Loop Question

Hi, I am writing a bash shell script. I will have a section of code devoted to a for loop: for FILE in *.hdf *.L2; do echo hello world done The complication is that sometimes there will not be any files with a ".hdf" extension, or sometimes there won't be any with a ".L2" extension. Since... (4 Replies)
Discussion started by: msb65
4 Replies

9. Shell Programming and Scripting

while loop question

while do print What is the next device number to be added to $dgroup? print Press \<Enter\> if there are no more devices to be added. read dev_num export dev_num symld -g $dgroup -sid $sname add dev $dev_num done In this while... (2 Replies)
Discussion started by: stepnkev
2 Replies

10. Shell Programming and Scripting

Loop question

hi, how would i go about making a loop which gets each line from a single text file, set it to a variable and then print it to screen? thanks eg: #!/bin/sh FILE="somefile.txt" text_line="" what kind of loop would use here? (18 Replies)
Discussion started by: strike
18 Replies
Login or Register to Ask a Question