While loop - how to run processes one after another (2nd starts after first completes, and so on)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting While loop - how to run processes one after another (2nd starts after first completes, and so on)
# 1  
Old 02-19-2013
While loop - how to run processes one after another (2nd starts after first completes, and so on)

I'm a programming noob. I'm trying to run a memory intensive process for many files. But when I use the following script, it runs fine for the first 5-7 files, then runs out of memory. Monitoring the output files, it's clear the processes are going on in parallel. Once 5-7 of the files are being run at once, I get a crash. I'd like to run the script so that the first process completes before the next one starts (or the effective equivalent). Any pointers?

Code:
#!/bin/bash

i=1

while [ $i -le 197 ]
do
my.program.command.here
i=$[$i+1]
done


Last edited by Scrutinizer; 02-19-2013 at 06:17 PM.. Reason: code tags
# 2  
Old 02-19-2013
The loop runs as one process, line by line.
What exactly is
Quote:
my.program.command.here
?
It must not end with a & character.

You can mitigate your problem by inserting a sleep command in the while loop.
# 3  
Old 02-19-2013
Sorry, I didn't think the program was relevant. I'm running software called usearch.

Code:
#!/bin/bash

i=1

while [ $i -le 197 ]
do
usearch -usearch_global file.$i -db db.file -strand both -id 0.5 -userout output.$i.tab 
i=$[$i+1]
done

I hadn't heard of the sleep command. So something like this would delay the processing of file.2 by 45 seconds from the initiation of the program on file.1?

Code:
#!/bin/bash

i=1

while [ $i -le 197 ]
do
usearch -usearch_global file.$i -db db.file -strand both -id 0.5 -userout output.$i.tab 
sleep 45
i=$[$i+1]
done

# 4  
Old 02-19-2013
Does your file spin off a child and then exit, so it creates a parallel situation. Sometimes you can stifle that by attaching all the child stdout and stderr until they close:
Code:
usearch -usearch_global file.$i -db db.file -strand both -id 0.5 -userout output.$i.tab 2>&1 | cat -u

No sleep is necessary if they run one at a time.
# 5  
Old 02-19-2013
I've tried both of these and for whatever reason neither has worked. Using sleep allowed the process to get through 12 files, but it crashed on the 13th. Without sleep it gets through 5-6 files. The output solution didn't seem to make a difference.
# 6  
Old 02-20-2013
Unless you present more details about what usearch does and how it works, we (and you) are doomed.
One - very inelegant - workaround would be to create nested loops (1 <= i <= 20) and (1 <= j <= 5) and run five instances of usearch at a time, wait for them to finish, and then go on with the next group.
# 7  
Old 02-20-2013
Run the first 5 usearch commands, then run
Code:
ps -eo rss,vsz,pid,comm | sort -n | tail
ps -eo pcpu,pid,comm | sort -n | tail

to present us the top 10 processes, and
Code:
uname

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need to write a shell script that starts one, then kills it, then starts another?

This is on a CentOS box, I have two scripts that need to run in order. I want to write a shell script that calls the first script, lets it run and then terminates it after a certain number of hours (that I specify of course), and then calls the second script (they can't run simultaneously) which... (3 Replies)
Discussion started by: btramer
3 Replies

2. Homework & Coursework Questions

When I run the script, the cursor starts on the wrong line?

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: It's a shell script using a looping logic, trap, tput, if, while. Most of the scripts in this book aren't written... (2 Replies)
Discussion started by: ckleinholz
2 Replies

3. Shell Programming and Scripting

Array Variable being Assigned Values in Loop, But Gone when Loop Completes???

Hello All, Maybe I'm Missing something here but I have NOOO idea what the heck is going on with this....? I have a Variable that contains a PATTERN of what I'm considering "Illegal Characters". So what I'm doing is looping through a string containing some of these "Illegal Characters". Now... (5 Replies)
Discussion started by: mrm5102
5 Replies

4. UNIX for Dummies Questions & Answers

Maximum no of processes that can run

Is there any limit in UNIX that only a MAXIMUM no of processes can run at a time? If so, it exceeds then what would be the error we would receive? (4 Replies)
Discussion started by: jansat
4 Replies

5. UNIX and Linux Applications

how to run more than two processes parallely

I would like to call a function called CIRCLE which is further beind called by other function but in a loop that CIRCLE fuction is being called. And this CIRCLE function starts another process which takes 3 hours to complete again, if i put that process in nohup &, I can go to the next command... (2 Replies)
Discussion started by: venugopalsmartb
2 Replies

6. UNIX for Dummies Questions & Answers

Two questions. First one; What are the ways in which a program starts to run.

This is my first post here so hello everyone! I know that a command of the programs name can start a program and clicking on a icon in GUI can as well as a startup shell script but how do I educate myself of the method that starts an application? Does the GUI run a script? What are the ways/ way... (2 Replies)
Discussion started by: theKbStockpiler
2 Replies

7. UNIX for Dummies Questions & Answers

how can I run something as root (modprobe, to be exact) every time computer starts.

I have the root password for my box, but I'm ignorant. So, every time I start my computer, I have to run this command /sbin/modprobe fuse as su, so that I can do other stuff (like mount remote directories locally using sshfs) I guess there's some file, like .bashrc, only it's applicable... (4 Replies)
Discussion started by: tphyahoo
4 Replies

8. UNIX for Advanced & Expert Users

script to run different shells which run different processes

Hi, Would like to ask the experts if anyone knows how to run a script like this: dtterm -title shell1 run process1 on shell1 dtterm -title shell2 run process2 on shell2 cheers! p/s: sorry if i used the wrong forum, quite concussed after watching world cup for several nights; but I... (2 Replies)
Discussion started by: mochi
2 Replies

9. UNIX for Advanced & Expert Users

Run away processes

Hi, My server runnning on SUN Solaris rel.5.5.1. I have been facing this issues for years. I have some Xbase databases running on the server. User is using emulation software to telnet to server for accessing application. If user logout application abnormally - by closing windows session, then... (2 Replies)
Discussion started by: lowtaiwah
2 Replies

10. Shell Programming and Scripting

How to run processes in parallel?

In a korn shell script, how can I run several processes in parallel at the same time? For example, I have 3 processes say p1, p2, p3 if I call them as p1.ksh p2.ksh p3.ksh they will run after one process finishes. But I want to run them in parallel and want to display "Process p1... (3 Replies)
Discussion started by: sbasak
3 Replies
Login or Register to Ask a Question