Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Run rest of script after parallel steps have completed

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Run rest of script after parallel steps have completed
# 1  
Old 10-05-2017
Run rest of script after parallel steps have completed
Hello gurus,

I produce a number of .loc files in parallel depending on number of .csv in the folder

Code:
for file in *csv
do
./process.sh $file > $file.loc &
done

then I want to compile all the output from the previous step into a single masterlocfile and then perform the rest of the steps
Code:
cat *.loc > masterlocfile

How can I make sure all the .loc files have been completely created (the for loop has ended) before moving onto the next steps including compilation?

I got some answers from google that involves a process id but I will have several different process id from the for loop. Please help.
# 2  
Old 10-05-2017
Hi.
Quote:
$ type wait help ; help wait
wait is a shell builtin
help is a shell builtin

wait: wait [-n] [id ...]
Wait for job completion and return exit status.

Waits for each process identified by an ID, which may be a process ID or a
job specification, and reports its termination status. If ID is not
given, waits for all currently active child processes, and the return
status is zero. If ID is a a job specification, waits for all processes
in that job's pipeline.

If the -n option is supplied, waits for the next job to terminate and
returns its exit status.

Exit Status:
Returns the status of the last ID; fails if ID is invalid or an invalid
option is given.
On a systerm like:
Code:
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.9 (jessie) 
bash GNU bash 4.3.30

Best wishes ... cheers, drl
These 2 Users Gave Thanks to drl For This Post:
MadeInGermany rbatte1
# 3  
Old 10-06-2017
Beware that if you have many files that you might hit processing limits (such as number of process per user, total process on the system etc.) so might need to break down your list into smaller chunks.

Is this likely to be the case are we looking at 100 or less files?



Robin
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Loop to run commands - after the previous instance completed

Hi All, I am trying to call a shell script in a loop. I want my first instance to complete, and then the 2nd instance of the command to start - and so on. eg. I am running this script 30 times. The wrapper script needs business date, from_time,to_time & server_name as inputs. script_name... (2 Replies)
Discussion started by: neil.k
2 Replies

2. UNIX for Advanced & Expert Users

Writing a REST server to run on FreeBSD -- how to structure for concurrency?

Hi All, I want to write a domain specific REST/HTTP server to run on FreeBSD. I have control over both the server side and the primary client intended to consume the service. My question is: how do you think it should be designed to support multiple connections, given: It will run on a... (6 Replies)
Discussion started by: LittleCookieMon
6 Replies

3. Shell Programming and Scripting

Run script in parallel in while loop

Hi I am running a loop which actually runs same script for different argument value passed to it. while read repID do echo "Starting for $repID"; date; perl process_report.pl $repID done<${FILE_TO_READ} However this runs in sequence. I want the loop to not to wait for perl to... (3 Replies)
Discussion started by: dashing201
3 Replies

4. Shell Programming and Scripting

Run a script in parallel

Hey, I am new to UNIX scripting . I have script (ex: start_script) that starts a job in 10 different servers one server after another.Now I want to modify the script so that the script starts the job in all servers parallely (at a time in all servers).and I need the choice of selecting the... (3 Replies)
Discussion started by: mpspsm
3 Replies

5. Shell Programming and Scripting

Run in series and Parallel

I have a list with four dates say load_date.lst contains 2010-01-01 2010-01-31 2010-03-01 2010-03-31 2010-05-01 2010-05-31 2010-07-01 2010-07-31 And I have directory /lll/src/sql with set of sql's 1_load.sql 2_load.sql 3_load.sql I want to run the sql'in series with respective to... (3 Replies)
Discussion started by: sol_nov
3 Replies

6. Shell Programming and Scripting

script - how to prevent in parallel run

I have one shell script which is being accessed by many jobs at same time. I want to make the script such that , other job should wait for the script if script is being used by some other job. Is there any way to implement it in script level ? Gops (1 Reply)
Discussion started by: Gopal_Engg
1 Replies

7. Shell Programming and Scripting

Need to run same script multiple times in parallel

Hi all, I have a requirement in which a script invokes a Java program. Lets say script ABC invokes a java program with cfg file a parameter. This script takes 10 minutes to execute . Like this ineed to run the program 10 times meaning 100 minutes if i do it sequentially. If i open... (2 Replies)
Discussion started by: rahman_riyaz
2 Replies

8. Shell Programming and Scripting

Run a command in parallel

Hi all, How do i run a command in parallel 50 times and capturing the result of each run in a separate file Eg: myApp arg1 > run1.txt myApp arg1 > run2.txt ::::::::::::::::::::::::: ::::::::::::::::::::::::: myApp arg1 > run50.txt The above way is sequential. ... (3 Replies)
Discussion started by: jakSun8
3 Replies

9. UNIX for Advanced & Expert Users

Run a script parallel for a month's worth:

Is there a utility that can be used in a shell script that would run a .sql file for 30 or 31 days in a month at the same time parallely. Please Advice. Thanks SD12. (2 Replies)
Discussion started by: sd12
2 Replies

10. Shell Programming and Scripting

Run a same script in parallel with diffs parameters

i have script say some_script.ksh that takes an argument I need to run some_script.ksh in background parallely at the sametime with different arguments. Once all the background jobs complete, i need to run this script again in parallel with another 5 set of arguments. Would really... (1 Reply)
Discussion started by: hyennah
1 Replies
Login or Register to Ask a Question