Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Start process on X number of files and then wait for the next batch
Top Forums Shell Programming and Scripting Start process on X number of files and then wait for the next batch Post 303034554 by busyboy on Monday 29th of April 2019 10:17:13 AM
Old 04-29-2019
Start process on X number of files and then wait for the next batch
Thanks for RudiC for his extraordinary help on organizing files in a batch of 10 using below code.

Code:
FL=($(ls)); 
for ((i=0;i<=${#FL[@]};i++)); do 
for j in ${FL[@]:$i:10};
do 
$batch  ${j}  ${j}.txt
done; 
echo "Pausing for next iteration";
echo "------------------------------------------------------------------------------------"; 
((i+=10)); 
done

Problem:

I want to run the batch on 10 files ( which takes a lot of time ) and then once the batch is finished , I want to execute the next iteration

if I do something like this,


Code:
for j in ${FL[@]:$i:10};
do 
$batch -in  ${j} -out  ${j}.txt &
done;

the problem with this approach is that every batch process is sent to background and I can't find if there are any processes running in the background for the current session so that I can start the next session.


Alternatively, if I run like this, which is again a sequential approach and doesn't fulfill the requirement of parallel run.

Code:
$batch -in  ${j} -out  ${j}.txt && $batch -in  ${j} -out  ${j}.txt && $batch -in  ${j} -out  ${j}.txt && $batch -in  ${j} -out  ${j}.txt && $batch -in  ${j} -out  ${j}.txt && $batch -in  ${j} -out  ${j}.txt

Regards,
NMR
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Process Wait on DG UX

Does anyone know what the equivalent command to pwait on Solaris is on DG/UX. I need my script to kick off a process and wait till it is complete before continuing with the script. (4 Replies)
Discussion started by: fabbas
4 Replies

2. UNIX for Dummies Questions & Answers

Append value(batch number) to start of records

Hi all, I am new to unix shell scripting and I am trying to append batch number that comes in Trailer record to the detailed record. TR|20080312|22881 |000000005|20080319|2202 LN|20080312|077777722220 |0000100000017|ABS LN|20080312|000799439326 |0000709943937|AA TR|20080313|22897 ... (6 Replies)
Discussion started by: kiran_418
6 Replies

3. Shell Programming and Scripting

wait command - cat it wait for not-chile process?

Did not use 'wait' yet. How I understand by now the wait works only for child processes, started background. Is there any other way to watch completion of any, not related process (at least, a process, owned by the same user?) I need to start a background process, witch will be waiting... (2 Replies)
Discussion started by: alex_5161
2 Replies

4. UNIX for Advanced & Expert Users

wait process

can any one please give me clear idea of wait process in UNIX system. I am using AIX 5.3 and see loots of wait process. I have very basic concept of wait process. If CPU has nothing to do then a wait process is generated per CPU. But i want know the detail how is it forked. Is wait a jombe... (2 Replies)
Discussion started by: pchangba1
2 Replies

5. Shell Programming and Scripting

Ksh Script to get the start minute of n number of process

How can i write a script.? which lists all X process and gets the start minute of each of them. thanks (1 Reply)
Discussion started by: Anteus
1 Replies

6. Solaris

Number of files - in start of year

Is there any way to find "Number of files" that exists on my solaris parition in the starting of 2009 year ? I know ctime or mtime will not help and unix wouldnt store creation time. Only hope i can see ( and i am not sure if that will help ) is that my system is up from last 2 years without... (5 Replies)
Discussion started by: rajwinder
5 Replies

7. Shell Programming and Scripting

How to make the parent process to wait for the child process

Hi All, I have two ksh script. 1st script calls the 2nd script and the second script calls an 'C' program. I want 1st script to wait until the 'C' program completes. I cant able to get the process id for the 'C' program (child process) to make the 1st script to wait for the second... (7 Replies)
Discussion started by: sennidurai
7 Replies

8. Shell Programming and Scripting

Using Make to batch process files

Hello all, I have a make question, and I was hoping somebody here might be able to point me in the right direction. Here is my issue; I have a command-line tool that I use to run a conversion on an input XML file that results in an output binary file. However, this particular tool needs to... (1 Reply)
Discussion started by: jujumbura
1 Replies

9. Shell Programming and Scripting

Executing a batch of files within a shell script with option to refire the individual files in batch

Hello everyone. I am new to shell scripting and i am required to create a shell script, the purpose of which i will explain below. I am on a solaris server btw. Before delving into the requirements, i will give youse an overview of what is currently in place and its purpose. ... (2 Replies)
Discussion started by: goddevil
2 Replies

10. Windows & DOS: Issues & Discussions

How to start a vbs from a windows batch file?

Morning, I'm trying to execute a vbs from a .bat file. Can someone tell me what the difference is between these statements: start c:\lib\runit.vbc c:\lib\runit.vbs When I run the batch with the 'start' parameter it doesn't seem to do anything. (1 Reply)
Discussion started by: Grueben
1 Replies

LEARN ABOUT DEBIAN

ecaccess-queue-list

ECACCESS-QUEUE-LIST(1p) 				User Contributed Perl Documentation				   ECACCESS-QUEUE-LIST(1p)

NAME

       ecaccess-queue-list - List available queues

SYNOPSIS

       ecaccess-queue-list -version|-help|-manual

       ecaccess-queue-list [-debug] [queue-name]

DESCRIPTION

       Two different types of queues are in use with ECaccess:

       1 - The ECaccess batch queues, which correspond to one system at ECMWF with its specific batch environment, e.g. ecgate will be the
       ECaccess queue, which will redirect jobs to ecgate at ECMWF, running LoadLeveler. This ECaccess queue will be given as argument when
       submitting a batch job.

       2 - The batch queues (or classes) on the systems at ECMWF will be given in the batch job with #@ class = for LoadLeveler.

       The names of the ECaccess queues and associated batch queues at ECMWF can be seen with this command. When an ECaccess queue-name is
       specified then the details for this queue are displayed (including the list of classes).

       The first ECaccess batch queue in the list is the default queue when submitting a batch job with the ecaccess-job-submit command.

ARGUMENTS

       queue-name (optional)
	       The name of the ECaccess queue to retrieve the details.

OPTIONS

       -version
	       Display version number and exits.

       -help   Print a brief help message and exits.

       -manual Prints the manual page and exits.

       -debug  Display the SOAP messages exchanged.

EXAMPLES

       ecaccess-queue-list

       List all the ECaccess batch queues (or systems supporting job submission at ECMWF).

       ecaccess-queue-list ecgate

       List all the batch queues (or classes) on the ECMWF system ecgate.

SEE ALSO

       ecaccess.

perl v5.14.2							    2012-04-16						   ECACCESS-QUEUE-LIST(1p)
All times are GMT -4. The time now is 03:34 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy