Loop over array in batches


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Loop over array in batches
# 1  
Old 05-11-2013
Loop over array in batches

given an array of filenames files=(*) how can i loop over these in batches of four, or even better, work always with four files simultaneously.

i want do something with these files, always four of them simultaneously, and if one is ready the next should start.

one idea, but definitely not the best (not tested):

Code:
 
for i in $(seq 1 4 ${#files[@]})
 do
   ( do whatever with files[$i] ) &
   ( do whatever with files[$(($i+1))] ) &
   ( do whatever with files[$(($i+2))] )  &
   ( do whatever with files[$(($i+3))] )  &
   wait
 done

i think, this will do four tasks simultaneously, wait until all four are ready, and start with the next four files...

will this work?

or how can i start the next task if one of the first four is ready...
# 2  
Old 05-12-2013
what happens if the number of filenames in the list are not evenly divisible by 4?

Try this:
Code:
cnt=1
for fname in *
do
    dosomecommand $fname &
    cnt=$(( cnt + 1  ))
    [ $(( $cnt % 4 )) -eq 0 ]  && wait
done
wait

This is a safer and more general approach. The simple answer to your last question: you would have to develop an application that ran system() calls as worker threads, with a job control thread feeding the worker threads the new jobs as they become free.
This User Gave Thanks to jim mcnamara For This Post:
# 3  
Old 05-13-2013
thank you jim,

i will use your solution...

Quote:
you would have to develop an application that ran system() calls as worker threads, with a job control thread feeding the worker threads the new jobs as they become free.
that's too complicated for me...

dietmar
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script to take cPanel backup in batches

I'm trying to resolve the below scenario by writing bash script. On my managed linux server I have about 30 accounts and there is not enough space to generate full 30 accounts backup in one go and transfer it via SFTP to my Backup Synology Server. What I'm thinking of doing is breaking the... (3 Replies)
Discussion started by: humble_learner
3 Replies

2. Shell Programming and Scripting

awk loop using array:wish to store array values from loop for use outside loop

Here's my code: awk -F '' 'NR==FNR { if (/time/ && $5>10) A=$2" "$3":"$4":"($5-01) else if (/time/ && $5<01) A=$2" "$3":"$4-01":"(59-$5) else if (/time/ && $5<=10) A=$2" "$3":"$4":0"($5-01) else if (/close/) { B=0 n1=n2; ... (2 Replies)
Discussion started by: klane
2 Replies

3. Shell Programming and Scripting

Array Variable being Assigned Values in Loop, But Gone when Loop Completes???

Hello All, Maybe I'm Missing something here but I have NOOO idea what the heck is going on with this....? I have a Variable that contains a PATTERN of what I'm considering "Illegal Characters". So what I'm doing is looping through a string containing some of these "Illegal Characters". Now... (5 Replies)
Discussion started by: mrm5102
5 Replies

4. Shell Programming and Scripting

Array with do while and if loop

Hi All, I am trying to run a do while for an array. And in the do while, I'm trying to get a user response. Depending on the the answer, I go ahead and do something or I move on to next element in the array. So far I can read the array, but I can't get the if statement to work. Any suggestions... (5 Replies)
Discussion started by: nitin
5 Replies

5. Shell Programming and Scripting

Running batches of files at a time from a script

Hi I have a script that performs a process on a file. I want to know how to include a function to run a batch of files? Here is my script #!/bin/bash #---------------------------------------------------------------------------------------------------------------------- #This... (2 Replies)
Discussion started by: ladyAnne
2 Replies

6. Shell Programming and Scripting

Array not surviving while loop

So I'm trying to read datafile into an array, with each line representing one variable in the array. I'm successful at first but somehow it keeps getting erased. i=0 grep '.*' datafile | while read line do echo $i array=$(echo $line) echo ${array} #printing array to make sure it's... (5 Replies)
Discussion started by: DrSammyD
5 Replies

7. UNIX and Linux Applications

Processing batches of files at a time

Hi I would like to run my script to process as many as 50 files at a time. Currently my script is being called like so: ./import.sh -f filename so I want to call my script in this way and it must execute every file in the directory /var/local/dsx/import (1 Reply)
Discussion started by: ladyAnne
1 Replies

8. Shell Programming and Scripting

Help with awk in array in while loop

Hi everyone:) I have 2 files - IN & OUT. Example: IN A:13:30 B:45:40 . . . UNLIMITED OUT Z:12:24 Y:20:15 . . . UNLIMITED I want first row of numbers of IN - OUT. Example 13-12 45-20 My code is (2 Replies)
Discussion started by: vincyoxy
2 Replies

9. UNIX for Dummies Questions & Answers

How to group lines into batches

Hi all I need to group all the lines that start with 2. For every 3 lines (that start with 2) i need to group them in one group and add 1 for each group. I need your help to batch the files content.The file look like this: 040171011140820070000000009650244002933170003000000075272... (2 Replies)
Discussion started by: ashikin_8119
2 Replies

10. Shell Programming and Scripting

Need to know abt script that invokes batches and get d log files if batches failed

hi , I need to know commands to be used in the script to invoke batches in order from other scripts and then run those batches,and how to take those logs of those batches which fails........If anyone give me a better idea to complete this entire task in a single script... (5 Replies)
Discussion started by: gopimeklord
5 Replies
Login or Register to Ask a Question