Hi guys,
I am struggling with adapting my script to increase the performance.
I created a ksh script to process a lot of files in parallel.
I would like to know how can I do in such a way that a constant number of processes is always up (until all is finished).
What I have is (not actual code):
This will start 5 processes, wait for all to finish and then start 5 more, etc.
I would like to always keep 5 processes up as it might be the case one job could finish faster than others.
I would like when one process (FileProcessor ) finishes, to start another (and have maximum 5 up)
Thanks in advance
Moderator's Comments:
Please use code tags next time for your code and data.
Last edited by vbe; 02-14-2013 at 10:36 AM..
Reason: code tags, see PM; sorry zaxxon... I scratched you code tags...added Comments
Unfortunately, it's not working.
The solution with counting the jobs should be fine but ..I don't know why it keeps starting all the jobs instead of maximum 5.
You aren't wait-ing for them when you need to. That's what made my program wait for a process to quit. The 'break' just makes it quit the loop -- which instantly starts over again and creates more. If you want to wait, you must call wait!
You aren't keeping a list of the running processes, either. If you want to wait for one of them, rather than all of them, you need to know which to wait for. That's what the set -- things are for, keeping and adjusting that list. Here's how I'm doing it; set -- changes the $1 $2 ... variables.
Are you running head -5every time to get the fifth line, and such? That's not good... Also, ls *.whatever is completely pointless, * does not need ls's help. There's no point doing a loop from 1 to maxproc either -- loop over the files, use logic to handle maxproc.
Last edited by Corona688; 02-14-2013 at 12:18 PM..
I'm using that head along with tail to get the respective file (for sure there's a better way). E.g.
will give the $i file from the ls list
The first while loop is to go through the number of files but, the increment will not be 1, it will be maxproc.
The break is in order to start only the necessary number of processes at the last iteration (lets say I have 13 files, it will start 5 + 5 + 3)
I tried exactly the code you provided, and it starts 5 parallel processes but, after a short while it will start ALL the rest (luckily I tested with small number of files)
I'm using "Linux illin135 2.6.18-238.12.1.el5 #1 SMP Sat May 7 20:18:50 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux" if it matters.
I tried exactly the code you provided, and it starts 5 parallel processes but, after a short while it will start ALL the rest (luckily I tested with small number of files)
Hello all,
I tried to parralise my treatments but after a while 'ps -ef' display all child process <defunct> (zombie)
Parent bash script to process all files (>100000) in directory:
for filename in /Data/*.txt; do
./child_pprocess.sh $filename &
done
exit(0)I understand that the... (1 Reply)
I am writing a script to kick off a process to gather logs on multiple nodes in parallel using "&". These processes create individual log files. Which I would like to filter and convert in CSV format after they are complete. I am facing following issues:
1. Monitor all Processes parallelly.... (5 Replies)
Hi,
I have a shell script that creates 2 parallel processes. When I press ctrl+c, i want the parallel process to get killed as well.
#!/bin/bash
cmd1="script1.py"
cmd2="script2.py"
${cmd1} &
pid1=$!
echo ${pid1}
${cmd2} &
pid2=$! (7 Replies)
QUESTION: How do I run processes in parallel, so that the counter (in counter.txt) would vary in value (instead of just "0" and "1")? That is, how to not sequentially run inc.sh and dec.sh?
The shared counter (a single number starting as 0) is in a file counter.txt.
counter.sh is (supposed to... (2 Replies)
Hi,
I have a generic FTP script which will be called by 28 different processes in parallel (through a GUI tool) may or may not be at the exact moment (there could be a delay of about a minute or so).
./FTP.ksh 1 (1 through 28)
This script after importing file from remote m/c... (1 Reply)
In a korn shell script, how can I run several processes in parallel at the same time?
For example, I have 3 processes say p1, p2, p3
if I call them as
p1.ksh
p2.ksh
p3.ksh
they will run after one process finishes. But I want to run them in parallel and want to display "Process p1... (3 Replies)