Sponsored Content
Top Forums Shell Programming and Scripting (bash) Script Processes in Parallel Post 303019410 by jim mcnamara on Thursday 28th of June 2018 07:29:07 PM
Old 06-28-2018
You cannot create 100000 simultaneous child processes. Limit it to 20. And step through all of your files 20 at a time.

Code:
#/bin/bash   
cnt=0
for filename in /Data/*.txt; 
do
   ./child_pprocess.sh $filename &
    cnt=$(( $cnt + 1 ))
    if [ $(( $cnt % 20 )) -eq 0 ] ; then
       wait    # wait for all 20 to finish
    fi
done
wait   # you can get here with a few processes running, so wait for all of them

This User Gave Thanks to jim mcnamara For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to run processes in parallel?

In a korn shell script, how can I run several processes in parallel at the same time? For example, I have 3 processes say p1, p2, p3 if I call them as p1.ksh p2.ksh p3.ksh they will run after one process finishes. But I want to run them in parallel and want to display "Process p1... (3 Replies)
Discussion started by: sbasak
3 Replies

2. Shell Programming and Scripting

Retention of Variable Value when a script is called by different processes in parallel- Linux 2.6.9

Hi, I have a generic FTP script which will be called by 28 different processes in parallel (through a GUI tool) may or may not be at the exact moment (there could be a delay of about a minute or so). ./FTP.ksh 1 (1 through 28) This script after importing file from remote m/c... (1 Reply)
Discussion started by: dips_ag
1 Replies

3. UNIX for Dummies Questions & Answers

Bash: Regulating the number of processes a script can spawn

I've been working on some scripts in which I spawn some background processes. I'd like to be able to limit the number of processes, but have my script spawn additional processes as previous tasks finish. So, let's say I have 20 tasks to complete. Any given task could take from 1 to 10 minutes. ... (7 Replies)
Discussion started by: treesloth
7 Replies

4. Shell Programming and Scripting

Parallel processes to INC- and DEC-rement shared counter

QUESTION: How do I run processes in parallel, so that the counter (in counter.txt) would vary in value (instead of just "0" and "1")? That is, how to not sequentially run inc.sh and dec.sh? The shared counter (a single number starting as 0) is in a file counter.txt. counter.sh is (supposed to... (2 Replies)
Discussion started by: courteous
2 Replies

5. Shell Programming and Scripting

script for multi-threaded bash processes

hey everyone, I'm having some trouble breaking down some code. It's simple a control script that takes machines meant to be backed up from a list. Then according to that will run multi-threaded processes up until the specified thread limit. for example if there are 4 machines to be backed up,... (2 Replies)
Discussion started by: terrell
2 Replies

6. Shell Programming and Scripting

Deleting all the parallel processes launched when the main script receives a ctrl+c

Hi, I have a shell script that creates 2 parallel processes. When I press ctrl+c, i want the parallel process to get killed as well. #!/bin/bash cmd1="script1.py" cmd2="script2.py" ${cmd1} & pid1=$! echo ${pid1} ${cmd2} & pid2=$! (7 Replies)
Discussion started by: sana.usha
7 Replies

7. Shell Programming and Scripting

Keep up constant number of parallel processes

Hi guys, I am struggling with adapting my script to increase the performance. I created a ksh script to process a lot of files in parallel. I would like to know how can I do in such a way that a constant number of processes is always up (until all is finished). What I have is (not actual... (8 Replies)
Discussion started by: lurkerro
8 Replies

8. Shell Programming and Scripting

Bash script parallel tasks and command to wait untill complete?

Hello, im having bash script with while *** command1 && command2 && command3 && done i want to ask how i can prevent overloading server, by waiting untill all commands complete? any low resources intensive command like "wait" - i dont know if exist? (2 Replies)
Discussion started by: postcd
2 Replies

9. Shell Programming and Scripting

Parallel bash scripts

Need some help to replace bash script with parallel to speed up job on multiple files (400files.list is the file contains the absolute path to those 400 files). The bash script is to run the same program over the files repetitively. My bash_script.sh is: for sample in `cat 400files.list`; do... (3 Replies)
Discussion started by: yifangt
3 Replies

10. Shell Programming and Scripting

Monitoring processes in parallel and process log file after process exits

I am writing a script to kick off a process to gather logs on multiple nodes in parallel using "&". These processes create individual log files. Which I would like to filter and convert in CSV format after they are complete. I am facing following issues: 1. Monitor all Processes parallelly.... (5 Replies)
Discussion started by: shunya
5 Replies
ptree(1)																  ptree(1)

NAME
ptree - print process trees SYNOPSIS
/usr/bin/ptree [-a] [-c] [-z zone] [pid | user] ... ptree prints the process trees containing the specified pids or users, with child processes indented from their respective parent pro- cesses. An argument of all digits is taken to be a process-id, otherwise it is assumed to be a user login name. The default is all pro- cesses. The following options are supported: -a All. Print all processes, including children of process 0. -c Contracts. Print process contract memberships in addition to parent-child relationships. See process(4). This option implies the -a option. -z zone Zones. Print only processes in the specified zone. Each zone ID can be specified as either a zone name or a numerical zone ID. This option is only useful when executed in the global zone. The following operands are supported: pid Process-id or a list of process-ids. ptree also accepts /proc/nnn as a process-id, so the shell expansion /proc/* can be used to specify all processes in the system. user Username or list of usernames. Processes whose effective user IDs match those given are displayed. Example 1: Using ptree The following example prints the process tree (including children of process 0) for processes which match the command name ssh: $ ptree -a `pgrep ssh` 1 /sbin/init 100909 /usr/lib/ssh/sshd 569150 /usr/lib/ssh/sshd 569157 /usr/lib/ssh/sshd 569159 -ksh 569171 bash 569173 /bin/ksh 569193 bash The following exit values are returned: 0 Successful operation. non-zero An error has occurred. /proc/* process files See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWesu | +-----------------------------+-----------------------------+ |Interface Stability |See below. | +-----------------------------+-----------------------------+ The human readable output is Unstable. The options are Evolving. gcore(1), ldd(1), pargs(1), pgrep(1), pkill(1), plimit(1), pmap(1), preap(1), proc(1), ps(1), ppgsz(1), pwd(1), rlogin(1), time(1), truss(1), wait(1), fcntl(2), fstat(2), setuid(2), dlopen(3C), signal.h(3HEAD), core(4), proc(4), process(4), attributes(5), zones(5) 11 Oct 2005 ptree(1)
All times are GMT -4. The time now is 08:22 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy