Sponsored Content
Top Forums Shell Programming and Scripting (bash) Script Processes in Parallel Post 303019410 by jim mcnamara on Thursday 28th of June 2018 07:29:07 PM
Old 06-28-2018
You cannot create 100000 simultaneous child processes. Limit it to 20. And step through all of your files 20 at a time.

Code:
#/bin/bash   
cnt=0
for filename in /Data/*.txt; 
do
   ./child_pprocess.sh $filename &
    cnt=$(( $cnt + 1 ))
    if [ $(( $cnt % 20 )) -eq 0 ] ; then
       wait    # wait for all 20 to finish
    fi
done
wait   # you can get here with a few processes running, so wait for all of them

This User Gave Thanks to jim mcnamara For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to run processes in parallel?

In a korn shell script, how can I run several processes in parallel at the same time? For example, I have 3 processes say p1, p2, p3 if I call them as p1.ksh p2.ksh p3.ksh they will run after one process finishes. But I want to run them in parallel and want to display "Process p1... (3 Replies)
Discussion started by: sbasak
3 Replies

2. Shell Programming and Scripting

Retention of Variable Value when a script is called by different processes in parallel- Linux 2.6.9

Hi, I have a generic FTP script which will be called by 28 different processes in parallel (through a GUI tool) may or may not be at the exact moment (there could be a delay of about a minute or so). ./FTP.ksh 1 (1 through 28) This script after importing file from remote m/c... (1 Reply)
Discussion started by: dips_ag
1 Replies

3. UNIX for Dummies Questions & Answers

Bash: Regulating the number of processes a script can spawn

I've been working on some scripts in which I spawn some background processes. I'd like to be able to limit the number of processes, but have my script spawn additional processes as previous tasks finish. So, let's say I have 20 tasks to complete. Any given task could take from 1 to 10 minutes. ... (7 Replies)
Discussion started by: treesloth
7 Replies

4. Shell Programming and Scripting

Parallel processes to INC- and DEC-rement shared counter

QUESTION: How do I run processes in parallel, so that the counter (in counter.txt) would vary in value (instead of just "0" and "1")? That is, how to not sequentially run inc.sh and dec.sh? The shared counter (a single number starting as 0) is in a file counter.txt. counter.sh is (supposed to... (2 Replies)
Discussion started by: courteous
2 Replies

5. Shell Programming and Scripting

script for multi-threaded bash processes

hey everyone, I'm having some trouble breaking down some code. It's simple a control script that takes machines meant to be backed up from a list. Then according to that will run multi-threaded processes up until the specified thread limit. for example if there are 4 machines to be backed up,... (2 Replies)
Discussion started by: terrell
2 Replies

6. Shell Programming and Scripting

Deleting all the parallel processes launched when the main script receives a ctrl+c

Hi, I have a shell script that creates 2 parallel processes. When I press ctrl+c, i want the parallel process to get killed as well. #!/bin/bash cmd1="script1.py" cmd2="script2.py" ${cmd1} & pid1=$! echo ${pid1} ${cmd2} & pid2=$! (7 Replies)
Discussion started by: sana.usha
7 Replies

7. Shell Programming and Scripting

Keep up constant number of parallel processes

Hi guys, I am struggling with adapting my script to increase the performance. I created a ksh script to process a lot of files in parallel. I would like to know how can I do in such a way that a constant number of processes is always up (until all is finished). What I have is (not actual... (8 Replies)
Discussion started by: lurkerro
8 Replies

8. Shell Programming and Scripting

Bash script parallel tasks and command to wait untill complete?

Hello, im having bash script with while *** command1 && command2 && command3 && done i want to ask how i can prevent overloading server, by waiting untill all commands complete? any low resources intensive command like "wait" - i dont know if exist? (2 Replies)
Discussion started by: postcd
2 Replies

9. Shell Programming and Scripting

Parallel bash scripts

Need some help to replace bash script with parallel to speed up job on multiple files (400files.list is the file contains the absolute path to those 400 files). The bash script is to run the same program over the files repetitively. My bash_script.sh is: for sample in `cat 400files.list`; do... (3 Replies)
Discussion started by: yifangt
3 Replies

10. Shell Programming and Scripting

Monitoring processes in parallel and process log file after process exits

I am writing a script to kick off a process to gather logs on multiple nodes in parallel using "&". These processes create individual log files. Which I would like to filter and convert in CSV format after they are complete. I am facing following issues: 1. Monitor all Processes parallelly.... (5 Replies)
Discussion started by: shunya
5 Replies
PCAP_LOOP(3PCAP)                                                                                                                  PCAP_LOOP(3PCAP)

NAME
pcap_loop, pcap_dispatch - process packets from a live capture or savefile SYNOPSIS
#include <pcap/pcap.h> typedef void (*pcap_handler)(u_char *user, const struct pcap_pkthdr *h, const u_char *bytes); int pcap_loop(pcap_t *p, int cnt, pcap_handler callback, u_char *user); int pcap_dispatch(pcap_t *p, int cnt, pcap_handler callback, u_char *user); DESCRIPTION
pcap_loop() processes packets from a live capture or ``savefile'' until cnt packets are processed, the end of the ``savefile'' is reached when reading from a ``savefile'', pcap_breakloop() is called, or an error occurs. It does not return when live read timeouts occur. A value of -1 or 0 for cnt is equivalent to infinity, so that packets are processed until another ending condition occurs. pcap_dispatch() processes packets from a live capture or ``savefile'' until cnt packets are processed, the end of the current bufferful of packets is reached when doing a live capture, the end of the ``savefile'' is reached when reading from a ``savefile'', pcap_breakloop() is called, or an error occurs. Thus, when doing a live capture, cnt is the maximum number of packets to process before returning, but is not a minimum number; when reading a live capture, only one bufferful of packets is read at a time, so fewer than cnt packets may be processed. A value of -1 or 0 for cnt causes all the packets received in one buffer to be processed when reading a live capture, and causes all the packets in the file to be processed when reading a ``savefile''. (In older versions of libpcap, the behavior when cnt was 0 was undefined; different platforms and devices behaved differently, so code that must work with older versions of libpcap should use -1, nor 0, as the value of cnt.) callback specifies a pcap_handler routine to be called with three arguments: a u_char pointer which is passed in the user argument to pcap_loop() or pcap_dispatch(), a const struct pcap_pkthdr pointer pointing to the packet time stamp and lengths, and a const u_char pointer to the first caplen (as given in the struct pcap_pkthdr a pointer to which is passed to the callback routine) bytes of data from the packet. The struct pcap_pkthdr and the packet data are not to be freed by the callback routine, and are not guaranteed to be valid after the callback routine returns; if the code needs them to be valid after the callback, it must make a copy of them. RETURN VALUE
pcap_loop() returns 0 if cnt is exhausted, -1 if an error occurs, or -2 if the loop terminated due to a call to pcap_breakloop() before any packets were processed. It does not return when live read timeouts occur; instead, it attempts to read more packets. pcap_dispatch() returns the number of packets processed on success; this can be 0 if no packets were read from a live capture (if, for example, they were discarded because they didn't pass the packet filter, or if, on platforms that support a read timeout that starts before any packets arrive, the timeout expires before any packets arrive, or if the file descriptor for the capture device is in non-blocking mode and no packets were available to be read) or if no more packets are available in a ``savefile.'' It returns -1 if an error occurs or -2 if the loop terminated due to a call to pcap_breakloop() before any packets were processed. If your application uses pcap_breakloop(), make sure that you explicitly check for -1 and -2, rather than just checking for a return value < 0. If -1 is returned, pcap_geterr() or pcap_perror() may be called with p as an argument to fetch or display the error text. SEE ALSO
pcap(3PCAP), pcap_geterr(3PCAP), pcap_breakloop(3PCAP) 24 December 2008 PCAP_LOOP(3PCAP)
All times are GMT -4. The time now is 11:16 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy