Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help with shell script handling processes

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with shell script handling processes
# 1  
Old 02-10-2016
Linux Help with shell script handling processes
Hello

I have a file which has around 120 lines of commands.

I am trying to write a shell script like which reads the 'command' file and executes line by line with some additional (common argument) with maximum 6 commands active at a time. Each of these commands when executed takes time which is variable and cannot be implemented within the script.

Attached both the command_file and shell script which I am executing like below,
Code:
bash: $ nohup ./program01.sh > /usr/local/app/output.log &

Expected outcome, in the /usr/local/app/output.log is as below:

Code:
Running command for Documentation with name starting with A....
Running command for Documentation with name starting with B....
Running command for Documentation with name starting with C....
Running command for Documentation with name starting with D....
Running command for Documentation with name starting with E....
Running command for Documentation with name starting with F....
Running command for Documentation with name starting with G....
No of running is 7...waiting
Running command for Documentation with name starting with H....
No of running is 7...waiting
Running command for Documentation with name starting with I....
No of running is 7...waiting
Running command for Documentation with name starting with J....
No of running is 7...waiting
Running command for Documentation with name starting with K....
No of running is 7...waiting

The outcome I see, in the /usr/local/app/output.log is as below:

Code:
Running command for Documentation with name starting with A....
Running command for Documentation with name starting with B....
Running command for Documentation with name starting with C....
Running command for Documentation with name starting with D....
Running command for Documentation with name starting with E....
Running command for Documentation with name starting with F....
Running command for Documentation with name starting with G....
Running command for Documentation with name starting with H....
Running command for Documentation with name starting with I....
No of running is 7...waiting
No of running is 7...waiting
Running command for Documentation with name starting with M....
Running command for Documentation with name starting with N....
Running command for Documentation with name starting with O....


It is skipping J & K and starting to execute M, N, O and so on. How do I make the program to wait before all the previous lines from the command_file are executed and only then proceed to next line.

May be it's a simple trick, I am missing out. I am not a shell expert. Your help is highly welcomed.

Regards
JS
program_01.sh (688 Bytes) 
output.txt (2.1 KB) 
command_file.txt (14.9 KB) 
Last edited by Don Cragun; 02-11-2016 at 02:17 AM.. Reason: Add CODE and ICODE tags.
# 2  
Old 02-10-2016
If there is no precedence, why not simply split the 120 line file into 6 files of 20 lines each, and then start the 6 files. Remove the & from the end of each line. There will never be more than 6 processes.
This User Gave Thanks to jgt For This Post:
JackyShane_36
# 3  
Old 02-10-2016
Code:
#!/bin/bash

maxnum=6

# Initial load
c=1
while [ $c -le $maxnum ]; do
    read line || exit
    echo "$line" | sh &
    p[$c]=$!
    echo ${p[$c]}
    ((c++))
done

# Loop through all, if dead read and start new
while read line; do
    echo "$line"

    q=1
    while [ $q ]; do
        c=1
        while [ $c -le $maxnum ]; do
           kill -0 ${p[$c]} 2>/dev/null
           if [ $? -ne 0 ];then
               echo "$line" | sh &
               p[$c]=$!
               echo "run $c"
               q=""
               break
           fi
           ((c++))
        done
    done
done
# don't exit till remaining background jobs complete
wait

This User Gave Thanks to cjcox For This Post:
JackyShane_36
# 4  
Old 02-10-2016
If i may propose a command povided by my visual toolset TUI.
Can not open the attached command_list.txt - for the provided code, i expected to have each "regular valid command" on a single line.

Once TUI is installed, place this where your other files are.
Code:
tmpdir=/tmp	# Dir for temp/work files
LIMIT=6 	# amount of background processes
C=0		# counter

# Create tempfiles, as the tool is based on 'scripts' not 'commands'
while read line
do
	echo "$line" > $tmpdir/$$.$C
	((C++))
done<<command_file.txt

# run all the commands/scripts, but only LIMIT processes
tui-bgjob-mgr -l $LIMIT $tmpdir/$$.*

# Remove the tempfiles/commands
rm $tmpdir/$$.*

Alternativly, you could try this:
Code:
while read cmd;do $cmd;done<<command_file.txt

Which is basicly the same as calling it like:
Code:
bash command_file.txt

Smilie

Hope this helps
Last edited by sea; 02-10-2016 at 11:09 PM..
This User Gave Thanks to sea For This Post:
JackyShane_36
# 5  
Old 02-25-2016
Thanks everyone, but I was looking for something easier. With some more dragging around I found the best way to do it without too much complexity.

Adding the code below for reference.

Code:
 
#!/bin/bash
FILE2=/usr/local/app/command_file.log
while read line; do
command=$(echo $line)
log=$(echo $line | awk -F'"' '{$0=$2}1')
count=$(ps aux | grep -i "tscV6Connector" | grep -v "grep" | wc -l)
instance=$(echo $line | awk -F'"' '{$0=$4}1')
if [ $count -le 6 ];
then 
    echo "Running command for "$log" with name starting with "$instance"...."
    nohup "additional_command_parameters" > /usr/local/app/log/"$log"/"$log_$instance".log &
    sleep 10    #Sleep time for the extraction process.
    now1=$(date +'%d/%m/%Y %X');
    echo -e "*** TSO *** "$now1": Started extraction for MAGIC object type "$log" with name starting "$instance" process PID = "$!""
else 
    now2=$(date +'%d/%m/%Y %X');
    echo -e "*** TSO *** "$now2": Total MQL extraction process reached "$count"....\n*** TSO *** "$now2": waiting for previous process PID "$!" to complete"
    wait $!;    #Main wait statement when count >=7. Once wait is over, run the next command after wait.
    nohup "additional_command_parameters" > /usr/local/app/log/"$log"/"$log_$instance".log &
    sleep 10;    #Sleep time for the last MQL which was waiting before log files.
    now=$(date +'%d/%m/%Y %X');
    echo -e "*** TSO *** "$now3": Started extraction for MAGIC object type "$log" with name starting "$instance" process PID = "$!"\n*** TSO *** "$now3": Total number of extraction MQLs on DB are = "$count"\n"
fi
done < $FILE2

This way when 7th process is trying to start, it waits until 6th is completed and so on .....

A simple logic, works well enough Smilie Smilie Smilie

Experts are welcome to comment.

BR/
JS
# 6  
Old 02-25-2016
Quote:
Originally Posted by JackyShane_36
Thanks everyone, but I was looking for something easier. With some more dragging around I found the best way to do it without too much complexity.

Adding the code below for reference.

Code:
 
#!/bin/bash
FILE2=/usr/local/app/command_file.log
while read line; do
command=$(echo $line)
# The variable "command" is never used.  Why define it?
log=$(echo $line | awk -F'"' '{$0=$2}1')
count=$(ps aux | grep -i "tscV6Connector" | grep -v "grep" | wc -l)
# Change: grep -i "tscV6Connector" | grep -v "grep"
# in the pipeline above to: grep -i "[t]scV6Connector"
# and your script will run faster.
instance=$(echo $line | awk -F'"' '{$0=$4}1')
if [ $count -le 6 ];
then 
    echo "Running command for "$log" with name starting with "$instance"...."
    nohup "additional_command_parameters" > /usr/local/app/log/"$log"/"$log_$instance".log &
    sleep 10    #Sleep time for the extraction process.
# The variable "log_" has not been defined.  Should the output above be
# redirected to:  /usr/local/app/log/"$log/${log}_$instance.log"
    now1=$(date +'%d/%m/%Y %X');
    echo -e "*** TSO *** "$now1": Started extraction for MAGIC object type "$log" with name starting "$instance" process PID = "$!""
else 
    now2=$(date +'%d/%m/%Y %X');
    echo -e "*** TSO *** "$now2": Total MQL extraction process reached "$count"....\n*** TSO *** "$now2": waiting for previous process PID "$!" to complete"
    wait $!;    #Main wait statement when count >=7. Once wait is over, run the next command after wait.
    nohup "additional_command_parameters" > /usr/local/app/log/"$log"/"$log_$instance".log &
# See note above about ${log}_ versus $log_.
    sleep 10;    #Sleep time for the last MQL which was waiting before log files.
    now=$(date +'%d/%m/%Y %X');
# In the line above you define the variable "now".
# In the line below you use the undefined variable "now3".
    echo -e "*** TSO *** "$now3": Started extraction for MAGIC object type "$log" with name starting "$instance" process PID = "$!"\n*** TSO *** "$now3": Total number of extraction MQLs on DB are = "$count"\n"
fi
done < $FILE2

This way when 7th process is trying to start, it waits until 6th is completed and so on .....
Not necessarily. You wait for the last background job you started to finish, but it might have already completed before you counted the number of running jobs. So you could have 7 jobs running concurrently instead of 6. Should we assume that for what you are doing the difference between running 6 or 7 jobs at once doesn't really matter?
Quote:
A simple logic, works well enough Smilie Smilie Smilie

Experts are welcome to comment.

BR/
JS
Please note the comments I have added to you code in red above. Maybe you will find them useful.

You might also want to consider changing:
Code:
log=$(echo $line | awk -F'"' '{$0=$2}1')
instance=$(echo $line | awk -F'"' '{$0=$4}1')

to:
Code:
IFS='"' read -r junk log junk instance junk <<-EOF
        $line
EOF

to get rid of two invocations of awk for every line read from your input file.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Issue handling single quoted argument in shell script.

Below is my script that works fine and prints the desired output: #!/bin/ksh echo "$1" | while IFS= read -r dirpath do echo "DIRR_PATH:$dirpath" install_dir=$install_dir" "$dirpath done echo "Desired Output:$install_dir" Output: ./loopissue.sh... (10 Replies)
Discussion started by: mohtashims
10 Replies

2. Shell Programming and Scripting

Wait for one processes to complete in a shell script

Let's say I start process A.sh, then start process B.sh. I call both of them in my C.sh How can I make sure that B starts its execution only after A.sh finishes. I have to do this in loop.Execution time of A.sh may vary everytime. It is a parameterized script. (17 Replies)
Discussion started by: rafa_fed2
17 Replies

3. Solaris

Script on Solaris spawning 2 processes for one shell script execution

Hi, I am having a shell script on Solaris 10 which has a while loop as shown below. #!/usr/bin/ksh # while do sleep 60 done Name of the shell script is coldcentric.sh. I executed script /DATAWAREHOUSE/LOAD/Scripts/coldcentric.sh from a command task in Informatica worklow as... (3 Replies)
Discussion started by: chekusi
3 Replies

4. Shell Programming and Scripting

Shell script executed from Informatica ETL tool is spawning 2 processes for one script

Hi, I am having a shell script which has a while loop as shown below. while do sleep 60 done I am executing this script from Informatica ETL tool command task from where we can execute UNIX commands/scripts. When i do that, i am seeing 2 processes getting started for one script... (2 Replies)
Discussion started by: chekusi
2 Replies

5. Shell Programming and Scripting

Help with Handling multiple argument in shell script

Hi i have written a shell script that takes only single ip address from the user and calculates its latency and reliability, can you please tell me that what should be done if i want that user should enter 100 or 1000 ip address (5 Replies)
Discussion started by: Preeti_17
5 Replies

6. Shell Programming and Scripting

BASH - Handling background processes - distributed processing

NOTE: I am using BASH and Solaris 10 for this. Currently in the process of building a script that has a main "watcher" daemon that reads a configuration file and starts background processes based on it's global configuration. It is basically an infinite loop of configuration reading. Some of the... (4 Replies)
Discussion started by: dcarrion87
4 Replies

7. Shell Programming and Scripting

shell script for getting pid of spawn processes from shell

Hi, I am new this forum. I request you peoples help in understanding and finding some solution to my problem. Here it goes: I need to perform this set of actions by writing a shell script. I need to read a config file for the bunch of processes to execute. I need to fecth the pid of... (4 Replies)
Discussion started by: sachin4sachi
4 Replies

8. Shell Programming and Scripting

Shell script file handling

Hi ! /bin/sh set logdir1 "logDir/local/logname" #write the filename into a file echo $logdir1 >> logname.txt how do i exec the above echo command (1 Reply)
Discussion started by: nathgopi214
1 Replies

9. Shell Programming and Scripting

Shell script creating too many processes.

I have a shell script that I am running every 60 seconds, but it is creating this process to the point that it is causing the server to perfrom poorly. Below is my script, what can I change to prevent this? while true do java -classpath .....( all my classes here) >/dev/null 2>&1 ... (3 Replies)
Discussion started by: Miller_K
3 Replies

10. UNIX for Advanced & Expert Users

signal handling in shell script

Hi can any please tell me is it possible to catch the signal in a shell script like we do in C. if yes please give me some idea or a link. (4 Replies)
Discussion started by: Raom
4 Replies
Login or Register to Ask a Question