I have a file which has around 120 lines of commands.
I am trying to write a shell script like which reads the 'command' file and executes line by line with some additional (common argument) with maximum 6 commands active at a time. Each of these commands when executed takes time which is variable and cannot be implemented within the script.
Attached both the command_file and shell script which I am executing like below,
Expected outcome, in the /usr/local/app/output.log is as below:
The outcome I see, in the /usr/local/app/output.log is as below:
It is skipping J & K and starting to execute M, N, O and so on. How do I make the program to wait before all the previous lines from the command_file are executed and only then proceed to next line.
May be it's a simple trick, I am missing out. I am not a shell expert. Your help is highly welcomed.
If there is no precedence, why not simply split the 120 line file into 6 files of 20 lines each, and then start the 6 files. Remove the & from the end of each line. There will never be more than 6 processes.
#!/bin/bash
maxnum=6
# Initial load
c=1
while [ $c -le $maxnum ]; do
read line || exit
echo "$line" | sh &
p[$c]=$!
echo ${p[$c]}
((c++))
done
# Loop through all, if dead read and start new
while read line; do
echo "$line"
q=1
while [ $q ]; do
c=1
while [ $c -le $maxnum ]; do
kill -0 ${p[$c]} 2>/dev/null
if [ $? -ne 0 ];then
echo "$line" | sh &
p[$c]=$!
echo "run $c"
q=""
break
fi
((c++))
done
done
done
# don't exit till remaining background jobs complete
wait
If i may propose a command povided by my visual toolset TUI.
Can not open the attached command_list.txt - for the provided code, i expected to have each "regular valid command" on a single line.
Once TUI is installed, place this where your other files are.
Code:
tmpdir=/tmp # Dir for temp/work files
LIMIT=6 # amount of background processes
C=0 # counter
# Create tempfiles, as the tool is based on 'scripts' not 'commands'
while read line
do
echo "$line" > $tmpdir/$$.$C
((C++))
done<<command_file.txt
# run all the commands/scripts, but only LIMIT processes
tui-bgjob-mgr -l $LIMIT $tmpdir/$$.*
# Remove the tempfiles/commands
rm $tmpdir/$$.*
Thanks everyone, but I was looking for something easier. With some more dragging around I found the best way to do it without too much complexity.
Adding the code below for reference.
Code:
#!/bin/bash
FILE2=/usr/local/app/command_file.log
while read line; do
command=$(echo $line)
log=$(echo $line | awk -F'"' '{$0=$2}1')
count=$(ps aux | grep -i "tscV6Connector" | grep -v "grep" | wc -l)
instance=$(echo $line | awk -F'"' '{$0=$4}1')
if [ $count -le 6 ];
then
echo "Running command for "$log" with name starting with "$instance"...."
nohup "additional_command_parameters" > /usr/local/app/log/"$log"/"$log_$instance".log &
sleep 10 #Sleep time for the extraction process.
now1=$(date +'%d/%m/%Y %X');
echo -e "*** TSO *** "$now1": Started extraction for MAGIC object type "$log" with name starting "$instance" process PID = "$!""
else
now2=$(date +'%d/%m/%Y %X');
echo -e "*** TSO *** "$now2": Total MQL extraction process reached "$count"....\n*** TSO *** "$now2": waiting for previous process PID "$!" to complete"
wait $!; #Main wait statement when count >=7. Once wait is over, run the next command after wait.
nohup "additional_command_parameters" > /usr/local/app/log/"$log"/"$log_$instance".log &
sleep 10; #Sleep time for the last MQL which was waiting before log files.
now=$(date +'%d/%m/%Y %X');
echo -e "*** TSO *** "$now3": Started extraction for MAGIC object type "$log" with name starting "$instance" process PID = "$!"\n*** TSO *** "$now3": Total number of extraction MQLs on DB are = "$count"\n"
fi
done < $FILE2
This way when 7th process is trying to start, it waits until 6th is completed and so on .....
Thanks everyone, but I was looking for something easier. With some more dragging around I found the best way to do it without too much complexity.
Adding the code below for reference.
Code:
#!/bin/bash
FILE2=/usr/local/app/command_file.log
while read line; do
command=$(echo $line)
# The variable "command" is never used. Why define it?
log=$(echo $line | awk -F'"' '{$0=$2}1')
count=$(ps aux | grep -i "tscV6Connector" | grep -v "grep" | wc -l)
# Change: grep -i "tscV6Connector" | grep -v "grep"
# in the pipeline above to: grep -i "[t]scV6Connector"
# and your script will run faster.
instance=$(echo $line | awk -F'"' '{$0=$4}1')
if [ $count -le 6 ];
then
echo "Running command for "$log" with name starting with "$instance"...."
nohup "additional_command_parameters" > /usr/local/app/log/"$log"/"$log_$instance".log &
sleep 10 #Sleep time for the extraction process.
# The variable "log_" has not been defined. Should the output above be
# redirected to: /usr/local/app/log/"$log/${log}_$instance.log"
now1=$(date +'%d/%m/%Y %X');
echo -e "*** TSO *** "$now1": Started extraction for MAGIC object type "$log" with name starting "$instance" process PID = "$!""
else
now2=$(date +'%d/%m/%Y %X');
echo -e "*** TSO *** "$now2": Total MQL extraction process reached "$count"....\n*** TSO *** "$now2": waiting for previous process PID "$!" to complete"
wait $!; #Main wait statement when count >=7. Once wait is over, run the next command after wait.
nohup "additional_command_parameters" > /usr/local/app/log/"$log"/"$log_$instance".log &
# See note above about ${log}_ versus $log_.
sleep 10; #Sleep time for the last MQL which was waiting before log files.
now=$(date +'%d/%m/%Y %X');
# In the line above you define the variable "now".
# In the line below you use the undefined variable "now3".
echo -e "*** TSO *** "$now3": Started extraction for MAGIC object type "$log" with name starting "$instance" process PID = "$!"\n*** TSO *** "$now3": Total number of extraction MQLs on DB are = "$count"\n"
fi
done < $FILE2
This way when 7th process is trying to start, it waits until 6th is completed and so on .....
Not necessarily. You wait for the last background job you started to finish, but it might have already completed before you counted the number of running jobs. So you could have 7 jobs running concurrently instead of 6. Should we assume that for what you are doing the difference between running 6 or 7 jobs at once doesn't really matter?
Quote:
A simple logic, works well enough
Experts are welcome to comment.
BR/
JS
Please note the comments I have added to you code in red above. Maybe you will find them useful.
Below is my script that works fine and prints the desired output:
#!/bin/ksh
echo "$1" |
while IFS= read -r dirpath
do
echo "DIRR_PATH:$dirpath"
install_dir=$install_dir" "$dirpath
done
echo "Desired Output:$install_dir"
Output:
./loopissue.sh... (10 Replies)
Let's say I start process A.sh, then start process B.sh. I call both of them in my C.sh
How can I make sure that B starts its execution only after A.sh finishes.
I have to do this in loop.Execution time of A.sh may vary everytime.
It is a parameterized script. (17 Replies)
Hi,
I am having a shell script on Solaris 10 which has a while loop as shown below.
#!/usr/bin/ksh
#
while
do
sleep 60
done
Name of the shell script is coldcentric.sh. I executed script /DATAWAREHOUSE/LOAD/Scripts/coldcentric.sh from a command task in Informatica worklow as... (3 Replies)
Hi,
I am having a shell script which has a while loop as shown below.
while
do
sleep 60
done
I am executing this script from Informatica ETL tool command task from where we can execute UNIX commands/scripts. When i do that, i am seeing 2 processes getting started for one script... (2 Replies)
Hi i have written a shell script that takes only single ip address from the user and calculates its latency and reliability, can you please tell me that what should be done if i want that user should enter 100 or 1000 ip address (5 Replies)
NOTE: I am using BASH and Solaris 10 for this.
Currently in the process of building a script that has a main "watcher" daemon that reads a configuration file and starts background processes based on it's global configuration. It is basically an infinite loop of configuration reading. Some of the... (4 Replies)
Hi,
I am new this forum. I request you peoples help in understanding and finding some solution to my problem.
Here it goes:
I need to perform this set of actions by writing a shell script. I need to read a config file for the bunch of processes to execute.
I need to fecth the pid of... (4 Replies)
Hi
! /bin/sh
set logdir1 "logDir/local/logname"
#write the filename into a file
echo $logdir1 >> logname.txt
how do i exec the above echo command (1 Reply)
I have a shell script that I am running every 60 seconds, but it is creating this process to the point that it is causing the server to perfrom poorly. Below is my script, what can I change to prevent this?
while true
do
java -classpath .....( all my classes here)
>/dev/null 2>&1
... (3 Replies)