BASH Execution Delay / Speedup


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting BASH Execution Delay / Speedup
# 29  
Old 01-29-2015
Quote:
Originally Posted by gmark99
In my case, "my_job" does some database queries and decides to call a function that will result in a cross-processor message and then wait for a result to be returned.

I would still benefit from a lot of jobs in parallel if they're just waiting for results from another machine, right?
Well, how many queries can your database server really run at the same time? There's no point exceeding that. 5,000 processes sitting around waiting for the database server are no faster than 5 processes sitting around waiting for the database server.

Also, what exactly does your query do? Might it create lots of temporary tables, consume lots of memory, ask for things on opposite ends of the disk, lock tables so everyone else has to wait, etc? If you push any of your server's limits, be they bandwidth, disk, or CPU, you lose.

Spamming hundreds of network connections causes its own bottlenecks, too.
Quote:
Am I missing anything, or am I on the right track?
I think you should look into having one job doing n queries, instead n jobs doing 1 query. There's a lot of overhead in making and breaking connections over and over. Besides, hammering any kind of server like that is just rude. If you can do that, try 2 jobs doing n/2 queries and see if that's an improvement or not.

Last edited by Corona688; 01-29-2015 at 06:15 PM..
# 30  
Old 01-29-2015
Do those 'my_job's handle the waiting on external resources?

If so and if they are idle - as in waiting on the resource to become available, do they sleep a little bit, or are they constantly attempting to 'read' (didnt find a better word in this context) from that outside response??
# 31  
Old 01-29-2015
Quote:
Originally Posted by gmark99
In my case, "my_job" does some database queries and decides to call a function that will result in a cross-processor message and then wait for a result to be returned.

I would still benefit from a lot of jobs in parallel if they're just waiting for results from another machine, right? This waiting for outside resources is the basic reason for my wanting to run so many jobs in background.

Am I missing anything, or am I on the right track?
Have you heard of denial of service attacks? Flooding a database server with more requests than it can handle doesn't speed things up; it grinds service to a halt.

We assume that you want to increase transaction throughput rather than to increase the loads on your CPUs, disks, networks, etc. Increased loads on system facilities is not linearly related to throughput. Increasing the load on a loaded (or overloaded) resource frequently reduces throughput rather than increasing it.
# 32  
Old 01-30-2015
Unfortunately we built the database custom as well, so we'll have to test out the capacity. In reference to another comment, each job does six queries, and they're not very time sensitive at all. If we have to throttle requests, we can do that. What guidance you folks have provided so far is invaluable. I'll explore the advice I've gotten and come back if I have more questions, which I'm very likely to have.

Thanks!!!

---------- Post updated 01-30-15 at 09:55 AM ---------- Previous update was 01-29-15 at 07:45 PM ----------

Reply to sea:

The "my_job" processes just wait for the returns of function calls that read the data or that are operating hardware.

---------- Post updated at 09:59 AM ---------- Previous update was at 09:55 AM ----------

I have a couple more questions.

If I execute a "wait" command, does that cause a wait until ALL the child background processes have completed?

It was suggested I put in a "sleep 3" command after an "exit"? Did I get the right? Or after a "done"? Why, specifically?

Thanks!
# 33  
Old 01-30-2015
Quote:
Originally Posted by gmark99
If I execute a "wait" command, does that cause a wait until ALL the child background processes have completed?
Yes.

Some shells let you 'wait PID', to wait for one process in particular, but not all.
# 34  
Old 01-30-2015
Quote:
Originally Posted by gmark99
In reference to another comment, each job does six queries, and they're not very time sensitive at all.
Does 'not very time sensitive' mean 'very short and fast'?

In that case there could be a lot of benefit to doing the loop on the database instead of locally. The time it takes to make and break the connection could be on par with the time it takes to do the query, leading to quite a lot of waste.

Last edited by Corona688; 01-30-2015 at 01:32 PM..
# 35  
Old 02-10-2015
I usually double the core count, as threads are i/o blocked half the time at least. The idea is to keep the count in a managable range, and not waste resources with excess parallelism. A tool like bctl, at the end of a command generating pipeline, manages this for you with very low overhead, and then there is GNU Parallel, a much more complex but flexible paralelism tool.

Most bash, perl, shell speedup involves examining things in a loop, avoiding shelling out (`cmd` or '$(cmd)' ) when there is a shell built in that can do it, or using sed/awk in the pipeline to enhance the data stream so that a 'while read' sort of loop in eliminated or kept simpler. For instance, sed can convert a stream data lines into a stream of commands that can flow to bctl or a shell.

Last edited by DGPickett; 02-10-2015 at 11:25 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Why execution is different from orginal bash registry$database?

Hi all, here's my script #!/bin/ksh if then export DB_CREATE_PATH=`pwd` fi echo echo "********************--Menu--*****************************" echo "*** " echo "*** 1. Pre-Upgrade Steps "... (3 Replies)
Discussion started by: jediwannabe
3 Replies

2. Shell Programming and Scripting

How to simulate an execution queue with bash?

I'm running cygwin bash on windows 7 and I'm have some bat files that perform large builds and take a long time and a lot of memory. Therefor, I don't want to builds executing simultaneously (too much memory). How can I implement a queue so I can queue up multiple builds and only execute one... (2 Replies)
Discussion started by: siegfried
2 Replies

3. Shell Programming and Scripting

execution of a string being echoed in bash

hi all, I am trying to do a loop on a series of plotting function shown below: colorlist=(blue red green); n=0; for k in $xy; do psbasemap $range -JM$scale -B10g5 -X1 -Y1 -P -K > $outfile pscoast $range -JM$scale -B10g5 -D$res -P -W$lwidth -G$fill -O -K >> $outfile echo... (1 Reply)
Discussion started by: ida1215
1 Replies

4. Shell Programming and Scripting

Execution problems with BASH Shell Script

Hi I need help with my coding , first time I'm working with bash . What i must do is check if there is 3 .txt files if there is not 3 of them i must give an error code , if al three is there i must first arrange them in alphabetical order and then take the last word in al 3 of the .txt files... (1 Reply)
Discussion started by: linux newb
1 Replies

5. Shell Programming and Scripting

Execution Problems with bash script

Hello, can someone please help me to fix this script, I have a 2 files, one file has hostname information and second file has console information of the hosts in each line, I have written a script which actually reads each line in hostname file and should grep in the console file and paste the... (8 Replies)
Discussion started by: bobby320
8 Replies

6. Shell Programming and Scripting

Splitting file needs speedup

I've got a large file (2-4 gigs), made up of 4 columns. I'd like to split the file into two, based on the 2nd column value being even or odd. The following script does the job, but runs incredibly slow--I'm not sure it will complete this week. There must be a clever way to do this with just... (2 Replies)
Discussion started by: I.P. Freeley
2 Replies

7. Shell Programming and Scripting

Optimize and Speedup the script

Hi All, There is a script (test.sh) which is taking more CPU usage. I am attaching the script in this thread. Could anybody please help me out to optimize the script in a better way. Thanks, Gobinath (6 Replies)
Discussion started by: ntgobinath
6 Replies

8. Shell Programming and Scripting

execution time / runtime -- bash script please help!

Hello, I'm running a bash script and I'd like to get more accurate a runtime information then now. So far I've been using this method: STARTM=`date -u "+%s"` ......... *script function.... ......... STOPM=`date -u "+%s"` RUNTIMEM=`expr $STOPM - $STARTM` if (($RUNTIMEM>59)); then... (6 Replies)
Discussion started by: TehOne
6 Replies

9. Shell Programming and Scripting

Calculating delay time - bash

Hi, I am having the following problem. test > hourOfDay=06 ; delayTime=$(((9-$hourOfDay)*60)) ; echo $delayTime 180 test > hourOfDay=07 ; delayTime=$(((9-$hourOfDay)*60)) ; echo $delayTime 120 test > hourOfDay=08 ; delayTime=$(((9-$hourOfDay)*60)) ; echo $delayTime bash: (9-08: value... (5 Replies)
Discussion started by: jbsimon000
5 Replies

10. Shell Programming and Scripting

bash - delay expansion of variable

Hello - I have a bash script which does some logging, and I'd like to include the line number of the echo statement that pipes into $LOGGER: MYPID=$$ MYNAME=`basename $0` LOGGER="/usr/bin/logger -t $MYNAME($LINENO) -p daemon.error" ... echo 'this is an entry into the log file' | $LOGGER ... (3 Replies)
Discussion started by: scandora
3 Replies
Login or Register to Ask a Question