01-29-2015
Quote:
Originally Posted by
gmark99
In my case, "my_job" does some database queries and decides to call a function that will result in a cross-processor message and then wait for a result to be returned.
I would still benefit from a lot of jobs in parallel if they're just waiting for results from another machine, right?
Well, how many queries can your database server
really run at the same time? There's no point exceeding that. 5,000 processes sitting around waiting for the database server are no faster than 5 processes sitting around waiting for the database server.
Also, what exactly does your query do? Might it create lots of temporary tables, consume lots of memory, ask for things on opposite ends of the disk, lock tables so everyone else has to wait, etc? If you push any of your server's limits, be they bandwidth, disk, or CPU, you lose.
Spamming hundreds of network connections causes its own bottlenecks, too.
Quote:
Am I missing anything, or am I on the right track?
I think you should look into having one job doing n queries, instead n jobs doing 1 query. There's a lot of overhead in making and breaking connections over and over. Besides, hammering any kind of server like that is just rude. If you can do that, try 2 jobs doing n/2 queries and see if that's an improvement or not.
Last edited by Corona688; 01-29-2015 at 06:15 PM..
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hello -
I have a bash script which does some logging, and I'd like to include the line number of the echo statement that pipes into $LOGGER:
MYPID=$$
MYNAME=`basename $0`
LOGGER="/usr/bin/logger -t $MYNAME($LINENO) -p daemon.error"
...
echo 'this is an entry into the log file' | $LOGGER
... (3 Replies)
Discussion started by: scandora
3 Replies
2. Shell Programming and Scripting
Hi,
I am having the following problem.
test > hourOfDay=06 ; delayTime=$(((9-$hourOfDay)*60)) ; echo $delayTime
180
test > hourOfDay=07 ; delayTime=$(((9-$hourOfDay)*60)) ; echo $delayTime
120
test > hourOfDay=08 ; delayTime=$(((9-$hourOfDay)*60)) ; echo $delayTime
bash: (9-08: value... (5 Replies)
Discussion started by: jbsimon000
5 Replies
3. Shell Programming and Scripting
Hello,
I'm running a bash script and I'd like to get more accurate a runtime information then now.
So far I've been using this method:
STARTM=`date -u "+%s"`
.........
*script function....
.........
STOPM=`date -u "+%s"`
RUNTIMEM=`expr $STOPM - $STARTM`
if (($RUNTIMEM>59)); then... (6 Replies)
Discussion started by: TehOne
6 Replies
4. Shell Programming and Scripting
Hi All,
There is a script (test.sh) which is taking more CPU usage. I am attaching the script in this thread.
Could anybody please help me out to optimize the script in a better way.
Thanks,
Gobinath (6 Replies)
Discussion started by: ntgobinath
6 Replies
5. Shell Programming and Scripting
I've got a large file (2-4 gigs), made up of 4 columns. I'd like to split the file into two, based on the 2nd column value being even or odd. The following script does the job, but runs incredibly slow--I'm not sure it will complete this week.
There must be a clever way to do this with just... (2 Replies)
Discussion started by: I.P. Freeley
2 Replies
6. Shell Programming and Scripting
Hello,
can someone please help me to fix this script,
I have a 2 files, one file has hostname information and second file has console information of the hosts in each line, I have written a script which actually reads each line in hostname file and should grep in the console file and paste the... (8 Replies)
Discussion started by: bobby320
8 Replies
7. Shell Programming and Scripting
Hi
I need help with my coding , first time I'm working with bash .
What i must do is check if there is 3 .txt files if there is not 3 of them i must give an error code , if al three is there i must first arrange them in alphabetical order and then take the last word in al 3 of the .txt files... (1 Reply)
Discussion started by: linux newb
1 Replies
8. Shell Programming and Scripting
hi all, I am trying to do a loop on a series of plotting function shown below:
colorlist=(blue red green); n=0;
for k in $xy; do
psbasemap $range -JM$scale -B10g5 -X1 -Y1 -P -K > $outfile
pscoast $range -JM$scale -B10g5 -D$res -P -W$lwidth -G$fill -O -K >> $outfile
echo... (1 Reply)
Discussion started by: ida1215
1 Replies
9. Shell Programming and Scripting
I'm running cygwin bash on windows 7 and I'm have some bat files that perform large builds and take a long time and a lot of memory.
Therefor, I don't want to builds executing simultaneously (too much memory).
How can I implement a queue so I can queue up multiple builds and only execute one... (2 Replies)
Discussion started by: siegfried
2 Replies
10. Shell Programming and Scripting
Hi all,
here's my script
#!/bin/ksh
if
then
export DB_CREATE_PATH=`pwd`
fi
echo
echo "********************--Menu--*****************************"
echo "*** "
echo "*** 1. Pre-Upgrade Steps "... (3 Replies)
Discussion started by: jediwannabe
3 Replies
LEARN ABOUT CENTOS
system
SYSTEM(3) Linux Programmer's Manual SYSTEM(3)
NAME
system - execute a shell command
SYNOPSIS
#include <stdlib.h>
int system(const char *command);
DESCRIPTION
system() executes a command specified in command by calling /bin/sh -c command, and returns after the command has been completed. During
execution of the command, SIGCHLD will be blocked, and SIGINT and SIGQUIT will be ignored.
RETURN VALUE
The value returned is -1 on error (e.g., fork(2) failed), and the return status of the command otherwise. This latter return status is in
the format specified in wait(2). Thus, the exit code of the command will be WEXITSTATUS(status). In case /bin/sh could not be executed,
the exit status will be that of a command that does exit(127).
If the value of command is NULL, system() returns nonzero if the shell is available, and zero if not.
system() does not affect the wait status of any other children.
CONFORMING TO
C89, C99, POSIX.1-2001.
NOTES
If the _XOPEN_SOURCE feature test macro is defined (before including any header files), then the macros described in wait(2) (WEXITSTA-
TUS(), etc.) are made available when including <stdlib.h>.
As mentioned, system() ignores SIGINT and SIGQUIT. This may make programs that call it from a loop uninterruptible, unless they take care
themselves to check the exit status of the child. E.g.
while (something) {
int ret = system("foo");
if (WIFSIGNALED(ret) &&
(WTERMSIG(ret) == SIGINT || WTERMSIG(ret) == SIGQUIT))
break;
}
Do not use system() from a program with set-user-ID or set-group-ID privileges, because strange values for some environment variables might
be used to subvert system integrity. Use the exec(3) family of functions instead, but not execlp(3) or execvp(3). system() will not, in
fact, work properly from programs with set-user-ID or set-group-ID privileges on systems on which /bin/sh is bash version 2, since bash 2
drops privileges on startup. (Debian uses a modified bash which does not do this when invoked as sh.)
In versions of glibc before 2.1.3, the check for the availability of /bin/sh was not actually performed if command was NULL; instead it was
always assumed to be available, and system() always returned 1 in this case. Since glibc 2.1.3, this check is performed because, even
though POSIX.1-2001 requires a conforming implementation to provide a shell, that shell may not be available or executable if the calling
program has previously called chroot(2) (which is not specified by POSIX.1-2001).
It is possible for the shell command to return 127, so that code is not a sure indication that the execve(2) call failed.
SEE ALSO
sh(1), signal(2), wait(2), exec(3)
COLOPHON
This page is part of release 3.53 of the Linux man-pages project. A description of the project, and information about reporting bugs, can
be found at http://www.kernel.org/doc/man-pages/.
2010-09-10 SYSTEM(3)