Have each subshell write stderr and stdout to its own logfile
Hello,
As stated in the title, I do some hacked parallel processing by running multiple instances of bash scripts, each in their own subshell. The code looks like this,
Code:
# launch one batch-train script in background for each value in fold group list
for FOLD_GROUP in "${FOLD_GROUP_LIST[@]}"
do
# continue training on $FOLD_GROUP folds
./01_batch_script.sh $CORES \
$BATCH_PROJ \
$BATCH_STOP_ON_SUBSET \
$BATCH_STOP_ON_STAT \
$SET \
$FOLD_GROUP \
$RND_SEED \
$DATASET_STRING \
$ERRTOL \
$BATCHES \
$START_MODE \
$START_LR \
$MAX_EPOCH_BATCH \
$OA_PRINT_PRECISION \
$BATCH_PROC &
# to prevent terminal overrun
sleep 4
done
# wait for all subshells to return before resuming
wait
Despite the use of sleep to try and space things out, when I am working on these scripts I am writing allot of output and there is no way to keep the output from getting muddled when it is all going to one terminal.
I can generally work on just one instance if I need to, but it would also be nice to have each script report to its own log file so I can see which instance errors are associated with.
Is there a reasonable way to do this that someone can suggest?
How can I redirect and append stdout and stderr to a file when using cron? Here is my crontab file:
*/5 * * * * /dir/php /dir/process_fns.php >>& /dir/dump.txt
Cron gives me an 'unexpected character found in line' when trying to add my crontab file.
Regards,
Zach Curtis
POPULUS (8 Replies)
Hello
Before I get to my question let me explain the situation. I am writing a ksh script to startup several instances of an application. That is done by executing another ksh script (let's call it startApp) with some arguments. Now, startApp writes some information to stdout that I don't want... (3 Replies)
#!/usr/bin/perl
open(STDOUT, ">>$Textfile")
open(STDERR, ">>$Textfile")
print "program running\n";
$final = join("+", $initial,$final) #5
close (STDOUT);
close (STDERR);Hi all, above is my perl code. Notice i have captured the stdout and stderr to the same textfile. my code is expected to... (1 Reply)
I have been doing this:
make xyz &> xyz.log &; tail -f xyz.log
The problem with this is that you never can ge sure when "make xyz" is done.
How can I pipe both stderr and stdout into tee so both stderr and stdout are copied both to the display and to the log file?
Thanks,
Siegfried (3 Replies)
Hi,
I would like to avoid re-directing line by line to a file.
What is the best way to re-direct STDOUT to a file in a subshell?
Thanks in advance.
Cheers
Vj (1 Reply)
Hello
I read a lot of post related to this topic, but nothing helped me. :mad:
I'm running a ksh script with subshell what processing some ldap command. I need to check output for possible errors.
#!/bin/ksh
...
readinput < $QCHAT_INPUT |&
while read -p line
do
echo $line
... (3 Replies)
Can somebody explain to me why the diff output is not going to stderr?
Yet when I issue a diff from the command line the return code is -ne 1.
I am guessing diff always writes to stdout???
Is there away I can force the difff to write to stderr USING THE CURRENT
template. If possible, I... (5 Replies)
Hi all,
can someone help me with the next redirection?
i want to redirect the stdout+stderr of a command to the same file (this i can do by prog &> file)
but in addition i want to redirect only the stderr to a different file.
how can i do this please? (in BASH)
thanks. (4 Replies)
Hi,
I got a process (c written) that we usually run this way :
EmsChkQu >> /EMS/log/EmsChkQu.log 2>&1
When trying to use it as a service I defined it this way :
<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle... (6 Replies)
(Apologies for any typos.)
OSX 10.12.3 AND Windows 10.
This is for the serious Python experts on at least 3.5.x and above...
In script format sys.stdout.write() AND sys.stderr.write() seems to work correctly.
Have I found a serious bug in the interactive sys.stdout.write() AND... (2 Replies)
Discussion started by: wisecracker
2 Replies
LEARN ABOUT DEBIAN
libbash
LIBBASH(7) libbash Manual LIBBASH(7)NAME
libbash -- A bash shared libraries package.
DESCRIPTION
libbash is a package that enables bash dynamic-like shared libraries. Actually its a tool for managing bash scripts whose functions you may
want to load and use in scripts of your own.
It contains a 'dynamic loader' for the shared libraries ( ldbash(1)), a configuration tool (ldbashconfig(8)), and some libraries.
Using ldbash(1) you are able to load loadable bash libraries, such as getopts(1) and hashstash(1). A bash shared library that can be loaded
using
ldbash(1) must answer 4 requirments:
1. It must be installed in $LIBBASH_PREFIX/lib/bash (default is /usr/lib/bash).
2. It must contain a line that begins with '#EXPORT='. That line will contain (after the '=') a list of functions that the library
exports. I.e. all the function that will be usable after loading that library will be listed in that line.
3. It must contain a line that begins with '#REQUIRE='. That line will contain (after the '=') a list of bash libraries that are
required for our library. I.e. every bash library that is in use in our bash library must be listed there.
4. The library must be listed (For more information, see ldbashconfig(8)).
Basic guidelines for writing library of your own:
1. Be aware, that your library will be actually sourced. So, basically, it should contain (i.e define) only functions.
2. Try to declare all variables intended for internal use as local.
3. Global variables and functions that are intended for internal use (i.e are not defined in '#EXPORT=') should begin with:
__<library_name>_
For example, internal function myfoosort of hashstash library should be named as
__hashstash_myfoosort
This helps to avoid conflicts in global name space when using libraries that come from different vendors.
4. See html manual for full version of this guide.
AUTHORS
Hai Zaar <haizaar@haizaar.com>
Gil Ran <ril@ran4.net>
SEE ALSO ldbash(1), ldbashconfig(8), getopts(1), hashstash(1)colors(1)messages(1)urlcoding(1)locks(1)Linux Epoch Linux