I want to use a linux cluster to copy a list of files. I want to split the processing over 3 nodes so that each node gets (more or less) an equal share.
My script (base.sh) to execute my copy script (copy.sh) looks something like:
Code:
#!/bin/bash
for NODE in 1 2 3
do
/sw/egs/bin/dsh -w node0${NODE} -e "/home/ts/scripts/copy.sh"
done
My copy.sh file is:
Code:
#/bin/bash
IDIR="/home/ts/test2/temp"
ODIR="/home/ts/test3/
FARRAY=( "$IDIR"/*.R )
COUNT=${#FARRAY[@]}
THIS_NODE=
TOTAL_NODES=3
for i in `seq 1 $COUNT`
do
THIS_FILE=${FARRAY[$i]}
REM=`expr $i % $TOTAL_NODES`
if [ $REM -eq 0 ]
then
$REM = $TOTAL_NODES
fi
if [ $REM -eq $THIS_NODE ]
then
cp $THIS_FILE $ODIR
fi
done
My questions:
1. How do I capture which node is running this job (THIS_NODE) in copy.sh?
2. How can I modify the base.sh script so that the total number of nodes can also be passed into the copy.sh script? (sys args? - how?)
Is there a better/shorter/sleeker way to do what I am doing? Any other suggestions?
My program usage takes the form for example;
$ theApp 2 "one or more words"
i.e. 3 command line arguments; application name, an integer, some text
My code includes the following 4 lines:
int anInteger;
char words;
sscanf(argv, "%d", &anInteger);
sscanf(argv, "%s", &message);
Based... (2 Replies)
I am trying to print command line arguments one per second. I have this
while
do
echo "6"
shift
echo "5"
shift
echo "4"
shift
echo "3"
shift
echo "2"
shift
echo "1"
shift
done (2 Replies)
I have this while loop and at the end I am trying to get it to tell me the last argument I entered. And with it like this all I get is the sentence with no value for $1. Now I tried moving done after the sentence and it printed the value of $1 after every number. I don't want that I just want... (2 Replies)
Hi, I 'm trying to learn the scripting language and am trying to create a script to open a C Program, allow the user to edit it, and then run it. What I have works but only when you enter the name to be compiled and the c program, but what if you only entered the 1 argument (cprogram.c) ? but I 'm... (3 Replies)
Hi I am executing a KSH script by passing command line arguments
example: Red Green Dark Red Blue
when I am splitting the arguments by using " "(Space) as delimiter
But the colour Dark Red is a single parameter. But it is getting splitted in between
How to avoid this. Please help Also... (4 Replies)
Hi,
On linux cluster, i created a script to delete all temp files older than 5 days.
i am able to execute the script "dsh -ea script.ksh" in management node directly
But when i schedule "dsh -ea script.ksh" in crontab in management node it tells dsh command not found.
How to solve... (2 Replies)
Store args passed in array but not the first 2 args.
# bash
declare -a arr=("$@")
s=$(IFS=, eval 'echo "${arr}"')
echo "$s"
output:
sh array.sh 1 2 3 4 5 6
1,2,3,4,5,6
Desired output:
sh array.sh 1 2 3 4 5 6
3,4,5,6 (2 Replies)
Hello all,
for security reasons my compagny imposes that my script be launch remotly via ssh under the users login shell.
So serverA launches the ssh command to serverB which has a local user with my script as a login shell.
Local script works like a charm on his own.
serverB$ grep... (20 Replies)
Hi,
I'm using a while loop for a given command "bowtie2" with several parameters.
mkdir clean
paste <(ls --quote-name ./qc/sg_*_R1_val_1.fq.gz) <(ls --quote-name ./qc/sg_*_R2_val_2.fq.gz) |sed 's/"./-1 ./' | sed 's/gz"\t/gz\t -2 /' | sed 's/"//g' |\
while read args ; do
... (5 Replies)
Hi,
I am trying to run the following on our cluster:
/sw/bin/dsh -w node02 -e "/home/pc/thello.sh"
but I get an error:
dsh: no machine specified
I used to be able to run it a few months back, but they seem to have reinstalled the OS or something..
The output of the
dsh -a... (2 Replies)
Discussion started by: pc2001
2 Replies
LEARN ABOUT CENTOS
shell-quote
SHELL-QUOTE(1) User Contributed Perl Documentation SHELL-QUOTE(1)NAME
shell-quote - quote arguments for safe use, unmodified in a shell command
SYNOPSIS
shell-quote [switch]... arg...
DESCRIPTION
shell-quote lets you pass arbitrary strings through the shell so that they won't be changed by the shell. This lets you process commands
or files with embedded white space or shell globbing characters safely. Here are a few examples.
EXAMPLES
ssh preserving args
When running a remote command with ssh, ssh doesn't preserve the separate arguments it receives. It just joins them with spaces and
passes them to "$SHELL -c". This doesn't work as intended:
ssh host touch 'hi there' # fails
It creates 2 files, hi and there. Instead, do this:
cmd=`shell-quote touch 'hi there'`
ssh host "$cmd"
This gives you just 1 file, hi there.
process find output
It's not ordinarily possible to process an arbitrary list of files output by find with a shell script. Anything you put in $IFS to
split up the output could legitimately be in a file's name. Here's how you can do it using shell-quote:
eval set -- `find -type f -print0 | xargs -0 shell-quote --`
debug shell scripts
shell-quote is better than echo for debugging shell scripts.
debug() {
[ -z "$debug" ] || shell-quote "debug:" "$@"
}
With echo you can't tell the difference between "debug 'foo bar'" and "debug foo bar", but with shell-quote you can.
save a command for later
shell-quote can be used to build up a shell command to run later. Say you want the user to be able to give you switches for a command
you're going to run. If you don't want the switches to be re-evaluated by the shell (which is usually a good idea, else there are
things the user can't pass through), you can do something like this:
user_switches=
while [ $# != 0 ]
do
case x$1 in
x--pass-through)
[ $# -gt 1 ] || die "need an argument for $1"
user_switches="$user_switches "`shell-quote -- "$2"`
shift;;
# process other switches
esac
shift
done
# later
eval "shell-quote some-command $user_switches my args"
OPTIONS --debug
Turn debugging on.
--help
Show the usage message and die.
--version
Show the version number and exit.
AVAILABILITY
The code is licensed under the GNU GPL. Check http://www.argon.org/~roderick/ or CPAN for updated versions.
AUTHOR
Roderick Schertler <roderick@argon.org>
perl v5.16.3 2010-06-11 SHELL-QUOTE(1)