My apologies. I was having flashbacks to a thread dealing with ten million files in one directory; the OP refused to believe his disk couldn't be multithreaded via some magic perl or python code... Now that that's out of the way!
Do all three machines share the same disk? If not, dsh won't be useful here!
1) Your loops are overcomplicated. Instead of
Code:
ARR=( whatever ) ; for i in `seq ...`
do
Code:
for FILE in whatever/*
do
...
done
2) for FILE in whatever/*, or shell globbing in general, will fail with 'too many arguments' when there are large numbers of files. Better to use a utility like ls or find and print to a pipe when you dont' know how many files there are.
3) Don't have your sub-programs check which files are "theirs" -- tell them which files are theirs. Feed them into the program so they don't have to guess. This avoids problems with them getting out of sync (if the folder has a file added to the dir before one runs and after another runs, for example).
Code:
#!/bin/bash
NODES=3
N=0
# If there are thousands of files, '*.R' will fail in the shell with 'too many arguments'.
# So we use find instead, which prints to a pipe, avoiding using
# arguments at all.
find /home/ts/test2/temp/ -mindepth 1 -maxdepth 1 -type f -name '*.R' | while read FILE
do
echo "file $FILE goes to node $N"
echo "$FILE" >> /tmp/$$-$N
let N=(N+1)%3
done
for ((N=0; N<NODES; N++))
do
# I am assuming dsh can read from standard input here.
# If this is wrong, that wont' work :(
/sw/egs/bin/dsh -w node0${N} -e "/home/ts/scripts/copy.sh" < /tmp/$$-$N &
rm /tmp/$$-$N
done
wait
Code:
#!/bin/bash
ODIR="/home/ts/test3/
while read FILE
do
echo "Got file $FILE"
echo cp "$FILE" "$ODIR"
done
My program usage takes the form for example;
$ theApp 2 "one or more words"
i.e. 3 command line arguments; application name, an integer, some text
My code includes the following 4 lines:
int anInteger;
char words;
sscanf(argv, "%d", &anInteger);
sscanf(argv, "%s", &message);
Based... (2 Replies)
I am trying to print command line arguments one per second. I have this
while
do
echo "6"
shift
echo "5"
shift
echo "4"
shift
echo "3"
shift
echo "2"
shift
echo "1"
shift
done (2 Replies)
I have this while loop and at the end I am trying to get it to tell me the last argument I entered. And with it like this all I get is the sentence with no value for $1. Now I tried moving done after the sentence and it printed the value of $1 after every number. I don't want that I just want... (2 Replies)
Hi, I 'm trying to learn the scripting language and am trying to create a script to open a C Program, allow the user to edit it, and then run it. What I have works but only when you enter the name to be compiled and the c program, but what if you only entered the 1 argument (cprogram.c) ? but I 'm... (3 Replies)
Hi I am executing a KSH script by passing command line arguments
example: Red Green Dark Red Blue
when I am splitting the arguments by using " "(Space) as delimiter
But the colour Dark Red is a single parameter. But it is getting splitted in between
How to avoid this. Please help Also... (4 Replies)
Hi,
On linux cluster, i created a script to delete all temp files older than 5 days.
i am able to execute the script "dsh -ea script.ksh" in management node directly
But when i schedule "dsh -ea script.ksh" in crontab in management node it tells dsh command not found.
How to solve... (2 Replies)
Store args passed in array but not the first 2 args.
# bash
declare -a arr=("$@")
s=$(IFS=, eval 'echo "${arr}"')
echo "$s"
output:
sh array.sh 1 2 3 4 5 6
1,2,3,4,5,6
Desired output:
sh array.sh 1 2 3 4 5 6
3,4,5,6 (2 Replies)
Hello all,
for security reasons my compagny imposes that my script be launch remotly via ssh under the users login shell.
So serverA launches the ssh command to serverB which has a local user with my script as a login shell.
Local script works like a charm on his own.
serverB$ grep... (20 Replies)
Hi,
I'm using a while loop for a given command "bowtie2" with several parameters.
mkdir clean
paste <(ls --quote-name ./qc/sg_*_R1_val_1.fq.gz) <(ls --quote-name ./qc/sg_*_R2_val_2.fq.gz) |sed 's/"./-1 ./' | sed 's/gz"\t/gz\t -2 /' | sed 's/"//g' |\
while read args ; do
... (5 Replies)
Hi,
I am trying to run the following on our cluster:
/sw/bin/dsh -w node02 -e "/home/pc/thello.sh"
but I get an error:
dsh: no machine specified
I used to be able to run it a few months back, but they seem to have reinstalled the OS or something..
The output of the
dsh -a... (2 Replies)
Discussion started by: pc2001
2 Replies
LEARN ABOUT DEBIAN
surfraw-update-path
SURFRAW-UPDATE-PATH(1) User Contributed Perl Documentation SURFRAW-UPDATE-PATH(1)NAME
surfraw-update-path - updates PATH in shell config files
SYNOPSIS
surfraw-update-path [-add] [-remove] [-check] [-sys] [-all] [-help] [-shell=SHELL]
DESCRIPTION
surfraw-update-path adds the surfraw elvi directory (/usr/lib/surfraw) to your PATH in your shell's config file.
Currently it supports bash, sh, csh, tcsh, ash, dash, ksh, pdksh, zsh, rc, and es
Don't forget to login again or source your login files for it to take effect.
OPTIONS -check
Checks to see if the surfraw config code is present. This is the default.
-add
Adds the surfraw config code.
-remove
Removes the surfraw config code
-sys
Updates the system-wide shell config instead of the user. Must be done as root.
-shell=SHELL
Selects the shell to configure. Defaults to the value of the $SHELL environment variable.
Currently supported shells are:
sh, ash, bash, dash, csh, tcsh, ksh, pdksh, zsh, rc, and es.
-all
Attempts to configure the startup files for all known shells
-help
Gives a usage message
RETURN VALUE -check returns 0 if the surfraw code is present in the file, 1 if it is not found, or 2 on error.
All other options return 0 on success, or 2 on error.
ENVIRONMENT
SHELL
Used to determine which shell to configure, if -shell is not given.
HOME
Used to find users config files.
ENV Used by posix-compliant shells to specify a startup rc file.
ZDOTDIR
Used to find user config files for zsh. If not set, defaults to HOME.
SEE ALSO surfraw(1), sh(1), ash(1), bash(1), dash(1), csh(1), tcsh(1), ksh(1), pdksh(1), zsh(1), rc(1), es(1)AUTHOR
Ian Beckwith <ianb@erislabs.net>
perl v5.12.4 2011-07-12 SURFRAW-UPDATE-PATH(1)
06-13-2014
Now that that's out of the way!