My apologies. I was having flashbacks to a thread dealing with ten million files in one directory; the OP refused to believe his disk couldn't be multithreaded via some magic perl or python code... Now that that's out of the way!
Do all three machines share the same disk? If not, dsh won't be useful here!
1) Your loops are overcomplicated. Instead of
Code:
ARR=( whatever ) ; for i in `seq ...`
do
Code:
for FILE in whatever/*
do
...
done
2) for FILE in whatever/*, or shell globbing in general, will fail with 'too many arguments' when there are large numbers of files. Better to use a utility like ls or find and print to a pipe when you dont' know how many files there are.
3) Don't have your sub-programs check which files are "theirs" -- tell them which files are theirs. Feed them into the program so they don't have to guess. This avoids problems with them getting out of sync (if the folder has a file added to the dir before one runs and after another runs, for example).
Code:
#!/bin/bash
NODES=3
N=0
# If there are thousands of files, '*.R' will fail in the shell with 'too many arguments'.
# So we use find instead, which prints to a pipe, avoiding using
# arguments at all.
find /home/ts/test2/temp/ -mindepth 1 -maxdepth 1 -type f -name '*.R' | while read FILE
do
echo "file $FILE goes to node $N"
echo "$FILE" >> /tmp/$$-$N
let N=(N+1)%3
done
for ((N=0; N<NODES; N++))
do
# I am assuming dsh can read from standard input here.
# If this is wrong, that wont' work :(
/sw/egs/bin/dsh -w node0${N} -e "/home/ts/scripts/copy.sh" < /tmp/$$-$N &
rm /tmp/$$-$N
done
wait
Code:
#!/bin/bash
ODIR="/home/ts/test3/
while read FILE
do
echo "Got file $FILE"
echo cp "$FILE" "$ODIR"
done
My program usage takes the form for example;
$ theApp 2 "one or more words"
i.e. 3 command line arguments; application name, an integer, some text
My code includes the following 4 lines:
int anInteger;
char words;
sscanf(argv, "%d", &anInteger);
sscanf(argv, "%s", &message);
Based... (2 Replies)
I am trying to print command line arguments one per second. I have this
while
do
echo "6"
shift
echo "5"
shift
echo "4"
shift
echo "3"
shift
echo "2"
shift
echo "1"
shift
done (2 Replies)
I have this while loop and at the end I am trying to get it to tell me the last argument I entered. And with it like this all I get is the sentence with no value for $1. Now I tried moving done after the sentence and it printed the value of $1 after every number. I don't want that I just want... (2 Replies)
Hi, I 'm trying to learn the scripting language and am trying to create a script to open a C Program, allow the user to edit it, and then run it. What I have works but only when you enter the name to be compiled and the c program, but what if you only entered the 1 argument (cprogram.c) ? but I 'm... (3 Replies)
Hi I am executing a KSH script by passing command line arguments
example: Red Green Dark Red Blue
when I am splitting the arguments by using " "(Space) as delimiter
But the colour Dark Red is a single parameter. But it is getting splitted in between
How to avoid this. Please help Also... (4 Replies)
Hi,
On linux cluster, i created a script to delete all temp files older than 5 days.
i am able to execute the script "dsh -ea script.ksh" in management node directly
But when i schedule "dsh -ea script.ksh" in crontab in management node it tells dsh command not found.
How to solve... (2 Replies)
Store args passed in array but not the first 2 args.
# bash
declare -a arr=("$@")
s=$(IFS=, eval 'echo "${arr}"')
echo "$s"
output:
sh array.sh 1 2 3 4 5 6
1,2,3,4,5,6
Desired output:
sh array.sh 1 2 3 4 5 6
3,4,5,6 (2 Replies)
Hello all,
for security reasons my compagny imposes that my script be launch remotly via ssh under the users login shell.
So serverA launches the ssh command to serverB which has a local user with my script as a login shell.
Local script works like a charm on his own.
serverB$ grep... (20 Replies)
Hi,
I'm using a while loop for a given command "bowtie2" with several parameters.
mkdir clean
paste <(ls --quote-name ./qc/sg_*_R1_val_1.fq.gz) <(ls --quote-name ./qc/sg_*_R2_val_2.fq.gz) |sed 's/"./-1 ./' | sed 's/gz"\t/gz\t -2 /' | sed 's/"//g' |\
while read args ; do
... (5 Replies)
Hi,
I am trying to run the following on our cluster:
/sw/bin/dsh -w node02 -e "/home/pc/thello.sh"
but I get an error:
dsh: no machine specified
I used to be able to run it a few months back, but they seem to have reinstalled the OS or something..
The output of the
dsh -a... (2 Replies)
Discussion started by: pc2001
2 Replies
LEARN ABOUT DEBIAN
gpgwrap
gpgwrap(1) General Commands Manual gpgwrap(1)NAME
gpgwrap - a small wrapper for gpg
SYNOPSIS
gpgwrap -V
gpgwrap -P [-v] [-i] [-a] [-p <file>]
gpgwrap -F [-v] [-i] [-a] [-c] [-p <file>] [-o <name>] [--] <file> [<file> ... ]
gpgwrap [-v] [-i] [-a] [-p <file>] [-o <name>] [--] gpg [gpg options]
DESCRIPTION
The GNU Privacy Guard (gpg) supplies the option --passphrase-fd. This instructs gpg to read the passphrase from the given file descriptor.
Usually this file descriptor is opened before gpg is executed via execvp(3). Exactly that is what gpgwrap is doing. The passphrase may be
passed to gpgwrap in 4 ways:
* as file path, whereat the passphrase is stored as plain text in the file
* it is piped from another program to the stdin of gpgwrap
* through the GPGWRAP_PASSPHRASE environment variable
* gpgwrap prompts for it
With no precautions the first point undermines the secure infrastructure gpg provides. But in pure batch oriented environments this may be
what you want. Otherwise if you are willing to enter passphrases once and don't want them to be stored as plain text in a file gpg-agent is
what you are looking for. Another security objection could be the use of the environment variable GPGWRAP_PASSPHRASE which contains the
passphrase and may be read by other processes of the same user.
OPTIONS -V, --version
Print out version and exit.
-P, --print
Get the passphrase and print it mangled to stdout.
-F, --file
Read gpg commands from the given files. If <file> is - it is read from stdin. Exactly one command per line is expected. The given
line is handled in the following way:
* In the first place the passphrase is mangled. This means that unusual characters are replaced by their backslash escaped octal
numbers.
* Secondly the mangled passphrase is stored in the environment variable GPGWRAP_PASSPHRASE.
* "exec gpgwrap -- " is prepended to each line, before the result is passed as argument to "sh -c".
-h, --help
Print out usage information.
-v, --verbose
Increase verbosity level.
-i, --interactive
Always prompt for passphrase (ignores -p and the environment variable).
-a, --ask-twice
Ask twice if prompting for a passphrase.
-c, --check-exit-code
While reading gpg commands from a file, gpgwrap ignores per default the exit code of its child processes. This option enables the
check of the exit code. If a child terminates abnormal or with an exit code not equal 0 gpgwrap stops immediately and does return
with this exit code. See also section BUGS.
-p <file>, --passphrase-file <file>
Read passphrase from <file>. If <file> is - it is read from stdin. The passphrase is expected to be in plain text. If this option
is not given the passphrase will be taken either from the environment variable GPGWRAP_PASSPHRASE or it will be prompted on the
controlling tty if the environment variable is not set.
-o <name>, --option-name <name>
Specify the name of the "--passphrase-fd" option understood by the program to be executed. This is useful if you want to use gpg-
wrap in combination with other programs than gpg.
LIMITATIONS
The given passphrase is subject to several limitations depending on the way it was passed to gpgwrap:
* There is a size limitation: the passphrase should be not larger than some kilobytes (examine the source code for the exact limit).
* gpgwrap allows you to use all characters in a passphrase even 00, but this does not mean that gpg will accept it. gpg may reject
your passphrase or may only read a part of it, if it contains characters like 12 (in C also known as
).
* If you set the environment variable GPGWRAP_PASSPHRASE you should take special care with the backslash character, because gpgwrap
uses backslash to escape octal numbers, (see option -F). Therefore write backslash itself as octal number: 134.
EXAMPLES
1.
gpgwrap -p /path/to/a/secret/file
gpg -c -z 0 --batch --no-tty
--cipher-algo blowfish < infile > outfile
Read passphrase from /path/to/a/secret/file and execute gpg to do symmetric encryption of infile and write it to outfile.
2.
gpgwrap -i -a
gpg -c -z 0 --batch --no-tty
--cipher-algo blowfish < infile > outfile
Same as above except that gpgwrap prompts twice for the passphrase.
3.
gpgwrap -F -i - <<EOL
gpg --decrypt --batch --no-tty < "$HOME/infile1" > "$HOME/outfile1"
gpg --decrypt --batch --no-tty < "$HOME/infile2" > "$HOME/outfile2"
gpg --decrypt --batch --no-tty < "$HOME/infile3" > "$HOME/outfile3"
gpg --decrypt --batch --no-tty < "$HOME/infile4" > "$HOME/outfile4"
EOL
gpgwrap prompts for the passphrase and executes four instances of gpg to decrypt the given files.
4.
GPGWRAP_PASSPHRASE="mysecretpassphrase"
export GPGWRAP_PASSPHRASE
gpgwrap -F -c -v /tmp/cmdfile1 - /tmp/cmdfile2 <<EOL
gpg --decrypt --batch --no-tty < "$HOME/infile1" > "$HOME/outfile1"
gpg --decrypt --batch --no-tty < "$HOME/infile2" > "$HOME/outfile2"
gpg --decrypt --batch --no-tty < "$HOME/infile3" > "$HOME/outfile3"
gpg --decrypt --batch --no-tty < "$HOME/infile4" > "$HOME/outfile4"
EOL
Same as above except that gpgwrap gets the passphrase via the environment variable, reads commands additionally from other files
and checks the exit code of every gpg instance. This means if one gpg command has a non zero exit code, no further commands are
executed. Furthermore gpgwrap produces verbose output.
5.
GPGWRAP_PASSPHRASE="$(gpgwrap -P -i -a)"
export GPGWRAP_PASSPHRASE
find . -maxdepth 1 -type f |
while read FILE; do
FILE2="$FILE.bz2.gpg"
bzip2 -c "$FILE" |
gpgwrap gpg -c -z 0 --batch --no-tty
--cipher-algo blowfish > "$FILE2" &&
touch -r "$FILE" "$FILE2" &&
rm -f "$FILE"
done
Read in passphrase, compress all files in the current directory, encrypt them and keep date from original file.
6.
find . -maxdepth 1 -type f -name '*.bz2.gpg' |
awk '{
printf("gpg --decrypt --batch --no-tty --quiet ");
printf("--no-secmem-warning < %s
", $0);
}' |
gpgwrap -F -i -c - |
bzip2 -d -c - |
grep -i 'data'
Decrypt all *.bz2.gpg files in the current directory, decompress them and print out all occurances of data. If you pipe the result
to less you get into trouble because gpgwrap and less try to read from the TTY at the same time. In such a case it is better to use
the environment variable to give the passphrase (the example above shows how to do this).
7.
GPGWRAP_PASSPHRASE="$(gpgwrap -P -i -a)"
export GPGWRAP_PASSPHRASE
gpgwrap -P |
ssh -C -x -P -l user host "
GPGWRAP_PASSPHRASE="$(cat)"
...
"
Prompt for a passphrase twice and write it to the GPGWRAP_PASSPHRASE environment variable.
8.
echo -n "Passphrase: "
stty -echo
read GPGWRAP_PASSPHRASE
echo
stty echo
export GPGWRAP_PASSPHRASE
Another way to prompt manually for the passphrase. It was needed in combination with older versions of gpgwrap, because they did
not upport -P. Be aware that with this method no automatic conversion to backslash escaped octal numbers takes place.
9.
echo "mysecretpassphrase" |
gpg --batch --no-tty --passphrase-fd 0
--output outfile --decrypt infile
Cheap method to give passphrase to gpg without gpgwrap. Note that you can't use stdin to pass a file to gpg, because stdin is
already used for the passphrase.
10.
gpg --batch --no-tty
--passphrase-fd 3 3< /path/to/a/secret/file
< infile > outfile
This is a more advanced method to give the passphrase, it is equivalent to Option -p of gpgwrap. This example should at least work
with the bash.
11.
gpg --batch --no-tty --passphrase-fd 3
3< <(echo "mysecretpassphrase")
< infile > outfile
Like above, but the passphrase is given directly. This example should at least work with the bash.
BUGS
In version 0.02 of gpgwrap the exit code of gpg was only returned if gpgwrap read the passphrase from a file. Since version 0.03, only -F
omits exit code checking by default, but it can be enabled with -c.
SEE ALSO
gpg, gpg-agent
AUTHOR
Karsten Scheibler
gpgwrap 0.04 gpgwrap(1)
06-13-2014
Now that that's out of the way!