05-03-2018
Parallel processing in AIX (bash shell scripting)
Hi all,
I am struggling to build a utility which can do the parallel processing. I achieved same in Linux using xargs -P but same is not working on AIX.
I am building file copy utility where I will have all required info in a file (like souce file info and target location details), now i need to copy files which should run in parallel like 50 files copy in parallel.
your guidance is very much appreciated.
Thanks
9 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
please give the difference between AIX shell scripting and Unix shell scripting. (2 Replies)
Discussion started by: haroonec
2 Replies
2. Shell Programming and Scripting
Hi
I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls
Regards
Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies
3. Shell Programming and Scripting
I have four scripts to run.
My 1st script will make script2 and script3 to run. I am setting a cron job for this script1 to run continuously.
This script1 will check for 2 text files and based on the existance of those text files it will initiate the script2 and script3.
Now my doubt is that... (2 Replies)
Discussion started by: RSC1985
2 Replies
4. Shell Programming and Scripting
hi i am preparing a set of batches for a set of files sequentially
There is a folder /xyz where all the files reside
now all the files starting with
01 - will be appended for one below other to form a batch batch01
then all the files starting with
02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies
5. Shell Programming and Scripting
Hello everybody,
I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel.
My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies
6. Shell Programming and Scripting
Hi
Say I am interested in processing a big data set over shell, and each process individually takes a long time, but many such processes can be pipe-lined, is there a way to do this automatically or efficiently in shell?
For example consider pinging a list addresses upto 5 times each. Of... (5 Replies)
Discussion started by: jamie_123
5 Replies
7. Shell Programming and Scripting
Hi Friends,
I am trying to write a shell which will invoke 3 CTAS (ORACLE create table XXX as select * from YYYY). The approximate time for one CTAS is around 25 mins. So i want to run the CTAS script parallely.
My pseudocode is as below.
Main script
nohup sh CTAS1.sh &... (3 Replies)
Discussion started by: Showdown
3 Replies
8. Shell Programming and Scripting
I have 10,000 + files, each of which I need to zip using bzip2.
Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies
9. Shell Programming and Scripting
HI,
I have a file(suppose 1 million Records), i wanted to divide file into chunks(100 small files), these all files should run parllell(e.x like threads) to utilize the process for fast processing.
Can you please provide a solution for the above issue.
Ex: Split -b 2m file.txt file,... (1 Reply)
Discussion started by: prawinmca
1 Replies
LEARN ABOUT DEBIAN
parallel-scp
PARALLEL-SCP(1) PARALLEL-SCP(1)
NAME
parallel-scp - parallel versions of scp
SYNOPSIS
parallel-scp [OPTIONS] -h hosts.txt local remote
DESCRIPTION
pssh provides a number of commands for executing against a group of computers, using SSH. It's most useful for operating on clusters of
homogenously-configured hosts.
parallel-scp copy files in parallel to a set of machines.
OPTIONS
-r --recursive
recusively copy directories (OPTIONAL)
-h --hosts
hosts file (each line "host[:port] [user]")
-l --user
username (OPTIONAL)
-p --par
max number of parallel threads (OPTIONAL)
-o --outdir
output directory for stdout files (OPTIONAL)
-e --errdir
output directory for stderr files (OPTIONAL)
-t --timeout
timeout (secs) (-1 = no timeout) per host (OPTIONAL)
-O --options
SSH options (OPTIONAL)
-v --verbose
turn on warning and diagnostic messages (OPTIONAL)
EXAMPLE
The following example runs hostname on three machines (IPs or hostnames) specified in the file ips.txt using login irb2 and saves the
output in /tmp/foo.
# parallel-scp -h ips.txt -l irb2 /etc/hosts /tmp/hosts
Success on 128.112.152.122:22
Success on 18.31.0.190:22
Success on 128.232.103.201:22
ENVIRONMENT
All four programs take similar sets of options. All of these options can be set using the following environment variables:
o PSSH_HOSTS
o PSSH_USER
o PSSH_PAR
o PSSH_OUTDIR
o PSSH_VERBOSE
o PSSH_OPTIONS
SEE ALSO
parallel-ssh(1), parallel-slurp(1), parallel-nuke(1), parallel-rsync(1), ssh(1), scp(1),
AUTHOR
Brent N. Chun <bnc@theether.org>
COPYING
Copyright: 2003, 2004, 2005, 2006, 2007 Brent N. Chun
NOTES
1. bnc@theether.org
mailto:bnc@theether.org
03/30/2009 PARALLEL-SCP(1)