Sponsored Content
Top Forums Shell Programming and Scripting Parallel processing in AIX (bash shell scripting) Post 303016774 by ankur singh on Thursday 3rd of May 2018 01:25:49 PM
Old 05-03-2018
Parallel processing in AIX (bash shell scripting)

Hi all,

I am struggling to build a utility which can do the parallel processing. I achieved same in Linux using xargs -P but same is not working on AIX.

I am building file copy utility where I will have all required info in a file (like souce file info and target location details), now i need to copy files which should run in parallel like 50 files copy in parallel.

your guidance is very much appreciated.

Thanks
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

difference between AIX shell scripting and Unix shell scripting.

please give the difference between AIX shell scripting and Unix shell scripting. (2 Replies)
Discussion started by: haroonec
2 Replies

2. Shell Programming and Scripting

parallel processing

Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies

3. Shell Programming and Scripting

shell scripting-processing multiple scripts

I have four scripts to run. My 1st script will make script2 and script3 to run. I am setting a cron job for this script1 to run continuously. This script1 will check for 2 text files and based on the existance of those text files it will initiate the script2 and script3. Now my doubt is that... (2 Replies)
Discussion started by: RSC1985
2 Replies

4. Shell Programming and Scripting

parallel processing

hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies

5. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

6. Shell Programming and Scripting

Parallel processing in bash?

Hi Say I am interested in processing a big data set over shell, and each process individually takes a long time, but many such processes can be pipe-lined, is there a way to do this automatically or efficiently in shell? For example consider pinging a list addresses upto 5 times each. Of... (5 Replies)
Discussion started by: jamie_123
5 Replies

7. Shell Programming and Scripting

Parallel processing of SQL through Shell

Hi Friends, I am trying to write a shell which will invoke 3 CTAS (ORACLE create table XXX as select * from YYYY). The approximate time for one CTAS is around 25 mins. So i want to run the CTAS script parallely. My pseudocode is as below. Main script nohup sh CTAS1.sh &... (3 Replies)
Discussion started by: Showdown
3 Replies

8. Shell Programming and Scripting

Parallel processing

I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies

9. Shell Programming and Scripting

Parallel Runs in UNIX/Linix Shell Scripting

HI, I have a file(suppose 1 million Records), i wanted to divide file into chunks(100 small files), these all files should run parllell(e.x like threads) to utilize the process for fast processing. Can you please provide a solution for the above issue. Ex: Split -b 2m file.txt file,... (1 Reply)
Discussion started by: prawinmca
1 Replies
PARALLEL-SCP(1) 														   PARALLEL-SCP(1)

NAME
parallel-scp - parallel versions of scp SYNOPSIS
parallel-scp [OPTIONS] -h hosts.txt local remote DESCRIPTION
pssh provides a number of commands for executing against a group of computers, using SSH. It's most useful for operating on clusters of homogenously-configured hosts. parallel-scp copy files in parallel to a set of machines. OPTIONS
-r --recursive recusively copy directories (OPTIONAL) -h --hosts hosts file (each line "host[:port] [user]") -l --user username (OPTIONAL) -p --par max number of parallel threads (OPTIONAL) -o --outdir output directory for stdout files (OPTIONAL) -e --errdir output directory for stderr files (OPTIONAL) -t --timeout timeout (secs) (-1 = no timeout) per host (OPTIONAL) -O --options SSH options (OPTIONAL) -v --verbose turn on warning and diagnostic messages (OPTIONAL) EXAMPLE
The following example runs hostname on three machines (IPs or hostnames) specified in the file ips.txt using login irb2 and saves the output in /tmp/foo. # parallel-scp -h ips.txt -l irb2 /etc/hosts /tmp/hosts Success on 128.112.152.122:22 Success on 18.31.0.190:22 Success on 128.232.103.201:22 ENVIRONMENT
All four programs take similar sets of options. All of these options can be set using the following environment variables: o PSSH_HOSTS o PSSH_USER o PSSH_PAR o PSSH_OUTDIR o PSSH_VERBOSE o PSSH_OPTIONS SEE ALSO
parallel-ssh(1), parallel-slurp(1), parallel-nuke(1), parallel-rsync(1), ssh(1), scp(1), AUTHOR
Brent N. Chun <bnc@theether.org> COPYING
Copyright: 2003, 2004, 2005, 2006, 2007 Brent N. Chun NOTES
1. bnc@theether.org mailto:bnc@theether.org 03/30/2009 PARALLEL-SCP(1)
All times are GMT -4. The time now is 08:16 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy