Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Parallel processing in AIX (bash shell scripting)

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parallel processing in AIX (bash shell scripting)
# 1  
Old 05-03-2018
Parallel processing in AIX (bash shell scripting)
Hi all,

I am struggling to build a utility which can do the parallel processing. I achieved same in Linux using xargs -P but same is not working on AIX.

I am building file copy utility where I will have all required info in a file (like souce file info and target location details), now i need to copy files which should run in parallel like 50 files copy in parallel.

your guidance is very much appreciated.

Thanks
# 2  
Old 05-03-2018
Did you consider running the copy commands in a loop, sending some (not all!) into background?

Don't expect too much from 50 parallel copy actions performance wise - disk operation and memory might be the constraints.
# 3  
Old 05-03-2018
Quote:
Originally Posted by ankur singh
I am building file copy utility where I will have all required info in a file (like souce file info and target location details), now i need to copy files which should run in parallel like 50 files copy in parallel.
There's usually no point trying to copy files in parallel. CPU is almost always faster than disk. Parallelizing can easily slow it down due to disk thrashing, and poison your filesystem with large amounts of fragmentation. I only know two situations where it's a win:
  • File operations with lots of CPU overhead, like extracting many large bzip files.
  • Software disk mirrors in Linux, where processes get arbitrarily assigned one disk each. (Only useful for reading.)

So could you explain your situation in more detail, please, so we can find out if this is one of those rare situations you'll come out ahead?
Last edited by Corona688; 05-03-2018 at 04:04 PM..
These 2 Users Gave Thanks to Corona688 For This Post:
rbatte1 RudiC
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parallel Runs in UNIX/Linix Shell Scripting

HI, I have a file(suppose 1 million Records), i wanted to divide file into chunks(100 small files), these all files should run parllell(e.x like threads) to utilize the process for fast processing. Can you please provide a solution for the above issue. Ex: Split -b 2m file.txt file,... (1 Reply)
Discussion started by: prawinmca
1 Replies

2. Shell Programming and Scripting

Parallel processing

I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies

3. Shell Programming and Scripting

Parallel processing of SQL through Shell

Hi Friends, I am trying to write a shell which will invoke 3 CTAS (ORACLE create table XXX as select * from YYYY). The approximate time for one CTAS is around 25 mins. So i want to run the CTAS script parallely. My pseudocode is as below. Main script nohup sh CTAS1.sh &... (3 Replies)
Discussion started by: Showdown
3 Replies

4. Shell Programming and Scripting

Parallel processing in bash?

Hi Say I am interested in processing a big data set over shell, and each process individually takes a long time, but many such processes can be pipe-lined, is there a way to do this automatically or efficiently in shell? For example consider pinging a list addresses upto 5 times each. Of... (5 Replies)
Discussion started by: jamie_123
5 Replies

5. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

6. Shell Programming and Scripting

parallel processing

hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies

7. Shell Programming and Scripting

shell scripting-processing multiple scripts

I have four scripts to run. My 1st script will make script2 and script3 to run. I am setting a cron job for this script1 to run continuously. This script1 will check for 2 text files and based on the existance of those text files it will initiate the script2 and script3. Now my doubt is that... (2 Replies)
Discussion started by: RSC1985
2 Replies

8. Shell Programming and Scripting

parallel processing

Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies

9. Shell Programming and Scripting

difference between AIX shell scripting and Unix shell scripting.

please give the difference between AIX shell scripting and Unix shell scripting. (2 Replies)
Discussion started by: haroonec
2 Replies
Login or Register to Ask a Question