Sponsored Content
Top Forums Shell Programming and Scripting Parallel processing in AIX (bash shell scripting) Post 303016783 by Corona688 on Thursday 3rd of May 2018 02:55:49 PM
Old 05-03-2018
Quote:
Originally Posted by ankur singh
I am building file copy utility where I will have all required info in a file (like souce file info and target location details), now i need to copy files which should run in parallel like 50 files copy in parallel.
There's usually no point trying to copy files in parallel. CPU is almost always faster than disk. Parallelizing can easily slow it down due to disk thrashing, and poison your filesystem with large amounts of fragmentation. I only know two situations where it's a win:
  • File operations with lots of CPU overhead, like extracting many large bzip files.
  • Software disk mirrors in Linux, where processes get arbitrarily assigned one disk each. (Only useful for reading.)

So could you explain your situation in more detail, please, so we can find out if this is one of those rare situations you'll come out ahead?

Last edited by Corona688; 05-03-2018 at 04:04 PM..
These 2 Users Gave Thanks to Corona688 For This Post:
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

difference between AIX shell scripting and Unix shell scripting.

please give the difference between AIX shell scripting and Unix shell scripting. (2 Replies)
Discussion started by: haroonec
2 Replies

2. Shell Programming and Scripting

parallel processing

Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies

3. Shell Programming and Scripting

shell scripting-processing multiple scripts

I have four scripts to run. My 1st script will make script2 and script3 to run. I am setting a cron job for this script1 to run continuously. This script1 will check for 2 text files and based on the existance of those text files it will initiate the script2 and script3. Now my doubt is that... (2 Replies)
Discussion started by: RSC1985
2 Replies

4. Shell Programming and Scripting

parallel processing

hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies

5. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

6. Shell Programming and Scripting

Parallel processing in bash?

Hi Say I am interested in processing a big data set over shell, and each process individually takes a long time, but many such processes can be pipe-lined, is there a way to do this automatically or efficiently in shell? For example consider pinging a list addresses upto 5 times each. Of... (5 Replies)
Discussion started by: jamie_123
5 Replies

7. Shell Programming and Scripting

Parallel processing of SQL through Shell

Hi Friends, I am trying to write a shell which will invoke 3 CTAS (ORACLE create table XXX as select * from YYYY). The approximate time for one CTAS is around 25 mins. So i want to run the CTAS script parallely. My pseudocode is as below. Main script nohup sh CTAS1.sh &... (3 Replies)
Discussion started by: Showdown
3 Replies

8. Shell Programming and Scripting

Parallel processing

I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies

9. Shell Programming and Scripting

Parallel Runs in UNIX/Linix Shell Scripting

HI, I have a file(suppose 1 million Records), i wanted to divide file into chunks(100 small files), these all files should run parllell(e.x like threads) to utilize the process for fast processing. Can you please provide a solution for the above issue. Ex: Split -b 2m file.txt file,... (1 Reply)
Discussion started by: prawinmca
1 Replies
xfs_copy(8)						      System Manager's Manual						       xfs_copy(8)

NAME
xfs_copy - copy the contents of an XFS filesystem SYNOPSIS
xfs_copy [ -bd ] [ -L log ] source target1 [ target2 ... ] DESCRIPTION
xfs_copy copies an XFS filesystem to one or more targets in parallel (see xfs(5)). The first (source) argument must be the pathname of the device or file containing the XFS filesystem. The remaining arguments specify one or more target devices or file names. If the pathnames specify devices, a copy of the source XFS filesystem is created on each device. The target can also be the name of a regular file, in which case an image of the source XFS filesystem is created in that file. If the file does not exist, xfs_copy creates the file. The length of the resulting file is equal to the size of the source filesystem. However, if the file is created on an XFS filesystem, the file consumes roughly the amount of space actually used in the source filesystem by the filesystem and the XFS log. The space saving is because xfs_copy seeks over free blocks instead of copying them and the XFS filesystem supports sparse files efficiently. xfs_copy should only be used to copy unmounted filesystems, read-only mounted filesystems, or frozen filesystems (see xfs_freeze(8)). Oth- erwise, the generated filesystem(s) would be inconsistent or corrupt. xfs_copy does not alter the source filesystem in any way. Each new (target) filesystem is identical to the original filesystem except that new filesystems each have a new unique filesystem identifier (UUID). Therefore, if both the old and new filesystems will be used as sepa- rate distinct filesystems, xfs_copy or xfsdump(8)/xfsrestore(8) should be used to generate the new filesystem(s) instead of dd(1) or other programs that do block-by-block disk copying. xfs_copy uses synchronous writes to ensure that write errors are detected. xfs_copy uses pthreads(7) to perform simultaneous parallel writes. xfs_copy creates one additional thread for each target to be written. All threads die if xfs_copy terminates or aborts. OPTIONS
-d Create a duplicate (true clone) filesystem. This should be done only if the new filesystem will be used as a replacement for the original filesystem (such as in the case of disk replacement). -b The buffered option can be used to ensure direct IO is not attempted to any of the target files. This is useful when the filesystem holding the target file does not support direct IO. -L log Specifies the location of the log if the default location of /var/tmp/xfs_copy.log.XXXXXX is not desired. DIAGNOSTICS
xfs_copy reports errors to both stderr and in more detailed form to a generated log file whose name is of the form /var/tmp/xfs_copy.log.XXXXXX or a log file specified by the -L option. If xfs_copy detects a write error on a target, the copy of that one target is aborted and an error message is issued to both stderr and the log file, but the rest of the copies continue. When xfs_copy termi- nates, all aborted targets are reported to both stderr and the log file. If all targets abort or if there is an error reading the source filesystem, xfs_copy immediately aborts. xfs_copy returns an exit code of 0 if all targets are successfully copied and an exit code of 1 if any target fails. NOTES
When moving filesystems from one disk to another, if the original filesystem is significantly smaller than the new filesystem, and will be made larger, we recommend that mkfs.xfs(8) and xfsdump(8)/xfsrestore(8) be used instead of using xfs_copy and xfs_growfs(8). The filesys- tem layout resulting from using xfs_copy/xfs_growfs is almost always worse than the result of using mkfs.xfs/xfsdump/xfsrestore but in the case of small filesystems, the differences can have a significant performance impact. This is due to the way xfs_growfs(8) works, and not due to any shortcoming in xfs_copy itself. CAVEATS
xfs_copy does not copy XFS filesystems that have a real-time section or XFS filesystems with external logs. In both cases, xfs_copy aborts with an error message. SEE ALSO
mkfs.xfs(8), xfsdump(8), xfsrestore(8), xfs_freeze(8), xfs_growfs(8), xfs(5). xfs_copy(8)
All times are GMT -4. The time now is 08:15 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy