parallel processing

Tags
shell scripts

 
Thread Tools Search this Thread
# 1  
Old 09-04-2009
parallel processing

hi i am preparing a set of batches for a set of files sequentially
There is a folder /xyz where all the files reside
now all the files starting with
01 - will be appended for one below other to form a batch batch01
then all the files starting with
02 - will be appended for one below other to form a batch batch02
then
03 - will be appended for one below other to form a batch batch03
then
04 - will be appended for one below other to form a batch batch04
..

and so on, now this is taking a lot of time while processing.
How can i improve the performance say include some type of parallel processing
to minimize the time.
Please Advice
# 2  
Old 09-04-2009

Post your script so we can see what you are doing wrong.
# 3  
Old 09-04-2009
Please find the script below

Code:
name00=ABC00`date +"%y%m%d%H"`
for i in 0[0]*.[tT][xX][tT]
do
cat ${i} >> ${name00}            
done

name01=ABC01`date +"%y%m%d%H"`
for i in 0[1]*.[tT][xX][tT]
do
cat ${i} >> ${name01}            
done
 

name02=ABC02`date +"%y%m%d%H"`
for i in 0[2]*.[tT][xX][tT]
do
cat ${i} >> ${name02}            
done

and so on

Last edited by bakunin; 09-04-2009 at 10:02 AM.. Reason: generously provided code-tags to the needy - please spend them yourself from now on
# 4  
Old 09-04-2009
Your code would probably benefit from the usage of "find" and the elimination of repetitive tasks like expanding the date over and over again:

Code:
chDate="$(date +"%y%m%d%H")"
typeset -Z2 iCounter=0

while [ $iCounter -le 99 ] ; do
     find /your/directory -name "${iCounter}*[tT][xX][tT]" -print > "ABC${iCounter}${chDate}"
     (( iCounter += 1 ))
done

I put an arbitrary end at 99 for demonstration purposes, adapt the script to what you really need. If this is not fast enough try backgrounding the "find"s by adding a " &" at the end of the line starting with "find".

I hope this helps.

bakunin
# 5  
Old 09-04-2009
First, what type of file system is your /xyz directory? What is the underlying hardware? How busy is it when you're running your script?

If your hardware is already maxed out, it's already maxed out and parallel processing won't help. In fact, it could even slow it down further and you'll likely get more disk contention.
# 6  
Old 09-04-2009
Code:
name00=ABC00`date +"%Y%m%d%h"`
cat 00*.[tT][xX][tT] > "$name00"

name01=ABC01`date +"%Y%m%d%h"`
cat 01*.[tT][xX][tT] > "$name01"

name02=ABC02`date +"%Y%m%d%h"`
cat 02*.[tT][xX][tT] > "$name02"

Or:

Code:
for n in 01 02 03 04 ...
do
  name=ABC$n`date +"%Y%m%d%h"`
  cat "$n"*.[tT][xX][tT] > "$name"
done

# 7  
Old 09-04-2009
Your code would probably benefit from the usage of "find" and the elimination of repetitive tasks like expanding the date over and over again:

Code:
chDate="$(date +"%y%m%d%H")"
typeset -Z2 iCounter=0

while [ $iCounter -le 99 ] ; do
     find /your/directory -name "${iCounter}*.[tT][xX][tT]" -print > "ABC${iCounter}${chDate}"
     (( iCounter += 1 ))
done

I put an arbitrary end at 99 for demonstration purposes, adapt the script to what you really need. If this is not fast enough try backgrounding the "find"s by adding a " &" at the end of the line starting with "find".

I hope this helps.

bakunin

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Parallel processing garethsays Shell Programming and Scripting 1 03-19-2016 09:02 AM
Parallel processing and error checking wahi80 Shell Programming and Scripting 7 02-16-2016 09:31 PM
Parallel processing - continued kumarjt Shell Programming and Scripting 3 08-21-2013 06:10 AM
Parallel processing of SQL through Shell Showdown Shell Programming and Scripting 3 08-21-2013 06:04 AM
Implement parallel processing kumarjt Shell Programming and Scripting 13 07-20-2013 06:42 AM
Parallel processing pthread temursalin Programming 0 07-12-2012 04:44 PM
Parallel processing in bash? jamie_123 Shell Programming and Scripting 5 06-14-2012 12:31 PM
PARALLEL PROCESSING IN PERL gvk25 Shell Programming and Scripting 0 04-14-2012 02:01 AM
Algorithms for Parallel Processing azar.zorn Programming 2 07-03-2011 12:18 PM
script parallel processing machpee Shell Programming and Scripting 2 06-04-2011 11:04 PM
How to make parallel processing rather than serial processing ?? Samb95 Shell Programming and Scripting 0 08-16-2010 04:56 AM
Need Help With Parallel Processing imnewtothis23 Shell Programming and Scripting 6 08-07-2008 02:15 AM
how to achieve following parallel processing thru unix manas_ranjan Shell Programming and Scripting 3 09-04-2007 08:54 AM
parallel processing audippa Shell Programming and Scripting 3 03-01-2007 12:01 PM
How to do parallel processing?? zing UNIX for Dummies Questions & Answers 1 06-23-2003 01:47 PM