parallel processing


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
parallel processing

hi i am preparing a set of batches for a set of files sequentially
There is a folder /xyz where all the files reside
now all the files starting with
01 - will be appended for one below other to form a batch batch01
then all the files starting with
02 - will be appended for one below other to form a batch batch02
then
03 - will be appended for one below other to form a batch batch03
then
04 - will be appended for one below other to form a batch batch04
..

and so on, now this is taking a lot of time while processing.
How can i improve the performance say include some type of parallel processing
to minimize the time.
Please Advice
# 3  
Please find the script below

Code:
name00=ABC00`date +"%y%m%d%H"`
for i in 0[0]*.[tT][xX][tT]
do
cat ${i} >> ${name00}            
done

name01=ABC01`date +"%y%m%d%H"`
for i in 0[1]*.[tT][xX][tT]
do
cat ${i} >> ${name01}            
done
 

name02=ABC02`date +"%y%m%d%H"`
for i in 0[2]*.[tT][xX][tT]
do
cat ${i} >> ${name02}            
done

and so on

Last edited by bakunin; 09-04-2009 at 09:02 AM.. Reason: generously provided code-tags to the needy - please spend them yourself from now on
# 4  
Your code would probably benefit from the usage of "find" and the elimination of repetitive tasks like expanding the date over and over again:

Code:
chDate="$(date +"%y%m%d%H")"
typeset -Z2 iCounter=0

while [ $iCounter -le 99 ] ; do
     find /your/directory -name "${iCounter}*[tT][xX][tT]" -print > "ABC${iCounter}${chDate}"
     (( iCounter += 1 ))
done

I put an arbitrary end at 99 for demonstration purposes, adapt the script to what you really need. If this is not fast enough try backgrounding the "find"s by adding a " &" at the end of the line starting with "find".

I hope this helps.

bakunin
# 5  
First, what type of file system is your /xyz directory? What is the underlying hardware? How busy is it when you're running your script?

If your hardware is already maxed out, it's already maxed out and parallel processing won't help. In fact, it could even slow it down further and you'll likely get more disk contention.
# 6  
Code:
name00=ABC00`date +"%Y%m%d%h"`
cat 00*.[tT][xX][tT] > "$name00"

name01=ABC01`date +"%Y%m%d%h"`
cat 01*.[tT][xX][tT] > "$name01"

name02=ABC02`date +"%Y%m%d%h"`
cat 02*.[tT][xX][tT] > "$name02"

Or:

Code:
for n in 01 02 03 04 ...
do
  name=ABC$n`date +"%Y%m%d%h"`
  cat "$n"*.[tT][xX][tT] > "$name"
done

# 7  
Your code would probably benefit from the usage of "find" and the elimination of repetitive tasks like expanding the date over and over again:

Code:
chDate="$(date +"%y%m%d%H")"
typeset -Z2 iCounter=0

while [ $iCounter -le 99 ] ; do
     find /your/directory -name "${iCounter}*.[tT][xX][tT]" -print > "ABC${iCounter}${chDate}"
     (( iCounter += 1 ))
done

I put an arbitrary end at 99 for demonstration purposes, adapt the script to what you really need. If this is not fast enough try backgrounding the "find"s by adding a " &" at the end of the line starting with "find".

I hope this helps.

bakunin
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Parallel processing
garethsays
I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished?... Shell Programming and Scripting
1
Shell Programming and Scripting
How to make parallel processing rather than serial processing ??
Samb95
Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log...... Shell Programming and Scripting
0
Shell Programming and Scripting
Need Help With Parallel Processing
imnewtothis23
Hi I am looking for some kind of feature in unix that will help me write a script that can invoke multiple processes in parallel. And make sure that the multiple parallel processes complete successfully before I proceed to the next step. Someone suggested something called timespid or...... Shell Programming and Scripting
6
Shell Programming and Scripting
parallel processing
audippa
Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M... Shell Programming and Scripting
3
Shell Programming and Scripting
How to do parallel processing??
zing
Hi All, I am working on solaris 8 sparc machine with 2 cpu. I am trying to run my application which generates files. I run multiple instance of the application, but the results don't seem to show as if it were runing parallely. When i run the application once it takes 12 secs to generate a...... UNIX for Dummies Questions & Answers
1
UNIX for Dummies Questions & Answers