UNIX for Dummies Questions & Answers

Hi all,

I have one script. Its job is to get 1 file from dirA and 1 file from dirB as parameters for another python script.

However, there are a lot of files in both dir A and B, so running this scripts really takes a lot of time. So I wonder if there are any ways that I can do this faster, by running it concurrently, maybe?

Much thanks in advance.
H.P

currently how you are taking the file from dirA and dirB

please post the script

Hi,

It's something like this:

Code:

for F in `cd A; echo a*`
do
    for G in `cd B; echo b*`
    do
        ./scripts/XYZ.py A/$F B/$G | gzip > out.$F.$G.gz
    done
done

The problem is that there are a lot files in A and B, so this takes very long time.

you can do it by running the python script in backgroud
Background Process - Foreground Process
or use nohup (en.wikipedia.org/wiki/Nohup)

Hi,

Thanks for your reply

I have tried nohup, it works nicely, yet still needs a lot of time. And I cannot continue doing things without the result from this step.

I have been waiting from yesterday morning until now. Just now with some simple calculations, I guess I need at least 77 days to get its done T___T. That is why I am here.

Hi,

nohup is there to make sure that there is no chance any of these will be stopped i.e. killing the shell. You don't want it running in the background you want it running quickly in parallel. But not a bad suggestion at all.


Your code , if that is what you use needs a lot of optimization.


For example if you have 1000 a* files your

Code:

for G in `cd B; echo b*`

seems to run forever.... what is the problem there? the 'cd' command....

change it to something like

Code:

VAR1=`cd A; echo a*`
for F in $VAR1

for a start, that should make it run a lot faster.


Also , depending on the optimization we get from the first debug (meainng you change your code, run it again and then we see how much time these changes have earned you) we use other methods.... like

Code:

./scripts/XYZ.py A/$F B/$G | gzip > out.$F.$G.gz &

Hi,

Thank you very much I will try it out tomorrow

And I also found something interesting: the task spooler command TS

Code:

http://vicerveza.homeunix.net/~viric/soft/ts/man_ts.html