Shell Programming and Scripting

After my final test, I just found a problem again.

I started the scanner to search for about 5 phrases and write the files into my result file, that works fine so far. But after about an hour and I dont know how many files, the process is no longer listed with "top", but it is still not finished.

Now I had a look in WHM and there I see the process python running with 40% CPU usage, getting smaller every 5 seconds. In my shell with top the cpu´s are not showing that load.

Any idea where that problem comes from? I just searched, but I cant find where a limit could be that stops python.

put something like this in your code

Code:

            if size <= 2048000:
                # open("logger.txt","a").write("doing "+os.path.join(r,files)+"\n")
                # print "doing ..." + os.path.join(r,files)
                o=open(outfile,"a")
                .......

this is create a logger.txt file. you can tail -f this file to check progress. OR you can just print to stdout the progress. Using top only shows you partial information. best is to use ps -ef |grep "process".
If its always printed, then i believe you really have A LOT of files to process.

I already added such a line that is always telling me what the script is doing, but I think there are really a lot of files.

There are about 1000 public_html directories and every folder has about 100-200 files.

So don´t know if it is a problem to work 500 000 files or even more.

well, if you really have THAT much files, there's really no choice right? between using find+xargs and one using Python, you can time both and use the one more efficient. I guess that's already a bonus if you find one that is fast enough. (or you can wait for some other solutions to come by)

From what you explained me python is a very nice way to write such files. I just need to ensure that the scanner is not stopping its work after 1 hour because that happened.

It will wait till the scanner now stops with the added print so that I could find out where it happened.