Shell Programming and Scripting

That code works now, but it has the same problem as the I started with. It starts working and after a few minutes the used cpu and mem goes down to nearly 0 and thats it.

I am not a Pro, so are there are limits? I just tried the scanner with a few thousand files and a few arguments, but it seems that it has chance to finish before something gets a timeout or something else.

Any idea on that or any other idea of working around the content search?

I see ..., that's not a code problem though and there are no certain limits as far as find/xargs is concerned, I'd rather suspect a hardware / OS issue.
Try to run the same code I posted on a different machine, and see how it behaves. This way at least, you have some idea of what's going on.

Just did some more testing. The problem seems to be find . I think there are too many files and subfolders. I just tried to limit the folder depth and it worked, for a smaller amount of files.

I have no idea where the limit could be. I Use CentOS5 on a machine with 4 cores and 4 gb ram, so if anyone has an idea please let me know.

Otherwise the solution would take a lot more of time.

if you have Python on Centos, here's an alternative

Code:

#!/usr/bin/env python
import os
outfile = os.path.join("/home","filepath","public_html","path","scans","scan_result.php")
for r,d,f in os.walk("/home"):
    if "public_html" in r:
        for files in f:
            size=os.path.getsize(os.path.join(r,files))
            if size <= 2048000:
                o=open(outfile,"a")
                for line in open(os.path.join(r,files)):
                    if "phrase1" in line or "phrase2" in line:
                        o.write(line)
                o.close()

@ghostdog74

Thanks for that piece of beautiful code.

I just modified it to save the path of the file with the content of the line und the load is more than okay. The maximum I have seen while testing is 70% of one CPU and the load is just around 1.5 .

I always thought that bash is the only thing that could work with low resources on my server.

Just one question left, the code you posted searches all public_html folders within the home directory, is there a limit of how many sub directories are scanned, or is it following the hierarchy until the end?

not true. for example, using too much pipes, also, the logic matters as well.

please read the documentation of the os.walk() method here. you can pass arguments(eg topdown) to os.walk() to limit your search.

Thanks again for the answer, I just want to search all sub directories, so it should work as you already wrote in code example.

I will now start my final test with a few more arguments to find.

Thanks again.