UNIX for Dummies Questions & Answers

Thanks Bakunin.

I ran the find pdf on its own and saved the output to a file. I have all the PDFs paths/filenames that meet my criteria.

When I was looking at grep -f, I could only find examples of reading pattern from file and searching in another file. How can I make it search the patterns in an entire folder?

So far all I find is similar to: grep -f file1 file2
Apologies in advance. I am very new to this.

Zig

Suppose you have a file file with the patterns you search for (the names of the .pdf files). How about something along the lines of:

Code:

 find /some/path -type f -name "*html" -exec grep -f file {} \;

I hope this helps.

bakunin

I tried the above and the grep command doesn't seem to work properly. I added a pdf to the pattern file that is linked from other pages but it wasn't found by the grep.
I am starting to think that the format of the patterns in the file matter. Is it possible it fails because of the format of the filenames in the pattern file? ie: abc-abc.pdf

So based on the above, I made a script that does the work:

Code:

#!/bin/ksh

for p in $(cat "path-to-patterns-file); do
    echo $p;
    find ./ -type f -name "*.html" -exec grep -l $p {} \; 2>/dev/null 1>>/path-ro-results;
done

I tested the above and it seems to work accurately. The PDF names are grabbed from the patterns file and the loop is working as expected as well. However, I calculated the amount of time it takes to complete a search for 1 file and it was 45 seconds. If I were to use 2000 pdfs file in the pattern, it would take 24 hours to complete all the searches.

I was wondering, is there a way to further optimize the script and speed it up. I was thinking for example if there is a way to stop the search once a single match is found and jump to the next pattern. I am only interested in PDFs with no results.

Thanks,
Zig

It's slow because you're running grep dozens to thousands of times for things it only needs to run a few times for. That's also a useless use of cat.

Your use of -l was a good idea, since it should stop at the first match in the file instead of hunting for all matches in a file.

The matching may not work because they're regular expressions, not filenames. Use -F to force grep to interpret the patterns as fixed strings instead.

Beware that blank lines in the file will mess it up. Everything matches against a blank expression.

Also, use xargs to run grep once for many files, which should help a lot.

Code:

find /some/path -type f -name "*.html"| xargs grep -l -F -f patternfile

For some reason the 1 line command doesn't work. This is what I have:

Code:

#!/bin/ksh

find /somepath -type f -name "*.html"| xargs grep -l -F -f /path-to-patterns-file {} \; 2>/dev/null 1>>/path-to-results;

The above doesn't return any results which is wrong. I am not sure what's wrong or if I have some syntax error somewhere.

Moderator's Comments:

edit by bakunin: please use [CODE] .. [/CODE]-tags when posting code or terminal output. Thank you

I figured out why the command was failing. The box has two grep utilities and the default one didn't support -f option. Once I used the other one the script worked as expected. Now I just need to fix the output so I know which string is being searched for and which file they were found in.

I want to thank everyone who helped me with this script. I appreciate it.
Zig