Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Find files and seach limit with grep

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Find files and seach limit with grep
# 1  
Old 06-17-2012
Find files and seach limit with grep
Hi,

I'm testing nginx-cache-purge and notice that grep searches all file content, and since the cache key is in the second line I would like to limit grep searching.

This is the script I'm using:

github.com/perusio/nginx-cache-purge

The line I would like to change and limit grep search to the Second line is:

Code:
find $2 - maxdepth 1 - type d | xargs - P $max_ pa ralle l - n 1 gre p - Rl " $1" | so rt - u

Seaching in google I found out that the command head -n can help me with this, but I can't get it to work.

Can someone help me with this?

Thanks
# 2  
Old 06-17-2012
Code:
find $2 - maxdepth 1 - type d | xargs - P $max_parallel -n 1 head -2 | grep -Rl "$1" | sort -u

# 3  
Old 06-17-2012
Hi,

Thanks for you help, but I have errors when I execute the command:

Code:
root@local:~# find /var/cache/nginx/portal -maxdepth 1 -type d | xargs -P 16 -n 1 head -2 | grep -Rl "topic" | sort -u
head: error reading `/var/cache/nginx/portal': Is a directory
head: error reading `/var/cache/nginx/portal/1': Is a directory
head: error reading `/var/cache/nginx/portal/a': Is a directory
head: error reading `/var/cache/nginx/portal/4': Is a directory
head: head: error reading `/var/cache/nginx/portal/c': Is a directory

Without head I get:

Code:
root@local:~# find /var/cache/nginx/portal -maxdepth 1 -type d | xargs -P 16 -n 1 grep -Rl "topic" | sort -u
/var/cache/nginx/portal/8/d2/c7271e2a8985029a4194f1a43c256d28
/var/cache/nginx/portal/8/d6/76ecc4bbe957b6b3cdf4cacc4ffc1d68
/var/cache/nginx/portal/8/dc/fc67dd7e7a375d72f9fa41cfa4967dc8
/var/cache/nginx/portal/8/e7/ba67ada1fa86c5636f55303f5d5e6e78
/var/cache/nginx/portal/8/e7/e716048797249a17af20db3d320b9e78
/var/cache/nginx/portal/8/ea/0deae0c338ae8eb56f4e61877c7dfea8
/var/cache/nginx/portal/9/0b/de72ff84eac8eafe1d867ad6057d30b9
/var/cache/nginx/portal/9/11/6501b6f1dd8a62319d4edb86ebe02119
/var/cache/nginx/portal/9/12/fc1b1c0d72d1880e65de4f31a6f72129
/var/cache/nginx/portal/9/1d/74e03f08932a8643952a55843dc781d9
...... etc

The full command can be changed if it's easy to archive the same results.
# 4  
Old 06-17-2012
Code:
find "$2" -mindepth 2 -type f | xargs -P $max_parallel awk '/topic/ {print FILENAME} FNR==2 {nextfile}' | sort -u

Untested.

nextfile is not part of posix awk, but it's available on the most popular implementations.

AWK uses extended regular expression, instead of grep's basic flavor. You may need to tweak your regular expressions if they're more complicated than a simple word like "topic".

Since this find prints file, and since you almost certainly don't want to spawn awk once per file, I dropped -n 1 in xargs. Further, sometimes parallelizing i/o bound tasks can degrade performance if the parallel invocations are not backed by different disks. Something to keep in mind. You might actually improve performance by lowering the value of -P (if not by eliminating it altogether).

Regards,
Alister
Last edited by alister; 06-17-2012 at 12:58 PM..
# 5  
Old 06-17-2012
Hi,

This is for a forum (IPB) and the regular expressions I use are simple ... just want to find file that matches urls like:

/topic/123-topic-name/{everything found after}
/forum/123-forum-name/{everything found after}

so, I really need to match hostname.com/topic/123-topic-name/

I've tested in debian squeeze and works very well ... faster than before as I supposed:

Code:
real    0m0.116s
user    0m0.026s
sys     0m0.091s

versus

Code:
real    0m0.502s
user    0m0.164s
sys     0m0.242s

With AWK I need to escape punctuation and slashes with backslashes!

Thanks

---------- Post updated at 06:36 PM ---------- Previous update was at 05:08 PM ----------

Hi,

since the list of files generated by this commands must be removed, can this be simplified instead of using the scrit above?

i Just need to remove the files listed, if any, the fastest I can.

thanks for your great help.
# 6  
Old 06-17-2012
Assuming no whitespace in filenames:
Code:
find "$2" -mindepth 2 -type f | xargs awk '/topic/ {print FILENAME} FNR==2 {nextfile}' | xargs rm

Tweak xargs' -P option as required.

Regards,
Alister
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

General find /grep question: files with lines ending in r

I am trying to find files that have lines in them that end in an r. I have been able to locate files by using the following command: find . -type f -name "*RECORDS"| xargs grep -l r$ However, I now want to find files that don't end in r anywhere. That means that no sentences or lines in... (9 Replies)
Discussion started by: newbie2010
9 Replies

2. UNIX for Dummies Questions & Answers

Limit Number of files

hi guys how can i limit number of files in a disk or partition ? or how can i make a limit to inode number for a disk or partition ? ext3 or ext4 file system (1 Reply)
Discussion started by: mhs
1 Replies

3. Shell Programming and Scripting

grep by limit search

Hi I am in new in unix,can any one tell how to grep the data by limit. suppose I have below data:- is :mSecs is :mSecs is :mSecs is :mSecs requirement is how to grep the data which is having count greater than 1000 msecs only. thanks in adnavce. (2 Replies)
Discussion started by: abhigrkist
2 Replies

4. UNIX for Dummies Questions & Answers

Using grep to find files that don't contain a string

Hi all, I am still learning my way around unix commands and I have the following question. I have a website and I want to search for all the html pages that don't contain a certain js file. The file I am searching for is located under /topfolder/js/rules.js . So I assume in my grep search I... (5 Replies)
Discussion started by: SyphaX
5 Replies

5. Shell Programming and Scripting

Any limit on files

I am doing an ftp of around 1010 files and I am using mput for this. For some reason its only transferring 10 or 20 files and the rest are not getting transferred. There is some socket error in the log. is there an issue if we have more than 50 or so files for mput. here is the o/p in the log... (2 Replies)
Discussion started by: dsravan
2 Replies

6. UNIX for Dummies Questions & Answers

Shell to seach all .mov files in a drive

I want a shell script which does: -> search all .mov files in a drive (say e: drive) which are above 800 mb of size -> Delete the searched .mov files which are above 800mb Thanks, Manoj (4 Replies)
Discussion started by: manoj.b
4 Replies

7. Shell Programming and Scripting

Is it better to grep and pipe to awk, or to seach with awk itself

This may just be a lack of experience talking, but I always assumed that when possible it was better to use a commands built in abilities rather than to pipe to a bunch of commands. I wrote a (very simple) script a while back that was meant to pull out a certain error code, and report back what... (4 Replies)
Discussion started by: DeCoTwc
4 Replies

8. AIX

Help Using Grep command to Find the string in the files in the root directory

I want to serch for a string in all the files in the root directory. i want the search to be limited to only the files in the directory i.e the search to be done only in the files not in the sub directory. the approaches tried are 1)grep "pattern string" this command was serching the... (3 Replies)
Discussion started by: Subbu_Angeline
3 Replies

9. Shell Programming and Scripting

grep line length limit

Hi Friends, I am having a funny problem with grep. When I run grep 'expr' file.txt things work fine. But when try to get the line number using the -n option, i.e, grep -n 'expr' file.txt I get a message, "grep: 0652-226 Maximum line length of 2048 exceeded." If the line has more than... (3 Replies)
Discussion started by: hnhegde
3 Replies

10. UNIX for Dummies Questions & Answers

grep something to find which files, lines contain something

in aix, i use grep something * to get the files, lines that contain something. but i couldnt get the same result from a sun box. what is the equivalent? thanks (4 Replies)
Discussion started by: yls177
4 Replies
Login or Register to Ask a Question