Performance issue while using find command


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Performance issue while using find command
# 1  
Old 10-19-2012
Performance issue while using find command

Hi,

I have created a shell script for Server Log Automation Process. I have used

find xargs grep command to search the string.

for Example,
Code:
 find -name | xargs grep "816995225" > test.txt

.


Here my problem is,

We have lot of records and we want to grep the string 4-5 times. so, it takes around 20 mins to finish the process.


so, how to search the string fastly using find xargs command

Please Suggest me.


Regards,
Nanthagopal A
# 2  
Old 10-19-2012
What you posted has some problems. Since the script works correctly but slowly could you please post exactly what code you have.

Also, are you repeating the grep over the same files 3-4 times?

Code:
find /path/to/files -name '*.log'  -exec grep 'pattern'

is another way to grep files. And your xargs approach is another way as well.
# 3  
Old 10-19-2012
Aside from posting the exact code that you are using (since what you posted isn't workable), as jim requested, also identify the operating system(s) and shell(s) on which the code will run (be specific, include version numbers).

find functionality beyond the standardized subset varies a great deal between implementations.

Regards,
Alister
# 4  
Old 10-27-2012
Quote:
Originally Posted by jim mcnamara
What you posted has some problems. Since the script works correctly but slowly could you please post exactly what code you have.

Also, are you repeating the grep over the same files 3-4 times?

Code:
find /path/to/files -name '*.log'  -exec grep 'pattern'

is another way to grep files. And your xargs approach is another way as well.


Hi,

I have 4 Sub-Process. for each process i want to grep some string and finally print each sub-process results.

I have used find | xargs grep comand for grepping, the code as follows

Code:
  find . -type f ! -name "*.bz2" ! -name "*~" -name "catalina.*" | xargs grep  xxx | grep "xx\|xx\|xx\|xx" | grep "status"

It takes 7-8 minutes for grepping the entire records in the path.


Note:

For each sub-process, i have to grep the full records. so, approximately 7*4 = 28 mins to finish the entire process.

So, how to improve the performance.

Please Suggest me.

Regards,
Nanthagopal A
# 5  
Old 10-27-2012
Try this

Code:
find . -type f ! -name "*.bz2" ! -name "*~" -name "catalina.*" | xargs awk '/xxx/&&/xx\|xx\|xx\|xx/&&/status/'

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Increase the performance of find command.

I'm trying to exclude 'BACKUP', 'STORE', 'LOGGER' folders while searching for all files under a directory "/tmp/moht" Once a file is found I wish to display the filename, the size of the file & the cksum value. Below is the command, I'm using: /opt/freeware/bin/find /tmp/moht -type d -name... (6 Replies)
Discussion started by: mohtashims
6 Replies

2. Shell Programming and Scripting

Performance Issue for a file search command

Hi All, This query is regarding performance improvement of a command. I have a list of IDs in a file (say file1 with single ID column) and file2 has the data rows. I need to get the IDs from file1 and search in file2, matching rows from file2 should be written to a file3. For this... (4 Replies)
Discussion started by: Tanu
4 Replies

3. Shell Programming and Scripting

Find command issue

Hi Guys, I have a file called error.logs. am just trying to display the content in the file which was modified last 1 day. I tried below command but it doesnt give the proper output. find /u/text/vinoth/bin "error.logs" -mtime -1 -exec cat {} \; >> mail.txt Any help is much... (21 Replies)
Discussion started by: Vinoth Kumar G
21 Replies

4. Shell Programming and Scripting

Issue in Find and mv command

Hi I am using the below code to find mv the files. Files are moving to the Target location as expected but find is displaying some errors like below. find ./ -name "Archive*" -mtime +300 -exec mv {} /mnt/X/ARC/ \; find: `./Archive_09-30-12': No such file or directory find:... (6 Replies)
Discussion started by: rakeshkumar
6 Replies

5. Shell Programming and Scripting

Find command issue

Guys, Here is my requirement.. Sample.cfg file="*log.gz *txt.gz" sample.sh #!/bin/sh . $HOME/Sample.cfg find . -name "$file" -mtime +20 -exec ls -la {} \; Its not finding the given *log.gz and txt.gz files. Could anyone please help me? (8 Replies)
Discussion started by: AraR87
8 Replies

6. HP-UX

Performance issue with 'grep' command for huge file size

I have 2 files; one file (say, details.txt) contains the details of employees and another file (say, emp.txt) has some selected employee names. I am extracting employee details from details.txt by using emp.txt and the corresponding code is: while read line do emp_name=`echo $line` grep -e... (7 Replies)
Discussion started by: arb_1984
7 Replies

7. Linux

find command issue

Hi, I am not root user. I am trying to find the file which has contains pattern "fvsfile" in root directory. If i run the find cmd then i got permission denied and all the files are listed include pattern files. i cant get file name yet find . print | xargs grep -i "fvsfile" I want... (2 Replies)
Discussion started by: Mani_apr08
2 Replies

8. Shell Programming and Scripting

Issue with Find Command

Hi All, I'm a bit new to Linux environment, moderately okay when it comes to Unix AIX. I'm facing an issue while trying to run a simple find command: $ for file in `find . -name *.*` > do > ls $file > done This is throwing the following error: Strangely, a few minutes... (4 Replies)
Discussion started by: adi_2_chaos
4 Replies

9. UNIX for Dummies Questions & Answers

Find command issue

I am currently using below command to get the 1st three characters of a file(PDM). Issue is, when i use find command in root dir, it finds all the files in sub dir also. How to limit the find command search to a given path only(ie: say only find file in apps/cmplus/datamigration/data path... (3 Replies)
Discussion started by: abhi_n123
3 Replies

10. Shell Programming and Scripting

An issue with find command.

Hi all, I have a shell script(ksh) which has the code as follows. ------------------ cd $mydir for i in `find ./ -type f -mtime +$k` do echo $i done ----------------------- And in $mydir , i have some files which have space in theie names like "Case att15". The out put of the... (6 Replies)
Discussion started by: rajugp1
6 Replies
Login or Register to Ask a Question