Command very Slow


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Command very Slow
# 1  
Old 05-18-2015
Tools Command very Slow

Hi,

The below command takes more than 2hrs.
Code:
result=$(find . -name "star_st*" -exec head -1 {} \; | grep "1175 876330")

Is there a way I can tweak the command to get better performance / quicker results.

Suggestions would be appreciated.
# 2  
Old 05-18-2015
run it against a smaller file system?

It's actually quite an efficient way of capturing any opening lines with that string.
# 3  
Old 05-18-2015
Try changing your script to:
Code:
result=$(find . -name "star_st*" -exec head -1 {} + | grep "1175 876330")

It should run a lot faster because it runs head for groups of files instead of having to run head for each file to be processed. The fork() and exec*() system calls needed to invoke head are expensive, relatively slow operations.

But, this assumes that a lot of files have names starting with star_st.

If . is on a remote filesystem, network issues could also have a significant impact.

Is there other heavy load on the system?

How big is the file hierarchy rooted in .?

How many files have names starting with star_st?
This User Gave Thanks to Don Cragun For This Post:
# 4  
Old 05-19-2015
Tools

Quote:
Originally Posted by Don Cragun
Try changing your script to:
Code:
result=$(find . -name "star_st*" -exec head -1 {} + | grep "1175 876330")

It should run a lot faster because it runs head for groups of files instead of having to run head for each file to be processed. The fork() and exec*() system calls needed to invoke head are expensive, relatively slow operations.

But, this assumes that a lot of files have names starting with star_st.

If . is on a remote filesystem, network issues could also have a significant impact.

Is there other heavy load on the system?

How big is the file hierarchy rooted in .?

How many files have names starting with star_st?
Hi, I have 99583 files in the directory with start_st
Code:
ls -l start_st* | wc -l
99583


I m on Linux
Code:
uname -a
Linux my_machine1 2.6.32-431.5.1.el6.x86_64 #1 SMP Fri Jan 10 14:46:43 EST 2014 x86_64 x86_64 x86_64 GNU/Linux

Is there other heavy load on the system? Answer: No

How big is the file hierarchy rooted in .? All the files are in the same directory.Answer: There are NO subdirectories.

If . is on a remote filesystem, network issues could also have a significant impact.: Answer: It is on the same Local File System.

Last edited by mohtashims; 05-19-2015 at 06:07 AM..
# 5  
Old 05-19-2015
It is depends on from which directory you are executing this find command and the number of files exists on your machine with the name of "star_st*".

if that . is / (root directory) and if your machine is configured with automount then it will try to mount all the auto mount filesystems and will perform find operation on them.

Last edited by subrkann; 05-19-2015 at 04:31 AM.. Reason: Unexpected code format.
# 6  
Old 05-19-2015
Quote:
How big is the file hierarchy rooted in . ? All the files are in the same directory.Answer: There are NO subdirectories.
Hello mohtashims,

Could you please try following command and let me know if this helps.
According to your statement alll files are in same directory without any sub directories so following may help you then.
Code:
awk '(NR==1 && $0 ~ /1175 876330/){print FILENAME}' /tmp/test13/star_st*

Above is just an example, so in spite of using /tmp/test13/star_st* use Actual_path/star_st* and let us know if this helps.

Thanks,
R. Singh
# 7  
Old 05-19-2015
here was a wrong suggestion :-)

Last edited by agent.kgb; 05-19-2015 at 05:11 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Why is SED so slow?

I have many files which contain about two million lines. Now I want to use sed to delete the 9th line and add a new line behind the 8th line. I use the command as follows: for((i=1;i<100;i++)); do echo $i; sed -i '9d' $i.dat; sed -i '8a this is a new line' $i.dat; done But it is... (3 Replies)
Discussion started by: wxuyec
3 Replies

2. Shell Programming and Scripting

Making a faster alternative to a slow awk command

Hi, I have a large number of input files with two columns of numbers. For example: 83 1453 99 3255 99 8482 99 7372 83 175 I only wish to retain lines where the numbers fullfil two requirements. E.g: =83 1000<=<=2000 To do this I use the following... (10 Replies)
Discussion started by: s052866
10 Replies

3. Solaris

Slow while running a command for the first time

I am facing a performance problem on a Solaris 10 Sparc V890 server, it is an old one I know. The first time we realized there is a problem with the server, is the time when ftp transfers are made. There were 4 other identical servers doing much better. Network drivers are checked and there... (3 Replies)
Discussion started by: royalliege
3 Replies

4. Shell Programming and Scripting

File processing is very slow with cut command

Dear All, I am using the following script to find and replace the date format in a file. The field18 in the file has the following format: "01/26/2010 11:55:14 GMT+04:00" which I want to convert into the following format "20100126115514" for this purpose I am using the following lines of codes:... (5 Replies)
Discussion started by: bilalghazi
5 Replies

5. Shell Programming and Scripting

slow command execution?

Dear World, I just wrote a script, which puzzled me somewhat. The siginficant code was: for file in `ls splits*`; # splits* came from a split command executed earlier do tail -$SomeNumber $file | cut -d" " -f6 > $file; done; The interesting thing is this: A few of the $files were... (2 Replies)
Discussion started by: BandGap
2 Replies

6. UNIX for Dummies Questions & Answers

scp is slow

All of the sudden scp got really slow ... from 2-3 seconds to 30 seconds. This happened for 5 hours, and then it went back to running fast. Why? If I use the -q qualifier which "Disables the progress meter" could this have any adverse effect? Thanks (1 Reply)
Discussion started by: tomstone_98
1 Replies

7. UNIX for Dummies Questions & Answers

Slow System

Hi, I have an SCO-Unix server running. There are some processes (unknown to me) which consume a lot of the system resources. This slows down the server dramatically. Is there a command or program that monitors what processes are using the cpu, disk, etc.. and tell me how excessive and how... (3 Replies)
Discussion started by: Hansaplast
3 Replies

8. Post Here to Contact Site Administrators and Moderators

Slow

The site has gone slow for quite some time... Can you do somethin abt it (2 Replies)
Discussion started by: DPAI
2 Replies
Login or Register to Ask a Question