Script execution is very slow when trying to find all files and their owners on HP-UX box


 
Thread Tools Search this Thread
Operating Systems HP-UX Script execution is very slow when trying to find all files and their owners on HP-UX box
# 1  
Old 08-14-2015
Script execution is very slow when trying to find all files and their owners on HP-UX box

Hi,

I have a HP-UX server were I need to list all the files in the entire file system, their directory path, last modified date, owner and group. I do not need to search the file contents. I created the script given below and I am excluding directories and files of type tmp, temp and log. The issue I am facing is the script runs very fast for some minutes but slows down and adds only 1 or 2 records per second. Is that normal or can this be expedited by some script tweeking? My server is supposed to have more than 500000 files and with this speed, script execution is taking too much time. I believe since I am not searching the file contents and only fetching the file name and some of its attributes, it should be very fast.

Code:
root_dir="/"

for dir in $(ls -d $root_dir*)
do

	for file in $(find $dir -type d \( -name '*tmp*' -o -name '*temp*' -o -name '*log*' \) -prune  -o -type f \( ! -name '*.log*' ! -name '*.LOG*' ! -name '*temp*' \) -print)
		 do
			last_modified_date=`ls -l $file | awk '{print $6, "", $7, "", $8}'`
			fileowner=`ls -l $file | awk '{print $3}'`
			filegroupowner=`ls -l $file | awk '{print $4}'`
			echo $(hostname),$(basename $file),$(dirname $file),$last_modified_date,$fileowner,$filegroupowner
		 done
done

I tried removing the print option but still the script execution is slow.

Last edited by Adyan Faruqi; 08-14-2015 at 04:51 PM.. Reason: spell check
# 2  
Old 08-14-2015
What you are doing is known as "thrashing the disk", the sort of thing you're only supposed to do late at night when interactive users aren't around to experience the lagging. (i.e. how the locate database is updated).

Walking every single directory to call stat on every single individual file is never going to be fast, period. Your implementation is far from efficient but I don't think there's much improvement to be had since the wall you're hitting is your disk speed.

Last edited by Corona688; 08-14-2015 at 05:41 PM..
# 3  
Old 08-14-2015
In addition to the many stat()s you do many fork()s.
Each call to external programs like ls, awk, hostname, basename, dirname is a fork().
The following has the number of fork's reduced:
Code:
# constant, evaluate once
hostname=$(hostname)
root_dir="/"

for dir in $root_dir*
do
  find "$dir" -type d \( -name '*tmp*' -o -name '*temp*' -o -name '*log*' \) -prune  -o -type f \( ! -name '*.[Ll][Oo][Gg]*' ! -name '*temp*' \) -exec ls -l {} + |
  while read permissions links owner groupowner size d1 d2 d3 filename
  do
    echo "$hostname,${filename##*/},${filename%/*},$d1 $d2 $d3,$owner,$groupowner"
  done
done

This User Gave Thanks to MadeInGermany For This Post:
# 4  
Old 08-14-2015
And, as with your previous thread (Error while script execution - 0403-029 there is not enough memory available now), there is no need for the outer loop; let find do its job:
Code:
# constant, evaluate once
hostname=$(hostname)
root_dir="/"
find "$root_dir" -type d \( -name '*tmp*' -o -name '*temp*' -o -name '*log*' \) \
    -prune  -o -type f \( ! -name '*.[Ll][Oo][Gg]*' ! -name '*temp*' \) \
    -exec ls -l {} + |
while read permissions links owner groupowner size d1 d2 d3 filename
do  echo "$hostname,${filename##*/},${filename%/*},$d1 $d2 $d3,$owner,$groupowner"
done

This User Gave Thanks to Don Cragun For This Post:
# 5  
Old 08-15-2015
Thanks MadeInGermany/Don, I will try to optimize the script as per the suggestions provided.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Improve script - slow process with big files

Gents, Please can u help me to improve this script to be more faster, it works perfectly but for big files take a lot time to end the job.. I see the problem is in the step (while) and in this part the script takes a lot time.. Please if you can find a best way to do will be great. ... (13 Replies)
Discussion started by: jiam912
13 Replies

2. UNIX for Dummies Questions & Answers

Awk/script to list the owners of files

I have to list the files in a directory and along with that, list the owner of each of those files. Can someone please help me with a way to get this info please? Gayathri (2 Replies)
Discussion started by: ggayathri
2 Replies

3. Shell Programming and Scripting

Find execution time of script

i am using bash START=$(date +%s) END=$(date +%s) DIFF=$(echo "$END - $START" ) this code is not working (14 Replies)
Discussion started by: rafa_fed2
14 Replies

4. Shell Programming and Scripting

Find only files/directories with different permissions/owners

My git post-update has the following lines in it to make sure the permissions are set right: find /usr/local/apache/htdocs -type d -print0 | xargs -0 chmod 755 find /usr/local/apache/htdocs -type f -print0 | xargs -0 chmod 644 chown -R apache:apache /usr/local/apache/htdocsThe only problem is... (5 Replies)
Discussion started by: dheian
5 Replies

5. Shell Programming and Scripting

Slow Script Execution.

Basically my requirement is to know the total number of free anonymous ports. anonymous port range is 32768- 65535. i wrote a script for that ********************************************** for i in {32768..65535} do netstat -an | grep $i > /dev/null if ... (21 Replies)
Discussion started by: mohtashims
21 Replies

6. Shell Programming and Scripting

slow command execution?

Dear World, I just wrote a script, which puzzled me somewhat. The siginficant code was: for file in `ls splits*`; # splits* came from a split command executed earlier do tail -$SomeNumber $file | cut -d" " -f6 > $file; done; The interesting thing is this: A few of the $files were... (2 Replies)
Discussion started by: BandGap
2 Replies

7. UNIX for Advanced & Expert Users

why will a unix box become slow... ???

Hi ! Can somebody tell me why -or- when will a unix box will become slow in processing .... where to go and chk the stats... for this ??? Pls help.. Thanks in advance... (2 Replies)
Discussion started by: dashok.83
2 Replies

8. Shell Programming and Scripting

Command/script to find size of Unix Box ?

Please could anyone provide me the Command/script to find the size and usage of Unix box ASAP ? (6 Replies)
Discussion started by: sakthifire
6 Replies

9. UNIX for Advanced & Expert Users

Access files from prev box after SSH to another box

i'm not much of an advanced unix programmer but I'm trying to write a script to access files on box1 after ssh from box 1 to box2. when ssh is invoked in the script i'm getting logged into box2 and losing complete touch with box1 which is normal i guess. but my main aim with my script is when i... (3 Replies)
Discussion started by: pharos467
3 Replies

10. UNIX for Dummies Questions & Answers

Painfully Slow SSH login on Solaris box

Running open solaris on a e420 that I recently picked up. Having issues sshing to it from either of my Linux boxes as its very slow to login (from the solaris box to the linux box it connects just fine. Here is the output of ssh -vvv. I have hightlighted where it seems the slowdown is. Does... (0 Replies)
Discussion started by: creedog
0 Replies
Login or Register to Ask a Question