Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Can anyone make this script run faster?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Can anyone make this script run faster?
# 1  
Old 07-29-2008
Can anyone make this script run faster?
One of our servers runs Solaris 8 and does not have "ls -lh" as a valid command. I wrote the following script to make the ls output easier to read and emulate "ls -lh" functionality. The script works, but it is slow when executed on a directory that contains a large number of files. Can anyone make this script run faster?

TIA

J

###################################

Code:
#!/usr/bin/ksh

function ConvertBytes
	{
	if (( Convert_QTY > 1073741824 ))
		then
			Convert_QTY=`echo "$Convert_QTY / 1073741824" | bc -l`
			FormattedConvert_QTY=$(printf "%4.0f" "$Convert_QTY")
			FormattedConvert_QTY="${FormattedConvert_QTY} G"
		else
			if (( Convert_QTY > 1048576 ))
				then
					Convert_QTY=`echo "$Convert_QTY / 1048576" | bc -l`
					FormattedConvert_QTY=$(printf "%4.0f" "$Convert_QTY")
					FormattedConvert_QTY="${FormattedConvert_QTY} M"
				else
					if (( Convert_QTY > 1024 ))
						then
							Convert_QTY=`echo "$Convert_QTY / 1024" | bc -l`
							FormattedConvert_QTY=$(printf "%4.0f" "$Convert_QTY")
							FormattedConvert_QTY="${FormattedConvert_QTY} K"
						else
							FormattedConvert_QTY=$(printf "%4.0f" "$Convert_QTY")
							FormattedConvert_QTY="${FormattedConvert_QTY} B"
					fi
			fi
	fi
	}

	# ****************************************************************
	# *
	# *	Write the directory information to a temporary file
	# *
	# **********************************************+*****************

	TempFile_NME=/tmp/llh.ksh.$$.tmp

	ls -lF $ls_SW $FileSpecification | awk '{ print $1 "+" $2 "+" $3 "+" $4 "+" $5 "+" $6 "+" $7 "+" $8 "+" $9 }'>$TempFile_NME

	# ****************************************************************
	# *
	# *	Loop through the temporary file and display formatted information
	# *
	# **********************************************+*****************

	echo "\nPermission Lnk Owner  Group   Size       Date                   File Name"
	echo   "---------- --- ------ ----- -------- ------------ ----------------------------------------"

	while read InputRecord
		do
			Permissions_DSC=$(echo "$InputRecord" | cut -d"+" -f1)

			if [ $Permissions_DSC != "total" ]
				then
					Links_QTY=$(echo "$InputRecord" | cut -d"+" -f2)
					Links_QTY=$(expr "$Links_QTY" ) # Convert to integer
					FormattedLinks_QTY=$(printf "%3.0f" "$Links_QTY")

					Owner_NME=$(echo "$InputRecord" | cut -d"+" -f3)
					FormattedOwner_NME=$(printf "%-8s" "$Owner_NME")

					Group_NME=$(echo "$InputRecord" | cut -d"+" -f4)
					FormattedGroup_NME=$(printf "%-5s" "$Group_NME")

					Convert_QTY=$(echo "$InputRecord" | cut -d"+" -f5)
					Convert_QTY=$(expr "$Convert_QTY" ) # Convert to integer
					ConvertBytes
					FormattedSize_QTY=$FormattedConvert_QTY

					Create_DTE=$(echo "$InputRecord" | cut -d"+" -f6-8)
					Create_DTE=`echo $Create_DTE | tr "+" " "`
					FormattedCreate_DTE=$(printf "%-12s" "$Create_DTE")

					File_NME=$(echo "$InputRecord" | cut -d"+" -f9)
					FormattedFile_NME=$(printf "%-40s" "$File_NME")

					echo "$Permissions_DSC $FormattedLinks_QTY $FormattedOwner_NME $FormattedGroup_NME $FormattedSize_QTY $FormattedCreate_DTE $FormattedFile_NME"
			fi
		done<$TempFile_NME

	# ****************************************************************
	# *
	# *	Calculate total space consumed by the specified directory
	# *
	# **********************************************+*****************

	echo   "                            --------"

	TotalSize_STR=`du -ks $FileSpecification | awk '{ print $1 }'`
	TotalSize_QTY=$(expr "$TotalSize_STR" ) # Convert to integer

	if (( TotalSize_QTY > 1073741824 ))
		then
			TotalSize_QTY=`echo "$TotalSize_QTY / 1073741824" | bc -l`
			FormattedTotalSize_QTY=$(printf "%4.1f" "$TotalSize_QTY")
			FormattedTotalSize_QTY="${FormattedTotalSize_QTY} T"
		else
			if (( TotalSize_QTY > 1048576 ))
				then
					TotalSize_QTY=`echo "$TotalSize_QTY / 1048576" | bc -l`
					FormattedTotalSize_QTY=$(printf "%4.0f" "$TotalSize_QTY")
					FormattedTotalSize_QTY="${FormattedTotalSize_QTY} G"
				else
					if (( TotalSize_QTY > 1024 ))
						then
							TotalSize_QTY=`echo "$TotalSize_QTY / 1024" | bc -l`
							FormattedTotalSize_QTY=$(printf "%4.0f" "$TotalSize_QTY")
							FormattedTotalSize_QTY="${FormattedTotalSize_QTY} M"
						else
							FormattedTotalSize_QTY=$(printf "%4.0f" "$TotalSize_QTY")
							FormattedTotalSize_QTY="${FormattedTotalSize_QTY} K"
					fi
			fi
	fi

	echo "                              $FormattedTotalSize_QTY (for $FileSpecification and its subdirectories)"

# 2  
Old 07-30-2008
I would re-write it using awk or perl to handle all of the formatting and conversion of numbers to human-readable format. Consider that every single set of `...` or $(...) means a fork to a subshell and you can imagine why this would run slowly. The du part would also take quite some time on a large directory.
# 3  
Old 07-30-2008
'Annihilannic' is right, all you need to do is to avoid forking out commands. BTW, the "du" part cannot be tuned 'cos it depends on how much stuff you have

Assuming we follow the same logic (no major change of code), how much we can tune in that ksh script. Here is my attempt on an old Sun box.
The original script run time is 8.927s and the below modified script is 2.845s, that's more than 3 times better in performance. Basically I avoid a lot print and cut in the loop
Code:
#!/usr/bin/ksh

function ConvertBytes
  {
  if (( Convert_QTY > 1073741824 ))
    then
      FormattedConvert_QTY=`echo "$Convert_QTY / 1073741824" | bc -l`
      FormattedConvert_UNIT="G"
    else
      if (( Convert_QTY > 1048576 ))
        then
          FormattedConvert_QTY=`echo "$Convert_QTY / 1048576" | bc -l`
          FormattedConvert_UNIT="M"
        else
          if (( Convert_QTY > 1024 ))
            then
              FormattedConvert_QTY=`echo "$Convert_QTY / 1024" | bc -l`
              FormattedConvert_UNIT="K"
            else
              FormattedConvert_QTY=$Convert_QTY
              FormattedConvert_UNIT="B"
          fi
      fi
  fi
}

# ****************************************************************
# *
# *  Write the directory information to a temporary file
# *
# **********************************************+*****************

TempFile_NME=/tmp/llh.ksh.$$.tmp
trap "rm -f $TempFile_NME" 0 1 2 9 15

ls -lF $ls_SW $FileSpecification | awk '{ print $1 "+" $2 "+" $3 "+" $4 "+" $5 "+" $6 "+" $7 "+" $8 "+" $9 }'>$TempFile_NME

# ****************************************************************
# *
# *  Loop through the temporary file and display formatted information
# *
# **********************************************+*****************

echo "\nPermission Lnk Owner  Group   Size       Date                   File Name"
echo   "---------- --- ------ ----- -------- ------------ ----------------------------------------"

while read InputRecord
  do
    # $1:permission, $2:lnk, ...
    set -- `echo "$InputRecord" | sed -e 's/\+/ /g'`

    if [ "$1" != "total" ]
      then

        Convert_QTY="$5"
        ConvertBytes
        Create_DTE="$6 $7 $8"

        # 'set --' cannot extract '*' for executable
        # shell will expand '*'
        File_NME=$(echo "$InputRecord" | cut -d"+" -f9)

        printf "%s %3.0f %-8s %-5s %4.0f %s %-12s %-40s\n" "$1" "$2" "$3" "$4" "$FormattedConvert_QTY" "$FormattedConvert_UNIT" "$Create_DTE" "$File_NME"
    fi
  done<$TempFile_NME

# ****************************************************************
# *
# *  Calculate total space consumed by the specified directory
# *
# **********************************************+*****************

echo   "                            --------"

TotalSize_STR=`du -ks $FileSpecification | awk '{ print $1 }'`
TotalSize_QTY=$(expr "$TotalSize_STR" ) # Convert to integer

if (( TotalSize_QTY > 1073741824 ))
  then
    TotalSize_QTY=`echo "$TotalSize_QTY / 1073741824" | bc -l`
    FormattedTotalSize_QTY=$(printf "%4.1f" "$TotalSize_QTY")
    FormattedTotalSize_QTY="${FormattedTotalSize_QTY} T"
  else
    if (( TotalSize_QTY > 1048576 ))
      then
        TotalSize_QTY=`echo "$TotalSize_QTY / 1048576" | bc`
        FormattedTotalSize_QTY=$(printf "%4d" "$TotalSize_QTY")
        FormattedTotalSize_QTY="${FormattedTotalSize_QTY} G"
      else
        if (( TotalSize_QTY > 1024 ))
          then
            TotalSize_QTY=`echo "$TotalSize_QTY / 1024" | bc`
            FormattedTotalSize_QTY=$(printf "%4d" "$TotalSize_QTY")
            FormattedTotalSize_QTY="${FormattedTotalSize_QTY} M"
          else
            FormattedTotalSize_QTY=$(printf "%4d" "$TotalSize_QTY")
            FormattedTotalSize_QTY="${FormattedTotalSize_QTY} K"
        fi
    fi
fi

echo "                              $FormattedTotalSize_QTY (for $FileSpecification and its subdirectories)"

The best bet is to write a awk script to do the formatting, run time on this is 0.104s, that's 85 times faster than the original script
Code:
#! /bin/sh

ls -lF | awk '
NR>1 {
  fac=1
  unit="B"
  if($5>1073741824) {
    fac=1073741824
    unit="G"
  } else {
    if ($5>1048576) {
      fac=1048576
      unit="M"
    } else {
      if ($5>1024) {
        fac=1024
        unit="K"
      }
    }
  }
  printf("%s %3s %-8s %-5s %4.0f %s %s %2s %-5s %-12s\n",$1,$2,$3,$4,$5/fac,unit,$6,$7,$8,$9)
}'

# 4  
Old 07-30-2008
Quote:
Originally Posted by Annihilannic
Consider that every single set of `...` or $(...) means a fork to a subshell and you can imagine why this would run slowly.
Okay. That explains it.

Quote:
Originally Posted by Annihilannic
I would re-write it using awk or perl to handle all of the formatting and conversion of numbers to human-readable format.
Thanks for the feedback. Unfortunately, I have zero skill in Perl and only a little bit of skill with awk.
Last edited by shew01; 07-30-2008 at 01:43 PM..
# 5  
Old 07-30-2008
Quote:
Originally Posted by chihung
The best bet is to write a awk script to do the formatting, run time on this is 0.104s, that's 85 times faster than the original script
Wow! Nice job. This is very usable now.

Quote:
Originally Posted by chihung
BTW, the "du" part cannot be tuned 'cos it depends on how much stuff you have
The only reason that I was using du was that I was accumulating size from each file as I processed it, and I eventually overflowed the capacity of the integer that I was using when I processed a directory with a large number of big files. I'd prefer to display just the total size of the files in the current directory (rather than go through the subdirectories like "du" does). Is this possible with awk?
Last edited by shew01; 07-30-2008 at 01:43 PM..
# 6  
Old 07-30-2008
Something like
Code:
find . -type f | xargs df -s

?
# 7  
Old 07-30-2008
Quote:
Originally Posted by shew01
I'd prefer to display just the total size of the files in the current directory (rather than go through the subdirectories like "du" does). Is this possible with awk?
This should do it.

Code:
ls -l | awk 'NF>=5 {tot+=$5}END {printf "%.f\n",tot}'

The NF>=5 is to skip the first line "total nnnn" which reports the number of 512 byte blocks used in the directory.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to make faster loop in multiple directories?

Hello, I am under Ubuntu 18.04 Bionic. I have one shell script run.sh (which is out of my topic) to run files under multiple directories and one file to control all processes running under those directories (control.sh). I set a cronjob task to check each of them with two minutes of intervals.... (3 Replies)
Discussion started by: baris35
3 Replies

2. Shell Programming and Scripting

How to make awk command faster?

I have the below command which is referring a large file and it is taking 3 hours to run. Can something be done to make this command faster. awk -F ',' '{OFS=","}{ if ($13 == "9999") print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12 }' ${NLAP_TEMP}/hist1.out|sort -T ${NLAP_TEMP} |uniq>... (13 Replies)
Discussion started by: Peu Mukherjee
13 Replies

3. Shell Programming and Scripting

Optimize shell script to run faster

data.file: contact { contact_name=royce-rolls modified_attributes=0 modified_host_attributes=0 modified_service_attributes=0 host_notification_period=24x7 service_notification_period=24x7 last_host_notification=0 last_service_notification=0 host_notifications_enabled=1... (8 Replies)
Discussion started by: SkySmart
8 Replies

4. Shell Programming and Scripting

awk changes to make it faster

I have script like below, who is picking number from one file and and searching in another file, and printing output. Bu is is very slow to be run on huge file.can we modify it with awk #! /bin/ksh while read line1 do echo "$line1" a=`echo $line1` if then echo "$num" cat file1|nawk... (6 Replies)
Discussion started by: mirwasim
6 Replies

5. Shell Programming and Scripting

Making script run faster

Can someone help me edit the below script to make it run faster? Shell: bash OS: Linux Red Hat The point of the script is to grab entire chunks of information that concerns the service "MEMORY_CHECK". For each chunk, the beginning starts with "service {", and ends with "}". I should... (15 Replies)
Discussion started by: SkySmart
15 Replies

6. Shell Programming and Scripting

Make script faster

Hi all, In bash scripting, I use to read files: cat $file | while read line; do ... doneHowever, it's a very slow way to read file line by line. E.g. In a file that has 3 columns, and less than 400 rows, like this: I run next script: cat $line | while read line; do ## Reads each... (10 Replies)
Discussion started by: AlbertGM
10 Replies

7. Shell Programming and Scripting

How to make copy work faster

I am trying to copy a folder which contains a list of C executables. It takes 2 mins for completion,where as the entire script takes only 3 more minutes for other process. Is there a way to copy the folder faster so that the performance of the script will improve? (2 Replies)
Discussion started by: prasperl
2 Replies

8. Red Hat

Re:How to make the linux pc faster

Hi, Can any one help me out in solving the problem i have a linux database server it is tooo slow that i am unable to open even the terminial is there any solution to get rid of this problem.How to make this server faster. Thanks & Regards Venky (0 Replies)
Discussion started by: venky_vemuri
0 Replies

9. Shell Programming and Scripting

awk help to make my work faster

hii everyone , i have a file in which i have line numbers.. file name is file1.txt aa bb cc "12" qw xx yy zz "23" we bb qw we "123249" jh here 12,23,123249. is the line number now according to this line numbers we have to print lines from other file named... (11 Replies)
Discussion started by: kumar_amit
11 Replies

10. Solaris

looking for different debugger for Solaris or to make sunstudio faster

im using the sunstudio but it is very slow , is there ant other GUI debugger for sun Solaris or at list some ways to make it faster ? im using to debug throw telnet connection connected to remote server thanks (0 Replies)
Discussion started by: umen
0 Replies
Login or Register to Ask a Question