Script performance issue


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script performance issue
# 1  
Old 10-08-2010
Script performance issue

hi

i have written a shell script which comapare a text file data with files within number of different directories.

example.
Text File: i have a file /u02/abc.txt which have almost 20000 file names
Directories: i have a path /u03 which have some subdirectories like a,b,c which have almost 40000 files.
i.e.
cd /u03
ls
a b c

My script take a file name from the text file and look this file in different subdirectories of a /u03 path.if found echo this file name in ExistFiles.txt file and if not found echo this file name in NotExistFiles.txt

but the problem is it took too much time. Is there any perfomance increasing suggestion. (note: i need to find each file in all a b c directories)

following is the script.
Code:
#!/bin/bash
#####################################
# Created by : Malik Shahid Hussain #
# Created on : 07-OCT-2010 #
# Created for: To Compare file-Dir #
#####################################
AtheerHD=/export/home/atheerbi/FILESRECORDS/
rm -f ${AtheerHD}*ExistFiles.txt
############################################
# Define Variable to take User Input #
############################################
read -p "Enter the full path of directory:" dir
read -p "Enter the text file name with full path:" txt
############################################
# Check For Inputs #
############################################
if [[ "$dir" = "" || "$txt" = "" ]];then
  echo -e '\E[37;41m'"\033[1mSorry: Need Dir Path and Filename \033[0m";
  exit -1
fi
############################################
# Check Temp file #
############################################
if [[ ! -e ${AtheerHD}tmpr.tx ]];then
  touch -f ${AtheerHD}tmpr.tx
fi
############################################
# Check For File Status #
############################################
while read line
do
  find $dir -name "$line" |head -1 >${AtheerHD}tmpr.tx 
  grep "$line" ${AtheerHD}tmpr.tx 2>dev>null
  if [[ $? -eq 0 ]];then
    echo $line >> ${AtheerHD}ExistFiles.txt
  else
    echo $line >> ${AtheerHD}NotExistFiles.txt
  fi
done < $txt
echo -e '\E[37;41m'"\033[1m For Exist Check==== ${AtheerHD}ExistFiles.txt \033[0m";
echo -e '\E[37;41m'"\033[1m For Not Exist Check==== ${AtheerHD}NotExistFiles.txt \033[0m";


Last edited by Franklin52; 10-08-2010 at 04:18 AM.. Reason: Removed size tags
# 2  
Old 10-08-2010
I would generate the entire list of files (find or shell glob) and than I would merge it with the file content - one operation to get all the names I'm interested in.
# 3  
Old 10-08-2010
Code:
sort $txt >txt.sorted
find /u03 -type f | sort >biglist.sorted
comm -12 txt.sorted biglist.sorted

If you have only these /u03/a|b|c directory that have a lot of files in it, and if the list of file you want to check is not big, you can then go for a
Code:
while read name
do
ls -d /u03/*/$name 2>/dev/null
done <$txt

You can also try to extract the list of the directory (instead of the file), and then check whether the files exists in it.
(I haven't tested it yet , it so might contains bugs)

Code:
while read a
do
eval find /u03 -type d -exec ls -d {}/$a\; 2>/dev/null 
done <$txt

Do not use the * in a directory that hold too many files , you will get the "argument list too long" error because the number of file is to high for the wildcard to be expanded.
These 2 Users Gave Thanks to ctsgnb For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Performance Issue - Shell Script

Hi, I am beginner in shell scripting. I have written a script to parse file(s) having large number of lines each having multiple comma separated strings. But it seems like script is very slow. It took more than 30mins to parse a file with size 120MB (523564 lines), below is the script code ... (4 Replies)
Discussion started by: imrandec85
4 Replies

2. Shell Programming and Scripting

Performance issue in shell script

Hi All, I am facing performance issue while rinning the LINUX shell script. I have file1 and file 2. File one is the source file and file 2 is lookup file. Need to replace if the pattern is matching in file1 with file2. The order of lookup file is important as if any match then exit... (8 Replies)
Discussion started by: ureddy
8 Replies

3. UNIX for Dummies Questions & Answers

awk script performance issue

Hello All, I have the below excerpt of code in my shell script and it taking long time to complete, though it prints the output quickly. Is there a way to make it come out once it finds the first instance as the file size of 4.7 GB it could be going through all lines of the data file to find for... (3 Replies)
Discussion started by: Ariean
3 Replies

4. AIX

Performance issue

Hi We have an AIX5.3 server with application which is written in C. We are facing server (lpar) hangs intermediately. If we open new telnet window prompts for user and takes hell of a time to authenticate, not only that if we run ps -aef then also it takes lot of time. surprisingly there is no... (2 Replies)
Discussion started by: powerAIX
2 Replies

5. AIX

Performance issue

Hi, We have 2 lpars on p6 blade. One of the lpar is having 3 core cpu with 5gb memory running sybase as database. An EOD process takes 25 min. to complete. Now we have an lpar on P7 server with entitled cpu capacity of 2 with 16 Gb memory and sybase as database. The EOD process which takes... (17 Replies)
Discussion started by: vjm
17 Replies

6. UNIX for Advanced & Expert Users

FTP-Shell Script-Performance issue

Hello All, Request any one of Unix/Linux masters to clarify on the below. How far it is feasible to open a new ftp connection for transferring each file when there are multiple files to be sent. I have developed shell script to send all files at single stretch but some how it doesnt suit to... (3 Replies)
Discussion started by: RSC1985
3 Replies

7. Shell Programming and Scripting

Performance issue with ftp script.

Hi All, I have written a script to FTP files from local server to remote server. When i try it for few number of files the scripts runs successfully. But the same script when i run for 200-300 files it gives me performanace issue by aborting the connection. Please help me out to improve the... (7 Replies)
Discussion started by: Shiv@jad
7 Replies

8. Shell Programming and Scripting

Performance issue with awk script.

Hi, The below awk script is taking about 1 hour to fetch just 11 records(columns). There are about 48000 records. The script file name is take_first_uniq.sh #!/bin/ksh if then while read line do first=`echo $line | awk -F"|" '{print $1$2$3}'` while read line2 do... (4 Replies)
Discussion started by: RRVARMA
4 Replies

9. UNIX for Advanced & Expert Users

performance issue

Hi, on a linux server I have the following : vmstat 2 10 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 4 0 675236 39836 206060 1617660 3 3 3 6 8 7 1 1 ... (1 Reply)
Discussion started by: big123456
1 Replies

10. UNIX for Advanced & Expert Users

Performance issue

Hello all, I just stuck up in an uncertain situation related to network performance... I am trying to access one of my remote client unix machine from a distant location.. The client machine is Ultra-5_10 , with SunOS 5.5.1 The ndd result ( hme1 )shows that the machine is hooked to a... (5 Replies)
Discussion started by: shibz
5 Replies
Login or Register to Ask a Question