Visit The New, Modern Unix Linux Community


Performance issue in Grepping large files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Performance issue in Grepping large files
# 1  
Performance issue in Grepping large files

I have around 300 files(*.rdf,*.fmb,*.pll,*.ctl,*.sh,*.sql,*.prog) which are of large size.
Around 8000 keywords(which will be in the file $keywordfile) needed to be searched inside those files.
If a keyword is found in a file..I have to insert the filename,extension,catagoery,keyword,occurrence to database.
I have implemented following code..but it is taking around 10-12 hours complete.
Could you please suggest how will i change it , so that it will be faster.
I am using Solaris .

Code:
 
/usr/xpg4/bin/find $tmpdir -type f -name "*.rdf" -o -name "*.fmb" -o -name "*.pll" -o -name "*.ctl" -o -name "*.sh" -o -name "*.sql" -o -name "*.prog"| while read filename
do
    while read keyword
    do
       matchCount=`/usr/xpg4/bin/grep -F -i -x "$keyword" "$filename" | wc -l`
       if [ $matchCount -ne 0 ];then
 
  out3=`echo "$filename"|awk -F\. '{print $2}'`
  
  bfilename=`basename "$filename"`
  
  case $out3 in
   'rdf')   catagoery="REPORT";;
      
   'fmb')   catagoery="FORM";;
   'sql')   catagoery="SQL FILE";;
   'pll')   catagoery="Library File";;
   'ctl')   catagoery="Control File";;
   'sh')   catagoery="Shell script";;
    *)    catagoery="OTHER";;
  esac 
  
  echo "bfilename,keyword,matchCount,out3,catagoery are:- $bfilename,$keyword,$matchCount,$out3,$catagoery"
  sqlplus -s $usrname/$password@$dbSID <<-SQL >> spot_fsearch.log
  INSERT INTO AA_DETAIL (FILE_NAME,DEP_OBJECT_NAME,OCCURANCE,FILE_TYPE,PROGRAM_TYPE) values ('$bfilename','$keyword',$matchCount,'$out3','$catagoery');
  UPDATE BB_DETAIL SET (DEP_OBJECT_TYPE,MODULE_SHORT_NAME,APPLICATION,OBJECT_STATUS,OBJ_ADDN_INFO) = (SELECT OBJECT_TYPE,MODULE_SHORT_NAME,APPLICATION,OBJECT_STATUS,OBJ_ADDN_INFO FROM CG_COMPARATIVE_MATRIX_TAB WHERE upper(OBJECT_NAME)=upper('$keyword') AND ROWNUM<2) WHERE upper(DEP_OBJECT_NAME) = upper('$keyword');
  UPDATE CC_CUSTOM_FILES_SUMMARY SET IMPACTED_BY_UPGRADE='$out2' WHERE FILE_NAME='$bfilename';
  quit;
 
SQL
       fi
    done < $keywordfile
done

# 2  
Searching 8000 keywords in 300 large files is quite something, but the program you show can be optimized for speed.
a) Don't open and reread the keyword file line by line for every file matching your pattern.
b) Don't run the grep process for every single keyword/file combination (300 x 8000 = 2.4 million times!)
c) Don't use wc -l piped to the greps (again 2.4 million times)
d) Don't run the sql command including login for every single keyword/file combination; collect the results into a file and insert & update afterwards.
# 3  
Yes Rudic..you have correctly pointed these points.

I can understand these are the problem in this code.
But i m not able to enhance the code to avoid these.

Can you pls suggest what kind of code change i can do for this.
# 4  
This is untested and far from complete; you need to experiment. It should replace your two while loops as it reads all the keywords, and then scans all the files found by your find command. It will produce an output that you can capture into a file that you can sqlload into your DB in one go; thereafter do the inserts and updates:
Code:
awk     'BEGIN          {CAT["rdf"]="REPORT"                
                         CAT["fmb"]="FORM"                 
                         CAT["sql"]="SQL FILE"                 
                         CAT["pll"]="Library File"                 
                         CAT["ctl"]="Control File"                 
                         CAT["sh"]= "Shell script"                
                        }

         FNR == NR      {KY[$0]; next}                                  # read in all the keywords

         FNR == 1 && FN {EXT = FN; sub (/.*\./,".", EXT)                # if new file, obtain the extension
                         for (i in MCNT)                                # for all matches,
                           print FN, i, MCNT[i], EXT, CAT[EXT]          # print out the old values 
                         FN = FILENAME                                  # retain FILENAME for next loop
                        }

                        {for (i in KY) if ($0 ~ i) MCNT[i]++}           # find matching keywords in each line

         END            {EXT = FN; sub (/.*\./,".", EXT)                # same as above for last file
                         for (i in MCNT) 
                           print FN, i, MCNT[i], EXT, CAT[EXT]
                        }
        ' $keywordfile $(find $tmpdir -type f -name ....)               # may blast your LINE_MAX

# 5  
Hi.

For multiple CPU hardware, introducing some parallelism might be useful in decreasing real time.

I think I would split the list of files to be examined into several pieces and then run whatever scanning program is desired on each piece.

There are several utilities to help with the control of simultaneous processes: xargs, parallel, etc.

Best wishes ... cheers, drl
# 6  
Hi Rudic,

I ran the the code which gave me error near the find command.

syntax error at line 25: `(' unexpected

So i have backquoted the find command and run as below.

Code:
 
keywordfile="keyword.txt"
/usr/xpg4/bin/awk    'BEGIN          {CAT["rdf"]="REPORT"                
                         CAT["fmb"]="FORM"                 
                         CAT["sql"]="SQL FILE"                 
                         CAT["pll"]="Library File"                 
                         CAT["ctl"]="Control File"                 
                         CAT["sh"]= "Shell script"                
                        }
         FNR == NR      {KY[$0]; next}                                  # read in all the keywords
         FNR == 1 && FN {EXT = FN; sub (/.*\./,".", EXT)                # if new file, obtain the extension
                         for (i in MCNT)                                # for all matches,
                           print FN, i, MCNT[i], EXT, CAT[EXT]          # print out the old values 
                         FN = FILENAME                                  # retain FILENAME for next loop
                        }
                        {for (i in KY) if ($0 ~ i) MCNT[i]++}           # find matching keywords in each line
         END            {EXT = FN; sub (/.*\./,".", EXT)                # same as above for last file
                         for (i in MCNT) 
                           print FN, i, MCNT[i], EXT, CAT[EXT]
                        }
        ' $keywordfile `/usr/xpg4/bin/find /usr/tmp/SB -type f -name "*.rdf" -o -name "*.fmb" -o -name "*.pll" -o -name "*.ctl" -o -name "*.sh" -o -name "*.sql" -o -name "*.prog"`

but it is giving me the error as below.

Code:
/usr/xpg4/bin/awk: line 16 (NR=7758): /DR$PV_ENTY_ATTR_TEXTS_U2$R/: unknown regex error

And i checked the keyword file and can see some of keywords contain $ symbol.So it is breaking.

And also some filenames contains space.


Please let me know what modification i should do here.

Thank you

Last edited by millan; 06-12-2013 at 09:12 AM..
# 7  
As I said: You need to experiment. Try printing the lines with matches. Try smaller files.
Why don't you create a, say, 10 keyword file, and work on a subset of two or three sample files that have a known set of keywords within?
The error msg you post points to the END section, i.e. the problem is within the last file. Which can be good news, as all the earlier files passed!

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #5
Difficulty: Easy
The UNIX trademark is a "certification mark".
True or False?

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script search, improve performance with large files

Hello, For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)
Discussion started by: SDohmen
15 Replies

2. Shell Programming and Scripting

Grepping verbal forms from a large corpus

I want to extract verbal forms from a large corpus of English. I have identified a certain number of patterns. Each pattern has the following structure SPACE word_CATEGORY where word refers to the verbal form and CATEGORY refers to the class of the verb The categories are identified as per the... (4 Replies)
Discussion started by: gimley
4 Replies

3. Shell Programming and Scripting

Grepping large list of files

Hi All, I need help to know the exact command when I grep large list of files. Either using ls or find command. However I do not want to find in the subdirectories as the number of subdirectories are not fixed. How do I achieve that. I want something like this: find ./ -name "MYFILE*.txt"... (2 Replies)
Discussion started by: angshuman
2 Replies

4. Red Hat

Empty directory, large size and performance

Hi, I've some directory that I used as working directory for a program. At the end of the procedure, the content is deleted. This directory, when I do a ls -l, appears to still take up some space. After a little research, I've seen on a another board of this forum that it's not really taking... (5 Replies)
Discussion started by: bdx
5 Replies

5. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ... (6 Replies)
Discussion started by: Souvik
6 Replies

6. Shell Programming and Scripting

replace issue with large files

I have the following problem: I have two files: S containing sentences (one in each row) and W containing files (one in each row). It might look like this: S: a b c apple d. e f orange g. h banana i j. W: orange banana apple My task is to replace in S all words that appear in W... (2 Replies)
Discussion started by: tootles564
2 Replies

7. Shell Programming and Scripting

Performance issue in UNIX while generating .dat file from large text file

Hello Gurus, We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this . Problem Definition: /Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below... (19 Replies)
Discussion started by: KRAMA
19 Replies

8. Shell Programming and Scripting

Grepping issue..

I found another problem with my disk-adding script today. When looking for disks, I use grep. When I grep for the following disk sizes: 5242880 I also pick up these as well: 524288000 How do I specifically pick out one or the other, using grep, without resorting to the -v option? ... (9 Replies)
Discussion started by: LinuxRacr
9 Replies

9. UNIX for Dummies Questions & Answers

Unix File System performance with large directories

Hi, how does the Unix File System perform with large directories (containing ~30.000 files)? What kind of structure is used for the organization of a directory's content, linear lists, (binary) trees? I hope the description 'Unix File System' is exact enough, I don't know more about the file... (3 Replies)
Discussion started by: dive
3 Replies

Featured Tech Videos