The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
.
google unix.com



Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Can you suggest a more efficient way for this? mikie Shell Programming and Scripting 1 11-20-2006 10:49 AM
Efficient way of Awk braindrain Shell Programming and Scripting 4 07-11-2006 02:01 PM
Is there a more efficient way? SelectSplat Shell Programming and Scripting 6 12-16-2005 04:59 PM
Efficient I/O S.P.Prasad High Level Programming 1 12-09-2003 01:38 PM
Efficient Dispatching uchachra UNIX for Advanced & Expert Users 1 01-10-2003 05:33 PM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Bulgarian Greek Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 05-03-2006
Glenn Arndt's Avatar
Glenn Arndt Glenn Arndt is offline Forum Advisor  
Anomalous Lurker
  
 

Join Date: Feb 2006
Location: Indianapolis, IN
Posts: 255
Angry Need more efficient log file grep

I'm writing a script that at one point needs to check the contents of another script's log file to determine how to proceed. An example record from the log file is:

"mcref04152006","060417","ANTH0415","282","272","476,983.37","465,268.44","loaded"

I want my script to return this record if:

1.) the record contains "mcref", which is declared earlier in the script as $reffilepattern, and
2.) the record contains "060417", which is declared earlier in the script as $todayyymmdd.

I don't need to be picky about which fields contain the data, because the format is consistent. I'm presently doing this to read the entire line into the variable $loadedref:

Code:
todayyymmdd=$(date +%y%m%d)
reflog=reflogfile.txt
reffilepattern="mcref"
...
loadedref=$(grep "\"$todayyymmdd\"" $reflog | grep $reffilepattern)
Obviously, this works just fine, except that piping the results of one grep into another grep seems unholy. The log file is small, so while efficiency is not a concern at this point, it will be eventually as the log file grows. I tried to combine this into one grep that says, "give me the record that contains both "mcref" (without quotes) AND "060417" (with quotes), with any number of characters in between (including none)." I've tried goofy stuff like:

Code:
grep ${reffilepattern}[A-Za-z0-9]*\"${todayyymmdd}\" $reflog
...which doesn't work. Can anyone give me some pointers here? I know I'm missing something really simple -- it's late in the day! Thanks in advance.
  #2 (permalink)  
Old 05-03-2006
x96riley3 x96riley3 is offline
Registered User
  
 

Join Date: Mar 2006
Location: Nashville
Posts: 79
I think what you have is fine. If your worried about performace then I suggest going to PERL. Perl is much faster and better at parsing files. It's what perl was meant for.

I was going to write the script but I got tired and I'm going home. I'll check back tomorrow.

-X
  #3 (permalink)  
Old 05-03-2006
DR_DAVE DR_DAVE is offline
Registered User
  
 

Join Date: May 2006
Location: Sun Diego, CA
Posts: 3
Would this work for you:

grep "${reffilepattern}.*${todayyymmdd}" $reflog

I tried on a test file with your data, seemed to work, let us know...
  #4 (permalink)  
Old 05-04-2006
Glenn Arndt's Avatar
Glenn Arndt Glenn Arndt is offline Forum Advisor  
Anomalous Lurker
  
 

Join Date: Feb 2006
Location: Indianapolis, IN
Posts: 255
Interesting -- grep "${reffilepattern}.*${todayyymmdd}" $reflog works (I can't believe I didn't think of that), but on a larger test file (about 60MB) it is actually 2 to 3 times slower than the original method of piping one grep through another. I'm guessing because the first grep narrows the records down for the second grep. I guess I'll just keep my original method. Thanks for the help, though. This is good to know!
Closed Thread

Bookmarks

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On




All times are GMT -4. The time now is 11:27 AM.


Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0