Shell Programming and Scripting

As you wish to go from a line that contains a date string to the end of the file it's probably safe to assume the lines in your file are in date order.

Knowing this it should be possible to write a program that uses a binary chop to seek to the starting line and then process from there. If the file is large this solution will be orders of magnitude faster that a sequential search.

This perl example seem to be pretty close to what I mean, the downside is that seeking into files in this manor is pretty low level and I cant really think on any elegant solution using unix scriping so it will most likley require a proper programming language like perl, python or C - you also have the added complexity of needing to compare date strings instead of straight text.

---------- Post updated at 01:47 PM ---------- Previous update was at 01:24 PM ----------

Another thought if file only has new data appended on the end - keep another text file with each date and the line number it starts on:

Code:

Jan-01-2001 1
Jan-02-2001 7311
Jan-03-2001 15779
...
May-25-2012 574983989

You can then read the file in and start processing from the line number you require:

Code:

LINE=$(grep "May-16-2012" indexfile.txt | awk '{print $2}')
sed -n $LINE',$p' bigfile.log | # <your processing here >

Another alternative:

Code:

awk '!p{if(/May 23, 2012 /)p=1}p' infile

Further to Corona688's approach, this should work cross platform:

Code:

{ sed '/May 23, 2012 /!d;q' ; cat ;} < infile

On Solaris you would probably need to use /usr/xpg4/bin/sed, so set your PATH variable in your script..

Yet another way to speed up might be to use of mawk, which is a faster awk in most cases.