Hi,
I am planning to implement a scheduled script that will go against my log files (every hour), search for a set of key words (errors, exceptions, faults etc). The script must be intelligent enough to scan only the new lines added to the log file since it last ran.

I can use grep for searching, how can I scan only new lines added to the log file since it last ran?

Any idea? any sample script?

You can find out how many lines are in the log file with wc -l. Use a second wc -l on the file to find out how many more lines were added - then just tail the difference. Save your last wc -l into a separate file for use on the next run

Run this command to start it all off (change the location and name of file to suit your needs)
wc -l logfile > /tmp/logfile.linecount

Get the number of lines from the linecount file (use cut or awk to get only the first field). Use that number and a new wc -l of the log file to find out how many new lines. Do your processing against only the new lines utilizing the tail command (see the man page). Save the new number of lines to your linecount file for the next run.

Here is the trick I think (Moderators please advise).

Use diff.
Suppose u r having file1 and file2. file2 is new and u want to search for occurence of LOTUS and red in new lines added in file2.
U can do this:

diff file1 file2 >> diff12
egrep 'LOTUS|red' diff12

This will work for single file.
If u have many files...U will have to right a shell script..
It will hae somewhat following structure.. (I have written algorithm so it won't be syntactically correct)

DO WHILE END OF NUMBER OF FILES.
integer i
string name
name = 'difffile' & string(i) & 'diff' --> if i = 4 then name = difffile4.diff
diff folder1/file1 folder2/file2 >> file2/name.diff
LOOP

this will give u a set of files with diferences in set of log files with only new lines.
(I am assuming that existing entries in a log file won't get updated or changed. E.g. if file1 contains line 1 as 'REDANDGREEN' then file 2 will contain line1 as REDANDGREEN and not REDANDORANGE. If the existing entries are going to get updated then u may need to process the files with differences again to keep only new line differences. Try out DIFF and try to investigate more.)
U can then fire a grep on these files with differences.

Let me know ur comments..
Thank
VEN

I have used the same solution as forwarded by RTM and it works very well that way ... the only thing is you might want to make sure your filecounter file is not in /tmp just in case of an unplanned reboot --- i put mine in /var/tmp which is not a volatile directory in my solaris setup ...