How to scan only new lines added in file?


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers How to scan only new lines added in file?
# 1  
Old 04-28-2005
How to scan only new lines added in file?

Hi,
I am planning to implement a scheduled script that will go against my log files (every hour), search for a set of key words (errors, exceptions, faults etc). The script must be intelligent enough to scan only the new lines added to the log file since it last ran.

I can use grep for searching, how can I scan only new lines added to the log file since it last ran?

Any idea? any sample script?
# 2  
Old 04-28-2005
You can find out how many lines are in the log file with wc -l. Use a second wc -l on the file to find out how many more lines were added - then just tail the difference. Save your last wc -l into a separate file for use on the next run

Run this command to start it all off (change the location and name of file to suit your needs)
wc -l logfile > /tmp/logfile.linecount

Get the number of lines from the linecount file (use cut or awk to get only the first field). Use that number and a new wc -l of the log file to find out how many new lines. Do your processing against only the new lines utilizing the tail command (see the man page). Save the new number of lines to your linecount file for the next run.
# 3  
Old 04-28-2005
Searching thru only new lines add

Here is the trick I think (Moderators please advise).

Use diff.
Suppose u r having file1 and file2. file2 is new and u want to search for occurence of LOTUS and red in new lines added in file2.
U can do this:

diff file1 file2 >> diff12
egrep 'LOTUS|red' diff12

This will work for single file.
If u have many files...U will have to right a shell script..
It will hae somewhat following structure.. (I have written algorithm so it won't be syntactically correct)

DO WHILE END OF NUMBER OF FILES.
integer i
string name
name = 'difffile' & string(i) & 'diff' --> if i = 4 then name = difffile4.diff
diff folder1/file1 folder2/file2 >> file2/name.diff
LOOP

this will give u a set of files with diferences in set of log files with only new lines.
(I am assuming that existing entries in a log file won't get updated or changed. E.g. if file1 contains line 1 as 'REDANDGREEN' then file 2 will contain line1 as REDANDGREEN and not REDANDORANGE. If the existing entries are going to get updated then u may need to process the files with differences again to keep only new line differences. Try out DIFF and try to investigate more.)
U can then fire a grep on these files with differences.

Let me know ur comments..
Thank
VEN
# 4  
Old 04-28-2005
I have used the same solution as forwarded by RTM and it works very well that way ... the only thing is you might want to make sure your filecounter file is not in /tmp just in case of an unplanned reboot --- i put mine in /var/tmp which is not a volatile directory in my solaris setup ...
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Error after I added several lines

I have added several lines to my code let cnt=0 if then if then cnt=`expr ${cnt} + 1` sftp_name=EOMMSRPT1 echo "EOMsftpname = ${sftp_name}" >> ${LOGF} ... (2 Replies)
Discussion started by: digioleg54
2 Replies

2. Shell Programming and Scripting

Scan a file in realtime and execute certain commands on encountering 5 consecutive identical lines

Mysql log has something like below: I need a bash shell script that will do the following: 1) The script will scan the mysql.log file constantly in real time (something like tail -F mysql.log) 2) If it encounters 5 consecutive identical lines then it would invoke some commands (say... (4 Replies)
Discussion started by: proactiveaditya
4 Replies

3. Shell Programming and Scripting

Log File Scan

I need to read the last line of a log file and save it, sleep for X minutes and read the last line again. If the line is the same, exit 1, otherwise sleep for X minutes until the last line contains 'Status: Process completed'. Can anyone offer advice here? Thanks. (2 Replies)
Discussion started by: mode09
2 Replies

4. Shell Programming and Scripting

I want Trailer to be added into the text file.

Hi folks, I want Trailer to be added into the txt file the format is below. flatfile-> abc.txt count of the file is 500 records. I want the trailer in this format: TRAILER|500 (pipe delimeter). Please suggest the comands ASAP. Rgds Ann (5 Replies)
Discussion started by: Haque123
5 Replies

5. Shell Programming and Scripting

how to check if a new line has been added to a file?

Have come up with the following but it doesn't seem to work.. Is there some other command i could use to get this to work? OUTPATH=/home/out PARMFILE=$OUTPATH/jobcount_test.txt LOG=$OUTPATH/job_count_monthlymail_log.txt HLOG=$OUTPATH/job_count_monthlymail_hlog.txt # echo " started at... (2 Replies)
Discussion started by: Jazmania
2 Replies

6. Shell Programming and Scripting

Scan a log file

hi guys, i am very new to scripting & lookin for a shell script/perl script which would scan another file with the keyword "no change" & take a count of the same. Let me know if any further details are required. (3 Replies)
Discussion started by: nhanda
3 Replies

7. Shell Programming and Scripting

Get the latest added part of a text file?

Hi, I want to get the latest added part of a text file sent over the network to for analysis. Is there a tool to use to keep track of which part of a text file that has already been analysed so only the "new" lines will be sent and handled? I have checked a few tools but I still donīt know how to... (3 Replies)
Discussion started by: pcrs
3 Replies

8. Shell Programming and Scripting

Perl script to scan back lines

Hi Perl gurus, I have this file to scan through. Sample lines below: 2008031A, USERNAME, 12345, give ABC, take XYZ, transaction submitted 2008031B, USERNAME, 12346, waiting for processing 2008031C, USERNAME, 12347, Retrieving response 2008031D, USERNAME, 12348, This is not a valid dealing... (3 Replies)
Discussion started by: gholdbhurg
3 Replies

9. Shell Programming and Scripting

File Inventory Scan

Is there such a script out there that will Analyze a folder and subfolders, getting file type utilization and file aging. A CSV file will be created with file information File,Size,Date Created,Last Modified,Last Accessed,Extension,File Type,Owner A Summary report text file will also be created... (4 Replies)
Discussion started by: seacros
4 Replies

10. Shell Programming and Scripting

File Scan

Hi everyone , i m working on Sun solaris and i have a file "smsapp.cur" which has information like this paragraph given below , there are millions of such paragraphs From:923212802736 To:923222326807 logMessage: 07-04-08 17:34:29 Getting message topup from code page default in language English... (2 Replies)
Discussion started by: Dastard
2 Replies
Login or Register to Ask a Question