Detecting new entries in log files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Detecting new entries in log files
# 1  
Old 08-07-2013
Detecting new entries in log files

Hello All,

I have a script that checks a log file. This will be checked periodically lets say every 5 minutes. I need to get new entries and process these new entries

What do you think is the best strategy here? all i can think of currently is to backup the old file and do
Code:
diff

. But future problem will be once the log increases in size, this method will be slower.

Any suggestion is appreciated.
# 2  
Old 08-07-2013
Use stat to get the modification timestamp of the file and store it on a variable. Compare it later to check changes. Of course the filesystem should be consistent with updates. Some filesystem may have no modification timestamp? I don't know. Perhaps.
# 3  
Old 08-07-2013
thanks. modification time is a good indication of something was updated. but i need to fetch the new entries and process it which stat cannot do.
# 4  
Old 08-07-2013
You could read the log in real time with tail -f if that helps. Beyond this, you may need to keep and old version and compare them. Using diff can be awkward because it adds editing messages if you are to shovel the output into ed so perhaps you would be better to count the lines in your saved version and then get extra lines from the current file, something like:-
Code:
cp logfile temp_logfile
old_lines=`grep -c "" old_logfile`
temp_lines=`grep -c "" temp_logfile`
((lines=$new_lines-$old_lines))
tail -n $lines temp_logfile
.... some other processing if you like.....
mv temp_logfile old_logfile

I've added the temporary log file so that messages can still be added to the main logfile as you are working on it without it skewing the output, so it givers you a fixed reference point.




i hope that this might help.



Robin
Liverpool/Blackburn
UK
This User Gave Thanks to rbatte1 For This Post:
# 5  
Old 08-07-2013
ill try your suggestion. thanks!
# 6  
Old 08-07-2013
Well basically you'll have to check the file's contents from time to time but using stat as the starting point would help save your disk from much IO.
# 7  
Old 08-07-2013
If the logfile have date stamp, you can look at them every 5 minutes and get only the whats new the last 5 min
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Redirecting log files to null writing junk into log files

Redirecting log files to null writing junk into log files. i have log files which created from below command exec <processname> >$logfile but when it reaches some size i am redirecting to null while process is running like >$logfile manually but after that it writes some junk into... (7 Replies)
Discussion started by: greenworld123
7 Replies

2. Shell Programming and Scripting

Monitor log entries in log files with no Date format? - Efficient logcheck?

is there a way to efficiently monitor logfiles that do not have a date or time format? i have several logs on several different servers that need to be monitored. but i realized writing a script for this would be very complex and time consuming giving the variety of things i need to check for i.e.... (2 Replies)
Discussion started by: SkySmart
2 Replies

3. Fedora

Missing entries in log files just before/after reboot

Hello world, One of the servers, a Fedora one,rebooted today (Luckily, a testbox). I tried to get the reason the server rebooted. After going through the messages, I think that the log entries just before and after reboot are missing. Please below: (****** is the server name, for privacy... (0 Replies)
Discussion started by: satish51392111
0 Replies

4. Shell Programming and Scripting

How can view log messages between two time frame from /var/log/message or any type of log files

How can view log messages between two time frame from /var/log/message or any type of log files. when logfiles are very big and especially many messages with in few minutes, I would like to display log messages between 5 minute interval. Could you pls give me the command? (1 Reply)
Discussion started by: johnveslin
1 Replies

5. Shell Programming and Scripting

Detecting dates in foldernames

Hi, I will name folders this way : DD-MM-YYYY (07-06-2011 for today). DATE=`date +%d-%m-%Y` mkdir $DATE They will contain a backup of the day. I want, in my backup script, add a command that will automatically delete folders that are a week old (in this case, when performing the backup of... (7 Replies)
Discussion started by: Always
7 Replies

6. Shell Programming and Scripting

need help in detecting errors

Hi All , I need a script to find errors in a particular and in a particular path Actually in my logs i`ve so many kinds of errors(i can even say as 100 types also).if i run the script i need to know the error (some errors can aviod ) so finally the script o/p should be a numeric... (3 Replies)
Discussion started by: radha254
3 Replies

7. Shell Programming and Scripting

Detecting incoming files without busy polling

Hello, I'd like to handle incoming (uploaded) files from a shell script, ideally without busy polling / waiting (e.g. running a cron task every 15'). Is there a command that would just sleep until a new entry has been created in a directory, allowing scripts such as the following: while... (9 Replies)
Discussion started by: baldyeti
9 Replies

8. Programming

Detecting interruptions in C

Hi. You may know how to detect when a interruption succeeded programming in C. Just like receiving a signal without blocking. Knowing when it was a keystroke (IRQ 2), or a mouse movement (12), or a disk access, etc. and getting actually for example the letter typed. Thanks a lot. (7 Replies)
Discussion started by: Ashrentum
7 Replies

9. UNIX for Dummies Questions & Answers

Detecting Second disk

Hello all, first of all, I apologise if I may ask stupid or obvious questions, but I'm new to UNIX and I think I need a little bit of help before I start gearing up :) Anyway, I have installed a Solaris 8 on a Sun machine, and it has 2 physical disks in it. However, it seems that it is only... (7 Replies)
Discussion started by: dragunu
7 Replies

10. UNIX for Dummies Questions & Answers

detecting drives

I know that Unix is different from windows in that it needs more manual configuring but how do I get Solaris 8 (Intel version) to recognize my floppy drive and cd-rom?? I mean does it automatically detect the drives at startup and I have to mount them or do I have to create the drives somehow and... (1 Reply)
Discussion started by: eloquent99
1 Replies
Login or Register to Ask a Question