How to read a fast written log file at Real time speed?


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users How to read a fast written log file at Real time speed?
# 1  
CPU & Memory How to read a fast written log file at Real time speed?

Hello All,

I am building a real time parser for a log file in my application.
The log file is continuously written at a very fast pace and gets rolled over every 10 minutes.

I have measured the speed and observed that around 1000 lines are written to it every second, each line about 30-40 characters.

I have tried using tail -F, but it always lags behind the speed at which the file is being written.
Could you suggest anything else I can use to read the file line by line quickly at Realtime speed ?

Below is what I have right now:
Code:
tail -F --lines=10000000 --retry --max-unchanged-stats=10 "$logFile" | while IFS= read -r line || [ -n "$line" ]
do
 -- some logic --
done

Thank you.
-CaQ
# 2  
That's not really an easy problem to solve. Any solution that meets your requirements has to take into account specifics of the OS, the file system, and even the physical hardware. Obviously such a solution isn't going to be very portable.

The first thing you have to figure out is what's fast enough? Because there's always going to be some delay in reading data from the file after it's been written.

You might be better served interposing something into the logging stream that splits the stream into two - one to the original logging system and one into your real-time parser.
This User Gave Thanks to achenle For This Post:
# 3  
Quote:
Originally Posted by achenle
That's not really an easy problem to solve. Any solution that meets your requirements has to take into account specifics of the OS, the file system, and even the physical hardware. Obviously such a solution isn't going to be very portable.

The first thing you have to figure out is what's fast enough? Because there's always going to be some delay in reading data from the file after it's been written.

You might be better served interposing something into the logging stream that splits the stream into two - one to the original logging system and one into your real-time parser.
Thanks for your post.
About what is fast enough, a delay of 30 seconds or so would be within the accepted level. But I have seen that on average, the tail command is able to read less than 70% of the file(within 10 minutes) before the file gets rolled over and script throws the error (as the log files are present on a shared NFS mount):
Code:
tail: error reading `<file-path>': Stale NFS file handle


Last edited by cool.aquarian; 02-26-2015 at 11:17 PM..
# 4  
Reliable read-behind-write on NFS? That's not going to happen because NFS is stateless. And yes, what you're trying to do is called "read-behind-write":

https://www.google.com/search?q=%22read+behind+write%22

If you have a requirement to monitor those logs, you need to monitor them another way. Reading them over NFS is not going to work.
# 5  
This is never going to work right on NFS. NFS is also probably why tail couldn't keep up. You should have the application send logs to you. How to do this depends on the application.

Last edited by Corona688; 02-27-2015 at 02:25 PM..
# 6  
Quote:
Originally Posted by achenle
Reliable read-behind-write on NFS? That's not going to happen because NFS is stateless
NFSv4 which is increasing the norm on Linux systems is statefull. Earlier versions were stateless.
# 7  
Hi there..

I managed to improve the performance by 10-fold for real time reading.

I analysed and found that most time was being taken by
Code:
while IFS= read -r line

I replaced the while loop with awk to read a single logical unit(each about 6000 lines) in one go.

Code:
tail -F --lines=10000000 --retry --max-unchanged-stats=10 "$FILE_NAME" | awk -f awk-script.txt

Now the speed of reading is able to more than keep up with speed of write.
Even when the file gets rolled over, it is able to catch-up with the new file and gives message like:
tail: `<file-name>' has been replaced; following end of new file

Last edited by cool.aquarian; 03-06-2015 at 07:40 PM..
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #366
Difficulty: Medium
Bill Joy was responsible for creating the third BSD Unix release in March 1978.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Log all the commands input by user at real time in /var/log/messages

Below is my script to log all the command input by any user to /var/log/messages. But I cant achieve the desired output that i want. PLease see below. function log2syslog { declare COMMAND COMMAND=$(fc -ln -0) logger -p local1.notice -t bash -i -- "$USER:$COMMAND" } trap... (12 Replies)
Discussion started by: invinzin21
12 Replies

2. Shell Programming and Scripting

Archiving or removing few data from log file in real time

Hi, I have a log file that gets updated every second. Currently the size has grown to 20+ GB. I need to have a command/script, that will try to get the actual size of the file and will remove 50% of the data that are in the log file. I don't mind removing the data as the size has grown to huge... (8 Replies)
Discussion started by: Souvik Patra
8 Replies

3. Shell Programming and Scripting

[solved] How to see log in real time?

Hi people I have a bash script with a line like this: python example.py >> log & But i can't see anything in the log file while python program is running only if the program ends seems to write the log file. "$ cat log" for example don't show anything until the program ends. Is there... (4 Replies)
Discussion started by: Tieso
4 Replies

4. HP-UX

HP-UX real time audit log writing

Hey all, I have a problem I was hoping to get some help on. So I have my two auditfiles, audfile1 and audfile2 that can be written to, I want to have the text version of them write to an NFS mount that I have set up. So i already know that i can do .secure/etc/audsp audfile1 > //nfsmount/folder/... (5 Replies)
Discussion started by: CleverRiver6
5 Replies

5. Shell Programming and Scripting

Read/Search file being written to giving error due to timing issues

The following is a piece of code to rename LOG_FILE_NEW to LOG_FILE once you get a result (either RUNNING or SHUTDOWN) RESULT="" sleep 30 while ; do sleep 10 RESULT=`sed -n '/RUNNING/'p ${LOG_FILE_NEW}` if ; then RESULT=`sed -n '/SHUTTING_DOWN/'p ${LOG_FILE_NEW}` fi done mv... (3 Replies)
Discussion started by: sonorous
3 Replies

6. Filesystems, Disks and Memory

data from blktrace: read speed V.S. write speed

I analysed disk performance with blktrace and get some data: read: 8,3 4 2141 2.882115217 3342 Q R 195732187 + 32 8,3 4 2142 2.882116411 3342 G R 195732187 + 32 8,3 4 2144 2.882117647 3342 I R 195732187 + 32 8,3 4 2145 ... (1 Reply)
Discussion started by: W.C.C
1 Replies

7. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies

8. Shell Programming and Scripting

Perl or Shell script to read a transaction log in real time

Hello, I have a Apache webserver running on RedHat. Its primary function is a proxy server for users accessing the internet. I have a transaction log that logs every transactions of every users. For users trying to access certain sites/content the transactions goes into a 302 redirect loop and... (2 Replies)
Discussion started by: bruno406
2 Replies

9. Shell Programming and Scripting

Real time log file redirect

Hi all, i would like to write the shell script program, it can monitor the access_log "real time" when the access_log writing the line contain "abcdef" the program will be "COPY" this line into a file named "abcdef.txt", do the same thing if the contain "123456" "COPY" it into a file named... (3 Replies)
Discussion started by: eric_wong_ch
3 Replies

10. UNIX for Dummies Questions & Answers

log users real time

hi.... how i can configurator a log file on real time....on unix solaris.... thanks a lot.... Best Regards... (3 Replies)
Discussion started by: chanfle
3 Replies

Featured Tech Videos