Sponsored Content
Top Forums UNIX for Advanced & Expert Users How to read a fast written log file at Real time speed? Post 302936623 by cool.aquarian on Thursday 26th of February 2015 06:10:08 PM
Old 02-26-2015
CPU & Memory How to read a fast written log file at Real time speed?

Hello All,

I am building a real time parser for a log file in my application.
The log file is continuously written at a very fast pace and gets rolled over every 10 minutes.

I have measured the speed and observed that around 1000 lines are written to it every second, each line about 30-40 characters.

I have tried using tail -F, but it always lags behind the speed at which the file is being written.
Could you suggest anything else I can use to read the file line by line quickly at Realtime speed ?

Below is what I have right now:
Code:
tail -F --lines=10000000 --retry --max-unchanged-stats=10 "$logFile" | while IFS= read -r line || [ -n "$line" ]
do
 -- some logic --
done

Thank you.
-CaQ
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

log users real time

hi.... how i can configurator a log file on real time....on unix solaris.... thanks a lot.... Best Regards... (3 Replies)
Discussion started by: chanfle
3 Replies

2. Shell Programming and Scripting

Real time log file redirect

Hi all, i would like to write the shell script program, it can monitor the access_log "real time" when the access_log writing the line contain "abcdef" the program will be "COPY" this line into a file named "abcdef.txt", do the same thing if the contain "123456" "COPY" it into a file named... (3 Replies)
Discussion started by: eric_wong_ch
3 Replies

3. Shell Programming and Scripting

Perl or Shell script to read a transaction log in real time

Hello, I have a Apache webserver running on RedHat. Its primary function is a proxy server for users accessing the internet. I have a transaction log that logs every transactions of every users. For users trying to access certain sites/content the transactions goes into a 302 redirect loop and... (2 Replies)
Discussion started by: bruno406
2 Replies

4. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies

5. Filesystems, Disks and Memory

data from blktrace: read speed V.S. write speed

I analysed disk performance with blktrace and get some data: read: 8,3 4 2141 2.882115217 3342 Q R 195732187 + 32 8,3 4 2142 2.882116411 3342 G R 195732187 + 32 8,3 4 2144 2.882117647 3342 I R 195732187 + 32 8,3 4 2145 ... (1 Reply)
Discussion started by: W.C.C
1 Replies

6. Shell Programming and Scripting

Read/Search file being written to giving error due to timing issues

The following is a piece of code to rename LOG_FILE_NEW to LOG_FILE once you get a result (either RUNNING or SHUTDOWN) RESULT="" sleep 30 while ; do sleep 10 RESULT=`sed -n '/RUNNING/'p ${LOG_FILE_NEW}` if ; then RESULT=`sed -n '/SHUTTING_DOWN/'p ${LOG_FILE_NEW}` fi done mv... (3 Replies)
Discussion started by: sonorous
3 Replies

7. HP-UX

HP-UX real time audit log writing

Hey all, I have a problem I was hoping to get some help on. So I have my two auditfiles, audfile1 and audfile2 that can be written to, I want to have the text version of them write to an NFS mount that I have set up. So i already know that i can do .secure/etc/audsp audfile1 > //nfsmount/folder/... (5 Replies)
Discussion started by: CleverRiver6
5 Replies

8. Shell Programming and Scripting

[solved] How to see log in real time?

Hi people I have a bash script with a line like this: python example.py >> log & But i can't see anything in the log file while python program is running only if the program ends seems to write the log file. "$ cat log" for example don't show anything until the program ends. Is there... (4 Replies)
Discussion started by: Tieso
4 Replies

9. Shell Programming and Scripting

Archiving or removing few data from log file in real time

Hi, I have a log file that gets updated every second. Currently the size has grown to 20+ GB. I need to have a command/script, that will try to get the actual size of the file and will remove 50% of the data that are in the log file. I don't mind removing the data as the size has grown to huge... (8 Replies)
Discussion started by: Souvik Patra
8 Replies

10. Shell Programming and Scripting

Log all the commands input by user at real time in /var/log/messages

Below is my script to log all the command input by any user to /var/log/messages. But I cant achieve the desired output that i want. PLease see below. function log2syslog { declare COMMAND COMMAND=$(fc -ln -0) logger -p local1.notice -t bash -i -- "$USER:$COMMAND" } trap... (12 Replies)
Discussion started by: invinzin21
12 Replies
TAIL(1) 							   User Commands							   TAIL(1)

NAME
tail - output the last part of files SYNOPSIS
tail [OPTION]... [FILE]... DESCRIPTION
Print the last 10 lines of each FILE to standard output. With more than one FILE, precede each with a header giving the file name. With no FILE, or when FILE is -, read standard input. Mandatory arguments to long options are mandatory for short options too. -c, --bytes=K output the last K bytes; alternatively, use -c +K to output bytes starting with the Kth of each file -f, --follow[={name|descriptor}] output appended data as the file grows; -f, --follow, and --follow=descriptor are equivalent -F same as --follow=name --retry -n, --lines=K output the last K lines, instead of the last 10; or use -n +K to output lines starting with the Kth --max-unchanged-stats=N with --follow=name, reopen a FILE which has not changed size after N (default 5) iterations to see if it has been unlinked or renamed (this is the usual case of rotated log files). With inotify, this option is rarely useful. --pid=PID with -f, terminate after process ID, PID dies -q, --quiet, --silent never output headers giving file names --retry keep trying to open a file even when it is or becomes inaccessible; useful when following by name, i.e., with --follow=name -s, --sleep-interval=N with -f, sleep for approximately N seconds (default 1.0) between iterations. With inotify and --pid=P, check process P at least once every N seconds. -v, --verbose always output headers giving file names --help display this help and exit --version output version information and exit If the first character of K (the number of bytes or lines) is a `+', print beginning with the Kth item from the start of each file, other- wise, print the last K items in the file. K may have a multiplier suffix: b 512, kB 1000, K 1024, MB 1000*1000, M 1024*1024, GB 1000*1000*1000, G 1024*1024*1024, and so on for T, P, E, Z, Y. With --follow (-f), tail defaults to following the file descriptor, which means that even if a tail'ed file is renamed, tail will continue to track its end. This default behavior is not desirable when you really want to track the actual name of the file, not the file descrip- tor (e.g., log rotation). Use --follow=name in that case. That causes tail to track the named file in a way that accommodates renaming, removal and creation. AUTHOR
Written by Paul Rubin, David MacKenzie, Ian Lance Taylor, and Jim Meyering. REPORTING BUGS
Report tail bugs to bug-coreutils@gnu.org GNU coreutils home page: <http://www.gnu.org/software/coreutils/> General help using GNU software: <http://www.gnu.org/gethelp/> Report tail translation bugs to <http://translationproject.org/team/> COPYRIGHT
Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. SEE ALSO
The full documentation for tail is maintained as a Texinfo manual. If the info and tail programs are properly installed at your site, the command info coreutils 'tail invocation' should give you access to the complete manual. GNU coreutils 8.12.197-032bb September 2011 TAIL(1)
All times are GMT -4. The time now is 10:20 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy