How to read a fast written log file at Real time speed?
Hello All,
I am building a real time parser for a log file in my application.
The log file is continuously written at a very fast pace and gets rolled over every 10 minutes.
I have measured the speed and observed that around 1000 lines are written to it every second, each line about 30-40 characters.
I have tried using tail -F, but it always lags behind the speed at which the file is being written.
Could you suggest anything else I can use to read the file line by line quickly at Realtime speed ?
Hi all,
i would like to write the shell script program, it can monitor the access_log "real time"
when the access_log writing the line contain "abcdef" the program will be "COPY" this line into a file named "abcdef.txt", do the same thing if the contain "123456" "COPY" it into a file named... (3 Replies)
Hello,
I have a Apache webserver running on RedHat. Its primary function is a proxy server for users accessing the internet. I have a transaction log that logs every transactions of every users. For users trying to access certain sites/content the transactions goes into a 302 redirect loop and... (2 Replies)
Hi,
I'm trying to figure out the best solution to the following problem, and I'm not
yet that much experienced like you. :-)
Basically I have to read a fairly large file, composed of "messages" , in order
to display all of them through an user interface (made with QT).
The messages that... (3 Replies)
I analysed disk performance with blktrace and get some data:
read:
8,3 4 2141 2.882115217 3342 Q R 195732187 + 32
8,3 4 2142 2.882116411 3342 G R 195732187 + 32
8,3 4 2144 2.882117647 3342 I R 195732187 + 32
8,3 4 2145 ... (1 Reply)
The following is a piece of code to rename LOG_FILE_NEW to LOG_FILE once you get a result (either RUNNING or SHUTDOWN)
RESULT=""
sleep 30
while ; do
sleep 10
RESULT=`sed -n '/RUNNING/'p ${LOG_FILE_NEW}`
if ; then
RESULT=`sed -n '/SHUTTING_DOWN/'p ${LOG_FILE_NEW}`
fi
done
mv... (3 Replies)
Hey all, I have a problem I was hoping to get some help on. So I have my two auditfiles, audfile1 and audfile2 that can be written to, I want to have the text version of them write to an NFS mount that I have set up. So i already know that i can do .secure/etc/audsp audfile1 > //nfsmount/folder/... (5 Replies)
Hi people
I have a bash script with a line like this:
python example.py >> log &
But i can't see anything in the log file while python program is running only if the program ends seems to write the log file.
"$ cat log" for example don't show anything until the program ends.
Is there... (4 Replies)
Hi,
I have a log file that gets updated every second. Currently the size has grown to 20+ GB. I need to have a command/script, that will try to get the actual size of the file and will remove 50% of the data that are in the log file. I don't mind removing the data as the size has grown to huge... (8 Replies)
Below is my script to log all the command input by any user to /var/log/messages. But I cant achieve the desired output that i want. PLease see below.
function log2syslog
{
declare COMMAND
COMMAND=$(fc -ln -0)
logger -p local1.notice -t bash -i -- "$USER:$COMMAND"
}
trap... (12 Replies)
Discussion started by: invinzin21
12 Replies
LEARN ABOUT DEBIAN
tail
TAIL(1) User Commands TAIL(1)NAME
tail - output the last part of files
SYNOPSIS
tail [OPTION]... [FILE]...
DESCRIPTION
Print the last 10 lines of each FILE to standard output. With more than one FILE, precede each with a header giving the file name. With
no FILE, or when FILE is -, read standard input.
Mandatory arguments to long options are mandatory for short options too.
-c, --bytes=K
output the last K bytes; alternatively, use -c +K to output bytes starting with the Kth of each file
-f, --follow[={name|descriptor}]
output appended data as the file grows; -f, --follow, and --follow=descriptor are equivalent
-F same as --follow=name --retry
-n, --lines=K
output the last K lines, instead of the last 10; or use -n +K to output lines starting with the Kth
--max-unchanged-stats=N
with --follow=name, reopen a FILE which has not changed size after N (default 5) iterations to see if it has been unlinked or
renamed (this is the usual case of rotated log files). With inotify, this option is rarely useful.
--pid=PID
with -f, terminate after process ID, PID dies
-q, --quiet, --silent
never output headers giving file names
--retry
keep trying to open a file even when it is or becomes inaccessible; useful when following by name, i.e., with --follow=name
-s, --sleep-interval=N
with -f, sleep for approximately N seconds (default 1.0) between iterations. With inotify and --pid=P, check process P at least
once every N seconds.
-v, --verbose
always output headers giving file names
--help display this help and exit
--version
output version information and exit
If the first character of K (the number of bytes or lines) is a `+', print beginning with the Kth item from the start of each file, other-
wise, print the last K items in the file. K may have a multiplier suffix: b 512, kB 1000, K 1024, MB 1000*1000, M 1024*1024, GB
1000*1000*1000, G 1024*1024*1024, and so on for T, P, E, Z, Y.
With --follow (-f), tail defaults to following the file descriptor, which means that even if a tail'ed file is renamed, tail will continue
to track its end. This default behavior is not desirable when you really want to track the actual name of the file, not the file descrip-
tor (e.g., log rotation). Use --follow=name in that case. That causes tail to track the named file in a way that accommodates renaming,
removal and creation.
AUTHOR
Written by Paul Rubin, David MacKenzie, Ian Lance Taylor, and Jim Meyering.
REPORTING BUGS
Report tail bugs to bug-coreutils@gnu.org
GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
Report tail translation bugs to <http://translationproject.org/team/>
COPYRIGHT
Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
SEE ALSO
The full documentation for tail is maintained as a Texinfo manual. If the info and tail programs are properly installed at your site, the
command
info coreutils 'tail invocation'
should give you access to the complete manual.
GNU coreutils 8.12.197-032bb September 2011 TAIL(1)