How to read a fast written log file at Real time speed?
Hello All,
I am building a real time parser for a log file in my application.
The log file is continuously written at a very fast pace and gets rolled over every 10 minutes.
I have measured the speed and observed that around 1000 lines are written to it every second, each line about 30-40 characters.
I have tried using tail -F, but it always lags behind the speed at which the file is being written.
Could you suggest anything else I can use to read the file line by line quickly at Realtime speed ?
Hi all,
i would like to write the shell script program, it can monitor the access_log "real time"
when the access_log writing the line contain "abcdef" the program will be "COPY" this line into a file named "abcdef.txt", do the same thing if the contain "123456" "COPY" it into a file named... (3 Replies)
Hello,
I have a Apache webserver running on RedHat. Its primary function is a proxy server for users accessing the internet. I have a transaction log that logs every transactions of every users. For users trying to access certain sites/content the transactions goes into a 302 redirect loop and... (2 Replies)
Hi,
I'm trying to figure out the best solution to the following problem, and I'm not
yet that much experienced like you. :-)
Basically I have to read a fairly large file, composed of "messages" , in order
to display all of them through an user interface (made with QT).
The messages that... (3 Replies)
I analysed disk performance with blktrace and get some data:
read:
8,3 4 2141 2.882115217 3342 Q R 195732187 + 32
8,3 4 2142 2.882116411 3342 G R 195732187 + 32
8,3 4 2144 2.882117647 3342 I R 195732187 + 32
8,3 4 2145 ... (1 Reply)
The following is a piece of code to rename LOG_FILE_NEW to LOG_FILE once you get a result (either RUNNING or SHUTDOWN)
RESULT=""
sleep 30
while ; do
sleep 10
RESULT=`sed -n '/RUNNING/'p ${LOG_FILE_NEW}`
if ; then
RESULT=`sed -n '/SHUTTING_DOWN/'p ${LOG_FILE_NEW}`
fi
done
mv... (3 Replies)
Hey all, I have a problem I was hoping to get some help on. So I have my two auditfiles, audfile1 and audfile2 that can be written to, I want to have the text version of them write to an NFS mount that I have set up. So i already know that i can do .secure/etc/audsp audfile1 > //nfsmount/folder/... (5 Replies)
Hi people
I have a bash script with a line like this:
python example.py >> log &
But i can't see anything in the log file while python program is running only if the program ends seems to write the log file.
"$ cat log" for example don't show anything until the program ends.
Is there... (4 Replies)
Hi,
I have a log file that gets updated every second. Currently the size has grown to 20+ GB. I need to have a command/script, that will try to get the actual size of the file and will remove 50% of the data that are in the log file. I don't mind removing the data as the size has grown to huge... (8 Replies)
Below is my script to log all the command input by any user to /var/log/messages. But I cant achieve the desired output that i want. PLease see below.
function log2syslog
{
declare COMMAND
COMMAND=$(fc -ln -0)
logger -p local1.notice -t bash -i -- "$USER:$COMMAND"
}
trap... (12 Replies)
Discussion started by: invinzin21
12 Replies
LEARN ABOUT FREEBSD
tail
TAIL(1) BSD General Commands Manual TAIL(1)NAME
tail -- display the last part of a file
SYNOPSIS
tail [-F | -f | -r] [-q] [-b number | -c number | -n number] [file ...]
DESCRIPTION
The tail utility displays the contents of file or, by default, its standard input, to the standard output.
The display begins at a byte, line or 512-byte block location in the input. Numbers having a leading plus ('+') sign are relative to the
beginning of the input, for example, ``-c +2'' starts the display at the second byte of the input. Numbers having a leading minus ('-') sign
or no explicit sign are relative to the end of the input, for example, ``-n 2'' displays the last two lines of the input. The default start-
ing location is ``-n 10'', or the last 10 lines of the input.
The options are as follows:
-b number
The location is number 512-byte blocks.
-c number
The location is number bytes.
-f The -f option causes tail to not stop when end of file is reached, but rather to wait for additional data to be appended to the
input. The -f option is ignored if the standard input is a pipe, but not if it is a FIFO.
-F The -F option implies the -f option, but tail will also check to see if the file being followed has been renamed or rotated. The
file is closed and reopened when tail detects that the filename being read from has a new inode number.
If the file being followed does not (yet) exist or if it is removed, tail will keep looking and will display the file from the begin-
ning if and when it is created.
The -F option is the same as the -f option if reading from standard input rather than a file.
-n number
The location is number lines.
-q Suppresses printing of headers when multiple files are being examined.
-r The -r option causes the input to be displayed in reverse order, by line. Additionally, this option changes the meaning of the -b,
-c and -n options. When the -r option is specified, these options specify the number of bytes, lines or 512-byte blocks to display,
instead of the bytes, lines or blocks from the beginning or end of the input from which to begin the display. The default for the -r
option is to display all of the input.
If more than a single file is specified, each file is preceded by a header consisting of the string ``==> XXX <=='' where XXX is the name of
the file unless -q flag is specified.
EXIT STATUS
The tail utility exits 0 on success, and >0 if an error occurs.
EXAMPLES
To display the last 500 lines of the file foo:
$ tail -n 500 foo
Keep /var/log/messages open, displaying to the standard output anything appended to the file:
$ tail -f /var/log/messages
SEE ALSO cat(1), head(1), sed(1)STANDARDS
The tail utility is expected to be a superset of the IEEE Std 1003.2-1992 (``POSIX.2'') specification. In particular, the -F, -b and -r
options are extensions to that standard.
The historic command line syntax of tail is supported by this implementation. The only difference between this implementation and historic
versions of tail, once the command line syntax translation has been done, is that the -b, -c and -n options modify the -r option, i.e., ``-r
-c 4'' displays the last 4 characters of the last line of the input, while the historic tail (using the historic syntax ``-4cr'') would
ignore the -c option and display the last 4 lines of the input.
HISTORY
A tail command appeared in PWB UNIX.
BSD March 16, 2013 BSD