Sponsored Content
Top Forums UNIX for Advanced & Expert Users How to read a fast written log file at Real time speed? Post 302937623 by cool.aquarian on Friday 6th of March 2015 03:15:47 PM
Old 03-06-2015
Hi there..

I managed to improve the performance by 10-fold for real time reading.

I analysed and found that most time was being taken by
Code:
while IFS= read -r line

I replaced the while loop with awk to read a single logical unit(each about 6000 lines) in one go.

Code:
tail -F --lines=10000000 --retry --max-unchanged-stats=10 "$FILE_NAME" | awk -f awk-script.txt

Now the speed of reading is able to more than keep up with speed of write.
Even when the file gets rolled over, it is able to catch-up with the new file and gives message like:
tail: `<file-name>' has been replaced; following end of new file

Last edited by cool.aquarian; 03-06-2015 at 06:40 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

log users real time

hi.... how i can configurator a log file on real time....on unix solaris.... thanks a lot.... Best Regards... (3 Replies)
Discussion started by: chanfle
3 Replies

2. Shell Programming and Scripting

Real time log file redirect

Hi all, i would like to write the shell script program, it can monitor the access_log "real time" when the access_log writing the line contain "abcdef" the program will be "COPY" this line into a file named "abcdef.txt", do the same thing if the contain "123456" "COPY" it into a file named... (3 Replies)
Discussion started by: eric_wong_ch
3 Replies

3. Shell Programming and Scripting

Perl or Shell script to read a transaction log in real time

Hello, I have a Apache webserver running on RedHat. Its primary function is a proxy server for users accessing the internet. I have a transaction log that logs every transactions of every users. For users trying to access certain sites/content the transactions goes into a 302 redirect loop and... (2 Replies)
Discussion started by: bruno406
2 Replies

4. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies

5. Filesystems, Disks and Memory

data from blktrace: read speed V.S. write speed

I analysed disk performance with blktrace and get some data: read: 8,3 4 2141 2.882115217 3342 Q R 195732187 + 32 8,3 4 2142 2.882116411 3342 G R 195732187 + 32 8,3 4 2144 2.882117647 3342 I R 195732187 + 32 8,3 4 2145 ... (1 Reply)
Discussion started by: W.C.C
1 Replies

6. Shell Programming and Scripting

Read/Search file being written to giving error due to timing issues

The following is a piece of code to rename LOG_FILE_NEW to LOG_FILE once you get a result (either RUNNING or SHUTDOWN) RESULT="" sleep 30 while ; do sleep 10 RESULT=`sed -n '/RUNNING/'p ${LOG_FILE_NEW}` if ; then RESULT=`sed -n '/SHUTTING_DOWN/'p ${LOG_FILE_NEW}` fi done mv... (3 Replies)
Discussion started by: sonorous
3 Replies

7. HP-UX

HP-UX real time audit log writing

Hey all, I have a problem I was hoping to get some help on. So I have my two auditfiles, audfile1 and audfile2 that can be written to, I want to have the text version of them write to an NFS mount that I have set up. So i already know that i can do .secure/etc/audsp audfile1 > //nfsmount/folder/... (5 Replies)
Discussion started by: CleverRiver6
5 Replies

8. Shell Programming and Scripting

[solved] How to see log in real time?

Hi people I have a bash script with a line like this: python example.py >> log & But i can't see anything in the log file while python program is running only if the program ends seems to write the log file. "$ cat log" for example don't show anything until the program ends. Is there... (4 Replies)
Discussion started by: Tieso
4 Replies

9. Shell Programming and Scripting

Archiving or removing few data from log file in real time

Hi, I have a log file that gets updated every second. Currently the size has grown to 20+ GB. I need to have a command/script, that will try to get the actual size of the file and will remove 50% of the data that are in the log file. I don't mind removing the data as the size has grown to huge... (8 Replies)
Discussion started by: Souvik Patra
8 Replies

10. Shell Programming and Scripting

Log all the commands input by user at real time in /var/log/messages

Below is my script to log all the command input by any user to /var/log/messages. But I cant achieve the desired output that i want. PLease see below. function log2syslog { declare COMMAND COMMAND=$(fc -ln -0) logger -p local1.notice -t bash -i -- "$USER:$COMMAND" } trap... (12 Replies)
Discussion started by: invinzin21
12 Replies
IO::Async::Handle(3pm)					User Contributed Perl Documentation				    IO::Async::Handle(3pm)

NAME
"IO::Async::Handle" - event callbacks for a non-blocking file descriptor SYNOPSIS
This class is likely not to be used directly, because subclasses of it exist to handle more specific cases. Here is an example of how it would be used to watch a listening socket for new connections. In real code, it is likely that the "Loop->listen" method would be used instead. use IO::Socket::INET; use IO::Async::Handle; use IO::Async::Loop; my $loop = IO::Async::Loop->new; my $socket = IO::Socket::INET->new( LocalPort => 1234, Listen => 1 ); my $handle = IO::Async::Handle->new( handle => $socket, on_read_ready => sub { my $new_client = $socket->accept; ... }, ); $loop->add( $handle ); For most other uses with sockets, pipes or other filehandles that carry a byte stream, the IO::Async::Stream class is likely to be more suitable. For non-stream sockets, see IO::Async::Socket. DESCRIPTION
This subclass of IO::Async::Notifier allows non-blocking IO on filehandles. It provides event handlers for when the filehandle is read- or write-ready. EVENTS
The following events are invoked, either using subclass methods or CODE references in parameters: on_read_ready Invoked when the read handle becomes ready for reading. on_write_ready Invoked when the write handle becomes ready for writing. on_closed Optional. Invoked when the handle becomes closed. This handler is invoked before the filehandles are closed and the Handle removed from its containing Loop. The "loop" will still return the containing Loop object. PARAMETERS
The following named parameters may be passed to "new" or "configure": read_handle => IO write_handle => IO The reading and writing IO handles. Each must implement the "fileno" method. Primarily used for passing "STDIN" / "STDOUT"; see the SYNOPSIS section of "IO::Async::Stream" for an example. handle => IO The IO handle for both reading and writing; instead of passing each separately as above. Must implement "fileno" method in way that "IO::Handle" does. on_read_ready => CODE on_write_ready => CODE on_closed => CODE CODE references for event handlers. want_readready => BOOL want_writeready => BOOL If present, enable or disable read- or write-ready notification as per the "want_readready" and "want_writeready" methods. It is required that a matching "on_read_ready" or "on_write_ready" are available for any handle that is provided; either passed as a callback CODE reference or as an overridden the method. I.e. if only a "read_handle" is given, then "on_write_ready" can be absent. If "handle" is used as a shortcut, then both read and write-ready callbacks or methods are required. If no IO handles are provided at construction time, the object is still created but will not yet be fully-functional as a Handle. IO handles can be assigned later using the "set_handle" or "set_handles" methods, or by "configure". This may be useful when constructing an object to represent a network connection, before the connect(2) has actually been performed yet. METHODS
$handle->set_handles( %params ) Sets new reading or writing filehandles. Equivalent to calling the "configure" method with the same parameters. $handle->set_handle( $fh ) Shortcut for $handle->configure( handle => $fh ) $handle->close This method calls "close" on the underlying IO handles. This method will then remove the handle from its containing loop. $handle->close_read $handle->close_write Closes the underlying read or write handle, and deconfigures it from the object. Neither of these methods will invoke the "on_closed" event, nor remove the object from the Loop if there is still one open handle in the object. Only when both handles are closed, will "on_closed" be fired, and the object removed. $handle = $handle->read_handle $handle = $handle->write_handle These accessors return the underlying IO handles. $fileno = $handle->read_fileno $fileno = $handle->write_fileno These accessors return the file descriptor numbers of the underlying IO handles. $value = $handle->want_readready $oldvalue = $handle->want_readready( $newvalue ) $value = $handle->want_writeready $oldvalue = $handle->want_writeready( $newvalue ) These are the accessor for the "want_readready" and "want_writeready" properties, which define whether the object is interested in knowing about read- or write-readiness on the underlying file handle. SEE ALSO
o IO::Handle - Supply object methods for I/O handles AUTHOR
Paul Evans <leonerd@leonerd.org.uk> perl v5.14.2 2012-10-24 IO::Async::Handle(3pm)
All times are GMT -4. The time now is 05:25 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy