Full Discussion: Real time log file redirect
Top Forums Shell Programming and Scripting Real time log file redirect Post 302137412 by eric_wong_ch on Monday 24th of September 2007 10:12:05 PM
hi Radoulov,

Thank you for your kindly reply^^Smilie

i would like to add the function to archive the log daily, can i do it like this?
but is not working, is it the "IF" statement is out of the "WHILE" loop ?

previousDay=`date --date "1 days ago" "+%Y%m%d"`
nowTime=`date "+%H%M%S"`

nohup tail -f access.log|while IFS= read -r;do
case $REPLY in
(*abcdef*)printf "%s\n" "$REPLY">>abcdef.txt;;
(*123456*)printf "%s\n" "$REPLY">>123456.txt;;
if [ $nowTime == 000000 ]; then
mv abcdef.txt abcdef.txt."$previousDay"
mv 123456.txt 123456.txt."$previousDay"
done &
Test Your Knowledge in Computers #711
Difficulty: Easy
The earliest computing machines had fixed programs.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

log users real time

hi.... how i can configurator a log file on real time....on unix solaris.... thanks a lot.... Best Regards... (3 Replies)
Discussion started by: chanfle
3 Replies

2. Shell Programming and Scripting

Perl or Shell script to read a transaction log in real time

Hello, I have a Apache webserver running on RedHat. Its primary function is a proxy server for users accessing the internet. I have a transaction log that logs every transactions of every users. For users trying to access certain sites/content the transactions goes into a 302 redirect loop and... (2 Replies)
Discussion started by: bruno406
2 Replies

3. Shell Programming and Scripting

shell script to replicate the log files from one location to another in real time

Hi, On the server, we have app log files in this location /app/logs/error.log On the same server, in a real time, we would like to replicate that into /var/ directory. if someone has already done this, please share the script. Thanks in advance. (4 Replies)
Discussion started by: lookinginfo
4 Replies

4. Shell Programming and Scripting

How to tail -f real time file.

How to tail -f real time file. I want to tail file created last time. The server is gen new file Always. . An example file. -rw-r--r-- 1 shinnie tiituck 251M Oct 18 05:39 20111018_00.log -rw-r--r-- 1 shinnie tiituck 251M Oct 18 11:18 20111018_01.log -rw-r--r-- 1 shinnie tiituck... (3 Replies)
Discussion started by: ooilinlove
3 Replies

5. Shell Programming and Scripting

How to tail real time file Generated every hour?

Hi Guys, I am developing a script to monitor log file Generated every hour. When creating a new file scripts not working. My Code . I want to monitor GatewayTransaction.yyyymmdd-hh.log while true; do newdate=$(date '+%Y%m%d') ; nowdate=$(date '+%Y%m%d-%H.log'); tail -f... (7 Replies)
Discussion started by: ooilinlove
7 Replies

6. HP-UX

HP-UX real time audit log writing

Hey all, I have a problem I was hoping to get some help on. So I have my two auditfiles, audfile1 and audfile2 that can be written to, I want to have the text version of them write to an NFS mount that I have set up. So i already know that i can do .secure/etc/audsp audfile1 > //nfsmount/folder/... (5 Replies)
Discussion started by: CleverRiver6
5 Replies

7. Shell Programming and Scripting

[solved] How to see log in real time?

Hi people I have a bash script with a line like this: python example.py >> log & But i can't see anything in the log file while python program is running only if the program ends seems to write the log file. "$ cat log" for example don't show anything until the program ends. Is there... (4 Replies)
Discussion started by: Tieso
4 Replies

8. Shell Programming and Scripting

Archiving or removing few data from log file in real time

Hi, I have a log file that gets updated every second. Currently the size has grown to 20+ GB. I need to have a command/script, that will try to get the actual size of the file and will remove 50% of the data that are in the log file. I don't mind removing the data as the size has grown to huge... (8 Replies)
Discussion started by: Souvik Patra
8 Replies

9. UNIX for Advanced & Expert Users

How to read a fast written log file at Real time speed?

Hello All, I am building a real time parser for a log file in my application. The log file is continuously written at a very fast pace and gets rolled over every 10 minutes. I have measured the speed and observed that around 1000 lines are written to it every second, each line about 30-40... (7 Replies)
Discussion started by: cool.aquarian
7 Replies

10. Shell Programming and Scripting

Log all the commands input by user at real time in /var/log/messages

Below is my script to log all the command input by any user to /var/log/messages. But I cant achieve the desired output that i want. PLease see below. function log2syslog { declare COMMAND COMMAND=$(fc -ln -0) logger -p local1.notice -t bash -i -- "$USER:$COMMAND" } trap... (12 Replies)
Discussion started by: invinzin21
12 Replies
LOG_DB_DAEMON(1)					User Contributed Perl Documentation					  LOG_DB_DAEMON(1)

log_db_daemon - Database logging daemon for Squid SYNOPSIS
log_db_daemon DSN [options] DESCRIPTOIN
This program writes Squid access.log entries to a database. Presently only accepts the squid native format DSN Database DSN encoded as a path. This is sent as the access_log file path. Sample configuration: access_log daemon:/host/database/table/username/password squid to leave a parameter unspecified use a double slash: access_log daemon://database/table/username/password squid Default "DBI:mysql:database=squid" --debug Write debug messages to Squid stderr or cache.log DESCRIPTION
This module exploits the new logfile daemon support available in squid 2.7 and 3.2 to store access log entries in a MySQL database. CONFIGURATION
Squid configuration access_log directive The path to the access log file is used to provide the database connection parameters. access_log daemon:/mysql_host:port/database/table/username/password squid The 'daemon' prefix is mandatory and tells squid that the logfile_daemon helper is to be used instead of the normal file logging. The last parameter tells squid which log format to use when writing lines to the log daemon. Presently squid format is supported. mysql_host:port Host where the mysql server is running. If left empty, 'localhost' is assumed. database Name of the database to connect to. If left empty, 'squid_log' is assumed. table Name of the database table where log lines are stored. If left empty, 'access_log' is assumed. username Username to use when connecting to the database. If left empty, 'squid' is assumed. password Password to use when connecting to the database. If left empty, no password is used. To leave all fields to their default values, you can use a single slash: access_log daemon:/ squid To specify only the database password, which by default is empty, you must leave unspecified all the other parameters by using null strings: access_log daemon://///password squid logfile_daemon directive This is the current way of telling squid where the logfile daemon resides. logfile_daemon /path/to/squid/libexec/logfile-daemon_mysql.pl The script must be copied to the location specified in the directive. Database configuration Let's call the database 'squid_log' and the log table 'access_log'. The username and password for the db connection will be both 'squid'. Database Create the database: CREATE DATABASE squid_log; User Create the user: GRANT INSERT,SELECT,CREATE ON squid_log.* TO 'squid'@'localhost' IDENTIFIED BY 'squid'; FLUSH PRIVILEGES; Note that only CREATE, INSERT and SELECT privileges are granted to the 'squid' user. This ensures that the logfile daemon script cannot change or modify the log entries. Table The Daemon will attempt to initialize this table if none exists when it starts. The table created should look like: CREATE TABLE access_log ( id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY, time_since_epoch DECIMAL(15,3), time_response INTEGER, ip_client CHAR(15), ip_server CHAR(15), http_status_code VARCHAR(10), http_reply_size INTEGER, http_method VARCHAR(20), http_url TEXT, http_username VARCHAR(20), http_mime_type VARCHAR(50), squid_hier_status VARCHAR(20), squid_request_status VARCHAR(20) ); VERSION INFORMATION
This document refers to "log_db_daemon" script version 0.5. The script has been developed and tested in the following environment: squid-2.7 Squid-3.2 mysql 5.0.26 and 5.1 perl 5.8.8 OpenSUSE 10.2 DATA EXTRACTION
Sample queries. Clients accessing the cache SELECT DISTINCT ip_client FROM access_log; Number of request per day SELECT DATE(FROM_UNIXTIME(time_since_epoch)) AS date_day, COUNT(*) AS num_of_requests FROM access_log GROUP BY 1 ORDER BY 1; Request status count To obtain the raw count of each request status: SELECT squid_request_status, COUNT(*) AS n FROM access_log GROUP BY squid_request_status ORDER BY 2 DESC; To calculate the percentage of each request status: SELECT squid_request_status, (COUNT(*)/(SELECT COUNT(*) FROM access_log)*100) AS percentage FROM access_log GROUP BY squid_request_status ORDER BY 2 DESC; To distinguish only between HITs and MISSes: SELECT 'hits', (SELECT COUNT(*) FROM access_log WHERE squid_request_status LIKE '%HIT%') / (SELECT COUNT(*) FROM access_log)*100 AS percentage UNION SELECT 'misses', (SELECT COUNT(*) FROM access_log WHERE squid_request_status LIKE '%MISS%') / (SELECT COUNT(*) FROM access_log)*100 AS pecentage; Response time ranges SELECT '0..500', COUNT(*)/(SELECT COUNT(*) FROM access_log)*100 AS percentage FROM access_log WHERE time_response >= 0 AND time_response < 500 UNION SELECT '500..1000', COUNT(*)/(SELECT COUNT(*) FROM access_log)*100 AS percentage FROM access_log WHERE time_response >= 500 AND time_response < 1000 UNION SELECT '1000..2000', COUNT(*)/(SELECT COUNT(*) FROM access_log)*100 AS percentage FROM access_log WHERE time_response >= 1000 AND time_response < 2000 UNION SELECT '>= 2000', COUNT(*)/(SELECT COUNT(*) FROM access_log)*100 AS percentage FROM access_log WHERE time_response >= 2000; Traffic by mime type SELECT http_mime_type, SUM(http_reply_size) as total_bytes FROM access_log GROUP BY http_mime_type ORDER BY 2 DESC; Traffic by client SELECT ip_client, SUM(http_reply_size) AS total_bytes FROM access_log GROUP BY 1 ORDER BY 2 DESC; Speed issues The MyISAM storage engine is known to be faster than the InnoDB one, so although it doesn't support transactions and referential integrity, it might be more appropriate in this scenario. You might want to append "ENGINE=MYISAM" at the end of the table creation code in the above SQL script. Indexes should be created according to the queries that are more frequently run. The DDL script only creates an implicit index for the primary key column. TODO
Table cleanup This script currently implements only the "L" (i.e. "append a line to the log") command, therefore the log lines are never purged from the table. This approach has an obvious scalability problem. One solution would be to implement e.g. the "rotate log" command in a way that would calculate some summary values, put them in a "summary table" and then delete the lines used to caluclate those values. Similar cleanup code could be implemented in an external script and run periodically independently from squid log commands. Testing This script has only been tested in low-volume scenarios (single client, less than 10 req/s). Tests in high volume environments could reveal performance bottlenecks and bugs. AUTHOR
Marcello Romani, marcello.romani@libero.it Amos Jeffries, amosjeffries@squid-cache.org COPYRIGHT AND LICENSE
Copyright (C) 2008 by Marcello Romani This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.8 or, at your option, any later version of Perl 5 you may have available. perl v5.16.3 2014-06-09 LOG_DB_DAEMON(1)

Featured Tech Videos

All times are GMT -4. The time now is 07:17 AM.
Unix & Linux Forums Content Copyright 1993-2019. All Rights Reserved.
Privacy Policy