I have a situation where I need to write a shell script to continuously monitor a log directory with multiple log files and perform following:
1. Read the latest log file continuously and grep "Success" OR "Failure"
2. As it capture either Success or Failure, it has to perform a database insert.
3. Log files are regularly changing and rotating, so how can we pointing the current logfile seamlessly
For Eg.
Lets say the log directory is /var/log
Log Files:
Code:
app-log0
app-log1
app-log2
...
Now the shell script has to read the latest logfile "app-log2" continuously and capture if "Success" OR "Failure" is coming. If it find it, perform the insert into a database table :
Code:
Insert into log_capture ...
And if a new log file created say "app-log3", it should start reading that log file and perform the same activity.
I know we can use tail -f that can continuously read a file, but I am not sure
1. how can I perform the database insert operation in between, whenever I get Success/Failure
2. How can "tail" pick a new file if the current file exhausted.
Any other suggestion/command is also welcome.
Thank you so much in advance.
Moderator's Comments:
Please use CODE tags as required by forum rules!
Last edited by RudiC; 06-15-2017 at 02:14 PM..
Reason: AddedCODE tags.
I have a directory location where in some logs gets generated whenever some application build is triggered.
I need to send the generated log as an email to end user.
I will get files like abcyyyy_mm_dd_time.log and next file will have different time for same day.
Need to run a cron and take the... (18 Replies)
Hi Guys
I need to have a shell script which reads a log file and insert a part of each line into the database. Some sample lines in the file are as below.
20091112091359 MED_RQACK : user_data=60173054304,100232120,20091112091359,;ask_status=0;ask_reason=OK;msg_id=20091112091319... (5 Replies)
Hi guys ,
I m trying to store the output of vmstat 1 10 into a database.
I have done the necessary homework to connect bash to interact with the database.
Program logic.
1)storing the output of vmstat 1 10 to a file named abc.txt
2)I m applying a filter to vmstat to get desired output... (4 Replies)
Hi
i am looking a way to look at a log file(log.txt) from the last time I've read it.
However after some days the main log file(log.txt) is rename to (log.txt.1).
So now i will have two log files as below.
log.txt.1
log.txt
Now, i have to read the log from the point where i have left... (3 Replies)
Hello All:
We are looking to search through 2000 files with around 21,000 statements where we have to search, replace and insert a pattern based on the following:
1) Parse through the file and check for CREATE MULTISET TABLE or CREATE SET TABLE statements.....and they always end with ON... (5 Replies)
I have a test file that I want to read and insert only certain lines into the
the table based on a filter.
1. Rread the log file 12 Hours back Getdate() -12 Hours
2. Extract the following information on for lines that say "DUMP is
complete"
A. Date
B. Database Name
C.... (2 Replies)
I have a log file that's created daily by this command:
sar -u 300 288 >> /var/log/usage/$(date "+%Y-%m-%d")_$(hostname)_cpu.log
It that contains data like this:
Linux 3.16.0-4-amd64 (myhostname) 08/15/2015 _x86_64_ (1 CPU)
11:34:17 PM CPU %user %nice ... (12 Replies)
Hi all,
I have a moderate size (300 lines) BASH Shell script that performs various tasks on different source reports (CSV files). One of the tasks that it performs, is to use SED to replace 'non-conforming' titles with conformant ones. For example "How to format a RAW Report" needs to become... (3 Replies)
Hi ,
Can anyone help me how do perform below requirement in unix.
Step1:we will receive multiple files weekly with same name(as below) in a folder(In folder we will have other files also def.dat,ghf.dat)
Filenames:
1) abc_20171204_052389.dat
2)abc_20171204_052428.dat
DON'T modify... (23 Replies)
Hi,
I have many files but with only 2 names , I want to load the data of that file into database through sqlldr with single control file. how can i do that ?????
Example:
switch_file
switch_file
billing_file
billing_file
now these files should be loaded into same database but different... (1 Reply)
Discussion started by: niti_sharma
1 Replies
LEARN ABOUT DEBIAN
varnishtop
VARNISHTOP(1)VARNISHTOP(1)NAME
varnishtop - Varnish log entry ranking
SYNOPSIS
varnishtop [-1] [-b] [-C] [-c] [-d] [-f] [-I regex] [-i tag] [-n varnish_name] [-r file] [-V] [-X regex] [-x tag]
DESCRIPTION
The varnishtop utility reads varnishd(1) shared memory logs and presents a continuously updated list of the most commonly occurring log
entries. With suitable filtering using the -I, -i, -X and -x options, it can be used to display a ranking of requested documents, clients,
user agents, or any other information which is recorded in the log.
The following options are available:
-1 Instead of presenting of a continuously updated display, print the statistics once and exit. Implies -d.
-b Include log entries which result from communication with a backend server. If neither -b nor -c is specified, varnishtop acts as if
they both were.
-C Ignore case when matching regular expressions.
-c Include log entries which result from communication with a client. If neither -b nor -c is specified, varnishtop acts as if they
both were.
-d Process old log entries on startup. Normally, varnishtop will only process entries which are written to the log after it starts.
-f Sort and group only on the first field of each log entry. This is useful when displaying e.g. stataddr entries, where the first
field is the client IP address.
-I regex
Include log entries which match the specified regular expression. If neither -I nor -i is specified, all log entries are included.
-i tag Include log entries with the specified tag. If neither -I nor -i is specified, all log entries are included.
-p period
Specifies the number of seconds to measure over, the default is 60 seconds. The first number in the list is the average number of
requests seen over this time period.
-n Specifies the name of the varnishd instance to get logs from. If -n is not specified, the host name is used.
-r file
Read log entries from file instead of shared memory.
-V Display the version number and exit.
-X regex
Exclude log entries which match the specified regular expression.
-x tag Exclude log entries with the specified tag.
EXAMPLES
The following example displays a continuously updated list of the most frequently requested URLs::
varnishtop -i RxURL
The following example displays a continuously updated list of the most commonly used user agents::
varnishtop -i RxHeader -C -I ^User-Agent
SEE ALSO
o varnishd(1)
o varnishhist(1)
o varnishlog(1)
o varnishncsa(1)
o varnishstat(1)HISTORY
The varnishtop utility was originally developed by Poul-Henning Kamp in cooperation with Verdens Gang AS and Varnish Software AS, and later
substantially rewritten by Dag-Erling Smorgrav. This manual page was written by Dag-Erling Smorgrav.
COPYRIGHT
This document is licensed under the same licence as Varnish itself. See LICENCE for details.
o Copyright (c) 2006 Verdens Gang AS
o Copyright (c) 2006-2011 Varnish Software AS
AUTHOR
Dag-Erling Smorgrav
1.0 2010-05-31 VARNISHTOP(1)
06-15-2017
