Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Read latest log files and perform database insert

Shell Programming and Scripting


Tags
tail

Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 06-15-2017
rish_max rish_max is offline
Registered User
 
Join Date: Feb 2009
Last Activity: 15 June 2017, 12:16 PM EDT
Posts: 12
Thanks: 0
Thanked 0 Times in 0 Posts
Read latest log files and perform database insert

Hi Experts,

I have a situation where I need to write a shell script to continuously monitor a log directory with multiple log files and perform following:

1. Read the latest log file continuously and grep "Success" OR "Failure"
2. As it capture either Success or Failure, it has to perform a database insert.
3. Log files are regularly changing and rotating, so how can we pointing the current logfile seamlessly

For Eg.
Lets say the log directory is /var/log
Log Files:

Code:
app-log0
app-log1
app-log2
...

Now the shell script has to read the latest logfile "app-log2" continuously and capture if "Success" OR "Failure" is coming. If it find it, perform the insert into a database table :


Code:
Insert into log_capture ...

And if a new log file created say "app-log3", it should start reading that log file and perform the same activity.

I know we can use tail -f that can continuously read a file, but I am not sure
1. how can I perform the database insert operation in between, whenever I get Success/Failure
2. How can "tail" pick a new file if the current file exhausted.

Any other suggestion/command is also welcome.

Thank you so much in advance.

Moderator's Comments:
Read latest log files and perform database insert Please use CODE tags as required by forum rules!

Last edited by RudiC; 06-15-2017 at 01:14 PM.. Reason: AddedCODE tags.
Sponsored Links
    #2  
Old Unix and Linux 06-15-2017
RudiC RudiC is online now Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 22 August 2017, 11:08 AM EDT
Location: Aachen, Germany
Posts: 11,189
Thanks: 287
Thanked 3,451 Times in 3,179 Posts
That "reading continuously" doesn't sound feasible, and I don't see the advantage of doing so. Why don't you run a (set of) script periodically, check for old and new log files, and then insert a bunch of records into your DB?
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
BASH script to read external file to perform text replacements? richardsantink Shell Programming and Scripting 3 06-13-2016 11:24 AM
How to read a text file line by line and insert into a database table? JolietJake Shell Programming and Scripting 2 10-09-2014 11:23 AM
Parse through ~21,000 Database DDL statements -- Fastest way to perform search, replace and insert madhunk Shell Programming and Scripting 5 05-11-2013 04:33 PM
Shell Script to read a tab delimited file and perform simple tasks jsmith6932 Homework & Coursework Questions 6 12-18-2012 06:17 PM
To Read a File and Insert a part of the lines into the database Somanadh Shell Programming and Scripting 5 11-16-2009 06:04 AM



All times are GMT -4. The time now is 11:09 AM.