Unix/Linux Go Back    

Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Read latest log files and perform database insert

Shell Programming and Scripting


Thread Tools Search this Thread Display Modes
Old Unix and Linux 06-15-2017   -   Original Discussion by rish_max
rish_max rish_max is offline
Registered User
Join Date: Feb 2009
Last Activity: 15 June 2017, 12:16 PM EDT
Posts: 12
Thanks: 0
Thanked 0 Times in 0 Posts
Read latest log files and perform database insert

Hi Experts,

I have a situation where I need to write a shell script to continuously monitor a log directory with multiple log files and perform following:

1. Read the latest log file continuously and grep "Success" OR "Failure"
2. As it capture either Success or Failure, it has to perform a database insert.
3. Log files are regularly changing and rotating, so how can we pointing the current logfile seamlessly

For Eg.
Lets say the log directory is /var/log
Log Files:


Now the shell script has to read the latest logfile "app-log2" continuously and capture if "Success" OR "Failure" is coming. If it find it, perform the insert into a database table :

Insert into log_capture ...

And if a new log file created say "app-log3", it should start reading that log file and perform the same activity.

I know we can use tail -f that can continuously read a file, but I am not sure
1. how can I perform the database insert operation in between, whenever I get Success/Failure
2. How can "tail" pick a new file if the current file exhausted.

Any other suggestion/command is also welcome.

Thank you so much in advance.

Moderator's Comments:
Read latest log files and perform database insert Please use CODE tags as required by forum rules!

Last edited by RudiC; 06-15-2017 at 02:14 PM.. Reason: AddedCODE tags.
Sponsored Links
Old Unix and Linux 06-15-2017   -   Original Discussion by rish_max
RudiC RudiC is online now Forum Staff  
Join Date: Jul 2012
Last Activity: 23 November 2017, 1:17 PM EST
Location: Aachen, Germany
Posts: 11,652
Thanks: 323
Thanked 3,620 Times in 3,325 Posts
That "reading continuously" doesn't sound feasible, and I don't see the advantage of doing so. Why don't you run a (set of) script periodically, check for old and new log files, and then insert a bunch of records into your DB?
Sponsored Links

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
BASH script to read external file to perform text replacements? richardsantink Shell Programming and Scripting 3 06-13-2016 12:24 PM
How to read a text file line by line and insert into a database table? JolietJake Shell Programming and Scripting 2 10-09-2014 12:23 PM
Parse through ~21,000 Database DDL statements -- Fastest way to perform search, replace and insert madhunk Shell Programming and Scripting 5 05-11-2013 05:33 PM
Shell Script to read a tab delimited file and perform simple tasks jsmith6932 Homework & Coursework Questions 6 12-18-2012 07:17 PM
To Read a File and Insert a part of the lines into the database Somanadh Shell Programming and Scripting 5 11-16-2009 07:04 AM

All times are GMT -4. The time now is 02:28 PM.