Read multiple log files and create output file and put the result
OS : Linux 2.6.9-67 - Red Hat Enterprise Linux ES release 4
Looking for a script that reads the following log files that gets generated everynight between 2 - 5am
One exception:
The log file Master_App_20090717.log gets data written into every 30 minutes in the 24hr period. This data is appended into the file and remains there. But each time the extract and transload start it start with the current time stamp in it begin line and end with current time stamp for a successful end or it could be Failed end.
In any case those entries remain in the file until we fix the issue and the next load starts and the results of next load will append to the same log file for that same day and will starts with the same begin line. So we need to make sure that script should not report previous failure that happend 35 mins ago and was reported by the same script 30 minutes ago. So we need to place a check that if it does find a failed end and it is more then 35 minutes ago then ignore it and continue reading the file and then report accordingly.
We will run this script in crontab
One script will run every morning after 5:00 am once all the load is finished or we will run 5 script seperately after a specific time to check the status
The other script will run 5 minutes pass every 30 minutes.
Output file :
We could have 2 output files one for script that runs after 5 am and one for every 30 minute script
or
we could have 5 output files one for each log plus one for 30 minutes script output log so altogether 6 outfiles.
In the output file
First Delete the content of the file if any then
Leave blank if it is successful
Write "ETL Load Failed" if it finds any failed end.
---------- Post updated at 10:33 AM ---------- Previous update was at 10:32 AM ----------
Thank you in advance for your Help as i am desperately seeking a solution. Thank you and all the best
Last edited by vgersh99; 07-20-2009 at 03:42 PM..
Reason: code tags, PLEASE!
Well I have a 3000 lines result log file that contains all the machine data when it does the testing... It has 3 different section that i am intrsted in
1) starting with "20071126 11:11:11 Machine Header 1"
1000 lines...
"End machine header 1"
2) starting with "20071126 12:12:12 Machine... (5 Replies)
Hi all,
I have to generate some report from shell script .We have stacktrace log file which generate hourly basis.
So now my q is that how this shell script will read all stacktrace log file for particlular day and parse accordingly desire output.
Any help or suggestion as i am newbie with... (1 Reply)
Hi All,
I have a to read a file and put the result in one line. The i am reading from contain the data like below.
1 signifies the beging of the new line. (6 Replies)
Howdy Folks,
I have a list that looks like this:
(file2.txt)
AAA
BBB
CCC
DDD
and there are 24 of these short words.
I am matching these patterns to another file with 755795 lines (file1.txt).
I have this code for matching:
awk -v f2=file2.txt '
BEGIN {
while(... (2 Replies)
Hi all,
I have a flat file as below;
470423495|1||TSA-A000073800||1|||1
471423495|1||TSA-A000073800||5|||5
472423495|1||TSA-A000073800||2|||7
473423495|1||TSA-A000073800||3|||3
I like to create a Unix script.
The script have to valuate the last two columns, if the values are... (4 Replies)
Dear All,
I am trying to write a Unix Script which fires a sql query. The output of the sql query gives multiple rows. Each row should be saved in a separate Unix File.
The number of rows of sql output can be variable. I am able save all the rows in one file but in separate files.
Any... (14 Replies)
HI,
I have 2 text files. file1 and file2.
file1.txt (There are no duplicates in this file)
1234
3232
4343
3435
6564
6767
1213
file2.txt
1234,wq,wewe,qwqw
1234,as,dfdf,dfdf
4343,asas,sdds,dsds
6767,asas,fdfd,fdffd
I need to search each number in file1.txt in file2.txt's 1st... (6 Replies)
I am required to create a CSV file reading last 200000 lines form a log file. I have to grep 3 parameters from this log file and write these parameters in the .csv file, with time stamp. This script will be setup in a cron job which will run every 10 minutes. I have written the script but it is... (5 Replies)
I have several problems with my problems: I hope you can help me.
1) the If else statement I am getting an error message. My syntax must be incorrect because the entire statement is throwing an error.
For example in filew.log if these items don't exist Memsize, SASFoundation and also if... (0 Replies)
Hi ,
I'm trying to list the files and output is written to a file. But when I execute the command , the output file is being listed. How to exclude it ?
/tmp
file1.txt
file2.txt
ls -ltr |grep -v '-' | awk print {$9, $5} > output.txt
cat output.txt
file1.txt
file2.txt
output.txt (8 Replies)