need to extract info from log files


 
Thread Tools Search this Thread
Operating Systems AIX need to extract info from log files
# 1  
Old 08-07-2007
need to extract info from log files

hi guys i need to extract information from log files generated by an application.


log file has the following lines for each process..
----------------------------------------------

Fri Aug 03 12:06:43 WST 2007 INFO: Running project PROJECT1
Fri Aug 03 12:06:43 WST 2007 INFO: Source Files For Project:PROJECT1
Fri Aug 03 12:06:43 WST 2007 INFO: [0]=/full/path/to/the/pdf/file/file name 001.pdf
Fri Aug 03 12:07:41 WST 2007 INFO: Done GC=1186114061306
Fri Aug 03 12:07:41 WST 2007 INFO: Returning true from deleteSources()
Fri Aug 03 12:07:41 WST 2007 INFO: Done running project PROJECT1

Fri Aug 03 12:06:43 WST 2007 INFO: Running project PROJECT2
Fri Aug 03 12:06:43 WST 2007 INFO: Source Files For Project:PROJECT2
Fri Aug 03 12:06:43 WST 2007 INFO: [0]=/full/path/to/the/pdf/file/file name 002.pdf
Fri Aug 03 12:07:30 WST 2007 INFO: Running plugin Extract Data from PDF
Fri Aug 03 12:07:30 WST 2007 INFO: Starting runPlugin()
Fri Aug 03 12:07:41 WST 2007 ERROR: Cannot process the file
Fri Aug 03 12:07:41 WST 2007 INFO: Done running project PROJECT2

Fri Aug 03 12:06:43 WST 2007 INFO: Running project PROJECT3
Fri Aug 03 12:06:43 WST 2007 INFO: Source Files For Project:PROJECT3
Fri Aug 03 12:06:43 WST 2007 INFO: [0]=/full/path/to/the/pdf/file/file name 003.pdf
Fri Aug 03 12:07:30 WST 2007 INFO: Running plugin Extract Data from PDF
Fri Aug 03 12:07:30 WST 2007 INFO: Starting runPlugin()
Fri Aug 03 12:07:41 WST 2007 ERROR: File Rollback
Fri Aug 03 12:07:41 WST 2007 INFO: Done running project PROJECT3

Result

file name 001.pdf : Success
file name 002.pdf : Failed
file name 003.pdf : Rollback


i could only extract the lines, but i could not format the output.. hope i will get a solution.

thanx

Last edited by kirantalla; 08-20-2007 at 05:18 AM..
# 2  
Old 08-07-2007
Kirantalla,
Your solution is very involved, as it requires date difference.

A complete solution must consider your times could be in different days,
your days could be in different months and your months could be in different
years, not forgetting leap years.

It requires a considerable amount of time and coding.

Is there an alternative way to display the dates as it appears in the file
without any conversion?

Can the reader visually subtract the difference?
# 3  
Old 08-07-2007
hi shelllife,

thanx for the reply. the extracted info is not for a user but is actually an input file for another process.

the date is not a consideration bcoz the whole process will not take more than a day .. if it crosses one day, the process is killed and restarted. so the extracted time will be within 24 hours.

thx
# 4  
Old 08-10-2007
which line is the filename ? *spaces.pdf or PY_ZHRPY_RCN02-AWSR
how can we match the error ? will it be written as ERROR ?
will there be more than one project in a log file
# 5  
Old 08-12-2007
thx fazliturk for the reply,

ok the file name can be anywhere in between the "Running project <projectname>" and "Done running project <projectname>" and will always a .pdf file and will be the only one name in between the "Running project" and "Done running project"
there will be a lot of projects in the log file .. and the pattern will be

Running project <ProjectName1>
[0]=/machine name/project name/queue name/file name1.pdf
ERROR=<Any Errors here>
Done running project<ProjectName1>
Running project <ProjectName2>
[0]=/machine name/project name/queue name/file name2.pdf
Done running project<ProjectName2>
Running project <ProjectName3>
[0]=/machine name/project name/queue name/file name3.pdf
Done running project<ProjectName3>
Running project <ProjectName4>
[0]=/machine name/project name/queue name/file name4.pdf
ERROR=<Any Errors here>
Done running project<ProjectName4>

file name and directory names will also have spaces

other information is also embedded in between the "Running" and "Done" lines.
# 6  
Old 08-14-2007
I'd suggest converting your dates to epoch time, that makes subtraction much easier, then converting that back to minutes, hours, or what ever else you want.
# 7  
Old 08-14-2007
Here's a script to convert dates into epoch time named "epoch":

Code:
##*******************************************************************************
## Script       epoch
## Purpose      Take date in formatted string as an argument and coverts it
##              into epoch time (seconds since 1/1/1970)
## Usage        ./epoch Aug 25 23:59:59 2007 GMT
##===============================================================================
## History
## 2007/01/15  Big Kahoona   Creation of script (some code from the internet)
##*******************************************************************************

year=$4
month=$1
day=$2
hour=$(echo $3 | cut -d: -f1)
min=$(echo $3 | cut -d: -f2)
sec=$(echo $3 | cut -d: -f3)
tz=$5

# Convert month to number
if [ $month = "Jan" ]; then
        month=1
elif [ $month = "Feb" ]; then
        month=2
elif [ $month = "Mar" ]; then
        month=3
elif [ $month = "Apr" ]; then
        month=4
elif [ $month = "May" ]; then
        month=5
elif [ $month = "Jun" ]; then
        month=6
elif [ $month = "Jul" ]; then
        month=7
elif [ $month = "Aug" ]; then
        month=8
elif [ $month = "Sep" ]; then
        month=9
elif [ $month = "Oct" ]; then
        month=10
elif [ $month = "Nov" ]; then
        month=11
elif [ $month = "Dec" ]; then
        month=12
fi

# leap days in past years
leapdays=$(( (year - 1969)/4 ))

# Is this year a leap year?
leap=$(( year % 4 == 0 ))

# Days in each month this year.
mdays[1]=31
mdays[2]=$((28+leap))
mdays[3]=31
mdays[4]=30
mdays[5]=31
mdays[6]=30
mdays[7]=31
mdays[8]=31
mdays[9]=30
mdays[10]=31
mdays[11]=30
mdays[12]=31

# days since the epoch, not counting earlier months this year
daycount=$(( (year - 1970) * 365 + leapdays + day - 1))

# Step through earlier months this year and add the days
m=$((month - 1))
while [ $m -ge 1 ]; do
        #echo "month=$m"
        daycount=$((daycount+${mdays[$m]} ))
        m=$((m-1))
done

# Now the seconds
epoch=$(( ( (daycount * 24 + hour) * 60 + min) * 60 + sec ))

# Add the time zones that apply to you
case "$tz" in
        #GMT) epoch=$((epoch + 0));;
        #EST) epoch=$((epoch + 18000));;
        CST) epoch=$((epoch + 21600));;
        #GMT) epoch=$((epoch + 21600));;
        #*)   epoch="ERROR: unrecognized on time zone";;
esac
echo "${epoch}"

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare two files and extract info

Hello, I have two files which look like this File 1 Name test1 status P Gene1 0.00236753 1 1.00E-01 Gene2 0.134187 2 2.00E-01 Gene3 0.000608716 2 3.00E-01 Gene4 0.0016234 1 4.00E-01 Gene5 0.000665868 2 5.00E-01and file 2 No Pos ... (2 Replies)
Discussion started by: nans
2 Replies

2. Shell Programming and Scripting

HELP: Shell Script to read a Log file line by line and extract Info based on KEYWORDS matching

I have a LOG file which looks like this Import started at: Mon Jul 23 02:13:01 EDT 2012 Initialization completed in 2.146 seconds. -------------------------------------------------------------------------------- -- Import summary for Import item: PolicyInformation... (8 Replies)
Discussion started by: biztank
8 Replies

3. Shell Programming and Scripting

Extract portion of log info based on specific word

Hi Gurus, I'm using HP-UX B.11.23 operating system. I've been trying to extract a specific wording for example: "A tool used by tp produced warnings" from my below log data, but could not find a way to solve it. My intention is, if the log contain the word: "A tool used by tp produced... (9 Replies)
Discussion started by: superHonda123
9 Replies

4. Shell Programming and Scripting

How to extract log info based on last month?

Hi Gurus I'm using HPUX B.11.23 ia64, the below sample log data was extracted from HPUX commnad: last -R. Sample data: root pts/ta userpc Wed Aug 11 09:46 - 20:21 (10:35) root pts/ta userpc Wed Aug 11 09:44 - 20:10 (10:34) root pts/ta userpc Wed Aug 11... (4 Replies)
Discussion started by: superHonda123
4 Replies

5. Shell Programming and Scripting

Extract info from log file and compute using time date stamp

Looking for a shell script or a simple perl script . I am new to scripting and not very good at it . I have 2 directories . One of them holds a text file with list of files in it and the second one is a daily log which shows the file completion time. I need to co-relate both and make a report. ... (0 Replies)
Discussion started by: breez_drew
0 Replies

6. Shell Programming and Scripting

Extract info from csv

I have some input file, which contains some lines which are comma separated. Eg. a,b,id=999],d d,f,g,id=345],x x,y,x,s,id=677],y I run a loop to read the lines one by one. What i want is to extract the value on the right of id=. I cannot do it by Awk, since the column number is not fixed.... (5 Replies)
Discussion started by: indianjassi
5 Replies

7. IP Networking

How to extract NIC and other info ,given Ip

Hi all, I am working on a networking project that requires me to find out the NIC on that particular machine and many more things. Now Given the IP and the subnet. I would like to know how we can extract such informations? I am talking in exclusivity to Solaris boxes!! The required... (6 Replies)
Discussion started by: wrapster
6 Replies

8. AIX

Extract info

Anyone have a better idea to automate extraction of info like ... "uname" "ifconfig" "ps efl" "netstat -ao" etc. from several hundred aix, solaris, red hat boxes? without logging into each box and manually performing these tasks and dumping them to individual files? thanks for any input (1 Reply)
Discussion started by: chm0dvii
1 Replies

9. UNIX for Advanced & Expert Users

How to find all the log files under the root directory -- Need Info

Hi I need to find all the log files under the root directory and delete them if necessary, if any one has a sample script who can share with me. Thanks (2 Replies)
Discussion started by: gkrishnag
2 Replies

10. UNIX for Dummies Questions & Answers

using cut to extract info

a simple question, how can i use cut (after using grep) to extract the last four digits on a line. so say i had a string http://blabla:9020, how would I extract the port?? -Fez (4 Replies)
Discussion started by: hafhaq
4 Replies
Login or Register to Ask a Question