Extract a value from log


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract a value from log
# 1  
Old 06-25-2011
Extract a value from log

Hi,

I have a log file:

130.37.129.78 - - [25/Jun/2011:12:35:07 +0000] "GET /ganglia/graph.php?m=load_one&z=small&c=cluster1&h=xen02.das3.cs.vu.nl&l=e2ecff&v=0.00&x=0.22&n=0&r=hour&su=1 &st=1309005296 HTTP/1.0" 200 6818 "http://130.37.197.12/ganglia/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20100101 Firefox/5.0" request:0.265

I would like to extract the value next to the string request: from the file. Could someone help me with this. Smilie
# 2  
Old 06-25-2011
Code:
awk -F: '{print $NF}' logfile

This User Gave Thanks to ctsgnb For This Post:
# 3  
Old 06-25-2011
Thank you, but the thing is I am extracting the value line by line from the log.

/***************************************/
cat $LOG | while read LINE
do
# Obtain the value time from each request
REQUEST=?

/******************************************/

Could you tell me how to modify the above command for this purpose.
# 4  
Old 06-25-2011
please post an example of your input as well as an example of the output you need

---------- Post updated at 02:39 PM ---------- Previous update was at 02:36 PM ----------

Code:
$ cat logfile
130.37.129.78 - - [25/Jun/2011:12:35:07 +0000] "GET /ganglia/graph.php?m=load_one&z=small&c=cluster1&h=xen02.das3.cs.vu.nl&l=e2ecff&v=0.00&x=0.22&n=0&r=hour&su=1 &st=1309005296 HTTP/1.0" 200 6818 "http://130.37.197.12/ganglia/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20100101 Firefox/5.0" request:0.265
130.37.129.78 - - [25/Jun/2011:12:35:07 +0000] "GET /ganglia/graph.php?m=load_one&z=small&c=cluster1&h=xen02.das3.cs.vu.nl&l=e2ecff&v=0.00&x=0.22&n=0&r=hour&su=1 &st=1309005296 HTTP/1.0" 200 6818 "http://130.37.197.12/ganglia/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20100101 Firefox/5.0" request:0.857
130.37.129.78 - - [25/Jun/2011:12:35:07 +0000] "GET /ganglia/graph.php?m=load_one&z=small&c=cluster1&h=xen02.das3.cs.vu.nl&l=e2ecff&v=0.00&x=0.22&n=0&r=hour&su=1 &st=1309005296 HTTP/1.0" 200 6818 "http://130.37.197.12/ganglia/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20100101 Firefox/5.0" request:0.376
$ awk -F: '{print $NF}' logfile
0.265
0.857
0.376
$

# 5  
Old 06-25-2011
Hi,

The input is a nginx log file with many requests of this syntax:

130.37.129.78 - - [25/Jun/2011:12:55:10 +0000] "GET /ganglia/graph.php?m=load_one&z=small&c=cluster1&h=xen02.das3.cs.vu.nl&l=e2ecff&v=0.00&x=0.32&n=0&r=hour&su=1 &st=1309006507 HTTP/1.0" 200 7085 "http://130.37.197.12/ganglia/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20100101 Firefox/5.0" request:0.252

I am parsing this log file line by line and extracting the request: value from each line and computing their average. Could you help me as to how to extract the value after the "request:" string from each line.
# 6  
Old 06-25-2011
With the same example file as above :

Code:
$ awk -F: '{++x;s+=$NF;print $NF}END{print "average=" s/x}' logfile
0.265
0.857
0.376
average=0.499333

---------- Post updated at 02:47 PM ---------- Previous update was at 02:43 PM ----------

I need to know how your logfile looks like .

Does it only contain lines that ends with a string "request:<value>" ?

If not then please post a representative example of how does your logfile look like (with all the lines, not only those that have the "resquest:<value>", so we can then be more accurate in the line selection before extracting the value and calculating the average
This User Gave Thanks to ctsgnb For This Post:
# 7  
Old 06-25-2011
Wow, that is perfect. Just what I needed in one line Smilie

One more thing, is there a way to store the value of average in another variable, since I am having a shell script which runs the above script inside it, so I would like to store the average value into a variable.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract data from log

I have logs in format ####<01-Mar-2015 03:48:18 o'clock GMT> <info> ####<01-Mar-2015 03:48:20 o'clock GMT> <info> ####<01-Mar-2015 03:48:30 o'clock GMT> <info> ####<01-Mar-2015 03:48:39 o'clock GMT> <info> I am looking out for a script which can extract data of last 15 minutes from the... (5 Replies)
Discussion started by: oberoi1403
5 Replies

2. UNIX for Dummies Questions & Answers

Extract Numbers from a log file

Hi, I am trying to grep/extract the number list from this log file, can I get some help on this. I can grep the word 'href' to see the numbers, but it is resulting with the complete line. Content of my file: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> <html> <head>... (4 Replies)
Discussion started by: Sajjadmehdi
4 Replies

3. Shell Programming and Scripting

Extract IP Address from Log File

I have a log file with several IP addresses in it:- 2012-12-06 16:05:05,885 NOTICE **SNMP** Alarm was created: (LicenseClientRejected) Client Remote Peer /125.111.64.125:2573 was Rejected Property=/125.111.64.125:2573, Client Remote Peer /125.111.64.125:2573 was Rejected **SNMP**.... (5 Replies)
Discussion started by: BHM
5 Replies

4. Shell Programming and Scripting

Extract all ERROR's on LOG File

Hello, As I am still in trouble to easly extract all error's by date in log file, I am deply requesting your help ... Here is an example of one error (the log file may contain a huge amont of error's, I must be able to extract all error for a day in parameter): ***** L1 FILE... (4 Replies)
Discussion started by: Aswex
4 Replies

5. Shell Programming and Scripting

Extract various information from a log file

Hye ShamRock If you can help me with this difficult task for me then it will save my day Logs : ================================================================================================================== ... (4 Replies)
Discussion started by: SilvesterJ
4 Replies

6. Shell Programming and Scripting

How to extract certain part of log file?

Hi there, I'm having some problem with UNIX scripting (ksh), perhaps somebody can help me out? For example: ------------ Sample content of my log file (text file): -------------------------------------- File1: .... info_1 ... info_2 ... info_3 ... File2: .... info_1 ... info_2 ...... (10 Replies)
Discussion started by: superHonda123
10 Replies

7. Shell Programming and Scripting

How to extract last week data from a log

HI, I have a log file (sync.log) in the following format. 05-14 05:34:56 INFO Hxxx Start: Id:xxx 05-14 05:45:32 INFO Hxxx End: Id:xxxx 05-19 11:23:21 INFO Hxxx Start: Id:xxx 05-19 11:34:45 INFO Hxxx End: Id:xxxx 05-20 02:09:47 INFO Hxxx Start: Id:xxx 05-20 02:36:03 INFO Hxxx End: Id:xxx... (5 Replies)
Discussion started by: praveenvi
5 Replies

8. Shell Programming and Scripting

Help: Log data extract

So I got this old webapp doing some strange format with its logging and I need to extract only specific parts. Basically the format of the log goes: Date:<space>timestamp User:<space>userid AppID:<space>applicationid Duration:<space>time(ms) <line> <line> Date:<space>timestamp... (1 Reply)
Discussion started by: kenm0j0
1 Replies

9. AIX

need to extract info from log files

hi guys i need to extract information from log files generated by an application. log file has the following lines for each process.. ---------------------------------------------- Fri Aug 03 12:06:43 WST 2007 INFO: Running project PROJECT1 Fri Aug 03 12:06:43 WST 2007 INFO: Source Files... (7 Replies)
Discussion started by: kirantalla
7 Replies

10. Shell Programming and Scripting

extract a portion of log file

hello, I want to grep the log file according to time and get the portion of log from one particular time to other. I can grep for individual lines by time but how should I print lines continuously from given start time till end till given end time. Appreciate your ideas, Thanks chandra (8 Replies)
Discussion started by: chandra004
8 Replies
Login or Register to Ask a Question