Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Taking a specific value from a log file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Taking a specific value from a log file
# 1  
Old 03-15-2013
Taking a specific value from a log file
Dear community,
I've a file contaning some logs like:
Code:
185413.854: [GC 3938735K->3100312K(4089472K), 0.0124750 secs]
185456.748: [GC 3897187K(4089472K), 0.5681710 secs]
185457.631: [GC 3940519K->3101353K(4089472K), 0.0107800 secs]
185467.213: [GC 3873521K(4089472K), 1.1164290 secs]
185468.913: [GC 3940265K->3102224K(4089472K), 0.0114570 secs]
185472.378: [GC 3369975K(4089472K), 0.4640150 secs]
185479.944: [GC 3940134K->3101807K(4089472K), 0.0154030 secs]
185482.828: [GC 3338284K(4089472K), 0.0680050 secs]
185486.855: [GC 3660673K(4089472K), 0.9342110 secs]
185490.946: [GC 3940426K->3101326K(4089472K), 0.0120510 secs]
185497.580: [GC 3649545K(4089472K), 0.7692390 secs]
185501.771: [GC 3940088K->3101499K(4089472K), 0.0123540 secs]
185501.787: [GC 3101727K(4089472K), 0.0061810 secs]
185511.343: [GC 3851751K(4089472K), 1.1011740 secs]
185513.458: [GC 3940411K->3101902K(4089472K), 0.0117240 secs]
185501.787: [GC 3101727K(4089472K), 0.0061810 secs]
185516.603: [GC 3361385K(4089472K), 0.4643180 secs]

What I need to do is extract the value in red/bold from the last line with "->". I tried:
Code:
# tail gclog.txt | grep "-" | awk -F"[>K]" '/->/{print $3}'
3101807
3101326
3101499
3101902

But I need only 3101902

Hope is clear, thanks.
Lucas
# 2  
Old 03-15-2013
U need only this 3101902 because it is the last entry???
# 3  
Old 03-15-2013
Code:
$ awk 'match ($0, "->") {gsub (/^.*>|[^0-9].*$/, ""); TMP=$0} END{print TMP}' file
3101902

This User Gave Thanks to RudiC For This Post:
Lord Spectre
# 4  
Old 03-15-2013
Depending on your OS you could also give a try to :
Code:
tac gclog.txt | sed '/-/!d;s/.*>//;s/K.*//;q'

or
Code:
tail -r gclog.txt | sed '/-/!d;s/.*>//;s/K.*//;q'

Should be faster because starting from end of file

Note that using the tail command without additional options you assume that the line that you want to extract is in the output of your tail command which may not be the case if the number of subsequent "uninteresting" lines exceed 9.
This User Gave Thanks to ctsgnb For This Post:
Lord Spectre
# 5  
Old 03-15-2013
Both of them work perfect! Thanks RudiC and ctsgnb! Smilie
Code:
# awk 'match ($0, "->") {gsub (/^.*>|[^0-9].*$/, ""); TMP=$0} END{print TMP}' $lastgc
or
# tac $lastgc | sed '/-/!d;s/.*>//;s/K.*//;q'

Btw, I believe the second one is faster, am I wrong? I'm asking that because the gclog sometime becomes huge!
# 6  
Old 03-15-2013
Quote:
Originally Posted by Lord Spectre
. . . Btw, I believe the second one is faster, am I wrong? I'm asking that because the gclog sometime becomes huge!
That would be an interesting question. awk will need to go through the entire file to the end to be sure it picks the last number wanted. With tac, you could take the first one and quit. BUT - tac still needs to read the entire file to find the last, before last etc. lines and present them to sed.

And, if there's many a non matching trailing line, this may become slower and slower.

Can you time the solutions with a couple of input files and report back?

EDIT: I created a huge file and did the timing:
Code:
$ time   awk 'match ($0, "->") {gsub (/^.*>|[^0-9].*$/, ""); TMP=$0} END{print TMP}' file 
3101902

real    0m0.081s
user    0m0.072s
sys     0m0.008s
$ time tac file | sed '/-/!d;s/.*>//;s/K.*//;q'
3101902

real    0m0.008s
user    0m0.000s
sys     0m0.000s

Very evident, one order of magnitude difference in execution times. In fact, with strace you can see, that awk opens the file and reads reads reads
Code:
open("file", O_RDONLY)                  = 3   
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fffeff21e78) = -1 ENOTTY (Inappropriate ioctl for device)
read(3, "185413.854: [GC 3938735K->310031"..., 4096) = 4096 
read(3, "570 secs]\n185472.378: [GC 336997"..., 4044) = 4044
read(3, "185490.946: [GC 3940426K->310132"..., 4096) = 4096 
read(3, "185513.458: [GC 3940411K->310190"..., 4096) = 4096

, while tac does an lseek (... SEEK_END), (... SEEK_SET) and starts from the rear:
Code:
open("file", O_RDONLY)                  = 3
lseek(3, 0, SEEK_END)                   = 5495040
. . . 
lseek(3, 5488640, SEEK_SET)             = 5488640
read(3, "14570 secs]\n185472.378: [GC 3369"..., 6400) = 6400
. . .

EDIT 2:
Code:
tac file | awk 'match ($0, "->") {gsub (/^.*>|[^0-9].*$/, ""); print; exit}'

is as fast!
Last edited by RudiC; 03-15-2013 at 12:24 PM..
This User Gave Thanks to RudiC For This Post:
Lord Spectre
# 7  
Old 03-15-2013
Great RudiC, I'll use tac for sure.
And thanks again for your feedback!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep a log file starting from a specific time to the end of file

I have a log file which have a date and time at the start of every line. I need to search the log file starting from a specific time to the end of file. For example: Starting point: July 29 2018 21:00:00 End point : end of file My concern is what if the pattern of `July 29 2018 21:00:00`... (3 Replies)
Discussion started by: erin00
3 Replies

2. UNIX for Beginners Questions & Answers

Selecting specific variable in log file

Hi there I am trying to look for a specific word in the log file and I am aware this can be done by grep for example. As there will be multiple entries for this I want to grep the last one to enter the log... how would I go about this - would I have to use tail? Thanks in advance Alex (4 Replies)
Discussion started by: simpsa27
4 Replies

3. UNIX for Beginners Questions & Answers

Viewing a specific timeframe of a log file

Hi guys Done a bit of research online but can't seem to figure it out, is there anyway of grepping or using sed to view a specific time period of a log file. I am trying to view a log file for Saturday 22nd April between 08:00 - 12:00 I saw this command online and tried but doesn't seem to... (10 Replies)
Discussion started by: simpsa27
10 Replies

4. Shell Programming and Scripting

Check specific content from log file

Hi all, i have a logfile which is continuously being updated. I have built a script to check for a specific content and if it is found, it sends a string into a file. Here's the current script: #!/bin/bash logfile=/opt/jboss-eap-6.3/standalone/log/server.log tail -fn0 $logfile | \... (7 Replies)
Discussion started by: nms
7 Replies

5. Shell Programming and Scripting

Need specific columns in a log file as excel.

Hi All... I am in need of few columns from a log file.. in .xls file... below is what i have tried. my log file has 16 colums with " ; " as delimiter, but i need randomn columns 1 2 3 4 5 6 10 11 16 in an excel. I tried to awk the columns with delimiter ; and it worked, below is the log... (5 Replies)
Discussion started by: nanz143
5 Replies

6. UNIX for Dummies Questions & Answers

Search for a specific String in a log file for a specific date range

Hi, I have log file which rolls out every second which is as this. HttpGenRequest - -<!--OXi dbPublish--> <created="2014-03-24 23:45:37" lastMsgId="" requestTime="0.0333"> <response request="getOutcomeDetails" code="114" message="Request found no matching data" debug="" provider="undefined"/>... (3 Replies)
Discussion started by: karthikprakash
3 Replies

7. Shell Programming and Scripting

Write out specific data from log to a new file

I got a huge log in zipped files, i need to write out lines by specific data and if the line with the same contains XML message with the same sessionID will be written to the file to. The log structure: 2013-08-16 16:31:06,810 ( 122: rogate) INFO - UId:10453, GId:5422: new... (16 Replies)
Discussion started by: batka
16 Replies

8. Shell Programming and Scripting

Extract Specific pattern - log file

Hello everyone, I am on AIX (6.1). I can only use shell (ksh) script. I can't do this on my own, so will do my best to explain my needs.I also do not know what is the best idea to make it work, so here is what I am thinking, but I may wrong. I need help to extract info on... (3 Replies)
Discussion started by: Aswex
3 Replies

9. Shell Programming and Scripting

How to read a specific value from a Log file?

Hi, I have a .log file in which it has many values. But i need some specific values. How it can be done using Shell Script. Please explain in detail. Thankx in advance. Sathish D V. (8 Replies)
Discussion started by: cooolthud
8 Replies

10. Shell Programming and Scripting

read specific text from a log file

Hi guys I need to retrieve the values in BOLD that I have mentioned in the below log file. I want to store those values in a variable, preferably the same name as the column name in the log file. if you paste the below mentioned log file in a notepad and remove the word wrap.. u will get a... (4 Replies)
Discussion started by: ragha81
4 Replies
Login or Register to Ask a Question