Visit Our UNIX and Linux User Community


Interesting awk/Perl/sed parsing challenge


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Interesting awk/Perl/sed parsing challenge
# 1  
Old 08-23-2013
Interesting awk/Perl/sed parsing challenge

I have a log with entries like:

Code:
out/target/product/imx53_smd/obj/STATIC_LIBRARIES/libwebcore_intermediates/Source/WebCore/bindings/V8HTMLVideoElement.cpp
[22:14:58]: target thumb C++: libwebcore <= out/target/product/imx53_smd/obj/STATIC_LIBRARIES/libwebcore_intermediates/Source/WebCore/bindings/V8ImageData.cpp
[22:14:59]: target thumb C++: libwebcore <=

If you notice the time stamps between the brackets.. what I need to do is parse this whole log (tens of thousands of lines) for the biggest gaps between any two consecutive time stamps. The time spans about 3.5 hours so the minutes column will be reset back to 0. I just want to know where the biggest delays occur in this build.
# 2  
Old 08-23-2013
What have you tried so far? Did you search these fora for starting points?
# 3  
Old 08-23-2013
If you really must see my attempt, I tried it first in bash, but I have been working all day and am already a little burnt.

Code:
#!/bin/bash
set count=0
awk -F':' '{print $2}' inputfile|grep -Ev '[a-zA-Z]' |grep -v 00|
while read line
do 
  [ -n $last_line ] && echo $(( line - last_line ))
    if [[ ! $count == "0" ]]; then 
        set last_line=${line}
    fi
set count +=1
done

# 4  
Old 08-23-2013
Not sure if your awk will allow for a regex as field separator, but give this a shot:
Code:
awk -F"[][:]"   '               {TIME = $2*3600 + $3*60 + $4}
                 NR > 2         {DELTA=TIME-LAST; if (DELTA>MAX) {MAX=DELTA; LINEMAX=NR}}
                                {LAST=TIME}
                 END            {print LINEMAX, MAX}
                ' file

Will not, alas, span midnight...

Last edited by RudiC; 08-23-2013 at 05:11 PM.. Reason: midnight comment
# 5  
Old 08-23-2013
I'm getting an error like umatched ], what are you doing with that field separator? I don't quite understand it. My awk does accept a regex -F but it didnt like yours. you have "[][:]" which gives a fatal error, but "[ ][:]" does not give an error. What does this regex mean? Are [ and ] literals? Perhaps escape them with backslash?
# 6  
Old 08-23-2013
I'm telling awk to use "]", "[", ":" as field separators. You can use just ":" and then, as the first thing, remove all "[" and "]" from $0 with gsub (). You'll have to use $1 .. $3, then.
# 7  
Old 08-23-2013
use $1..$3 where?

Previous Thread | Next Thread
Test Your Knowledge in Computers #955
Difficulty: Medium
Modern antivirus programs cannot detect virus-patterns inside ordinary programs by scanning them for so-called virus signatures.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk/sed line parsing

I'm new to shell programming, but I think I learn best by following an example. I'm trying to cook up an awk/sed script, but I obviously lack the required syntax skills to achieve it. The output that I get from running my ksh script looks like this: I need to search each numbered line for... (10 Replies)
Discussion started by: iskatel
10 Replies

2. UNIX for Advanced & Expert Users

Parsing through a file with awk/sed

I don't necessary have a problem, as I have a solution. It is just that there may be a better solution. GOAL: Part one: Parse data from a file using the "\" as a delimiter and extracting only the last delimiter. Part two: Parse same file and extract everything but the last delimited item. ... (8 Replies)
Discussion started by: OrangeYaGlad
8 Replies

3. Linux

Interesting challenge getting SSH from outside to my VMWare guest

Hi, I'm working on getting more acquainted with VMWare and the SUSE SLES OS, so I've downloaded and created a guest running SLES 11 SP1. This works great, no problems. However, there's some funny-ness (is that a word?) when I'm attempting/testing to SSH from outside my local LAN into the guest.... (3 Replies)
Discussion started by: brightstorm
3 Replies

4. Shell Programming and Scripting

Parsing with awk or sed

I want to delete corrupt records from a file through awk or sed. Can anyone help me with this Thanks Striker Change subject to a descriptive one, ty. (1 Reply)
Discussion started by: Rahul_us
1 Replies

5. Shell Programming and Scripting

Line Parsing using sed and awk

Hi Guys, I need help with processing data in a file, line by line. My file test.txt has X_Building_X5946/X0 BUT/U_msp/RdBuMon_d2_B_00 BUT/U_msp/FfRmDaMix_d2_Pi3 Test_Long xp=849.416 yp=245.82 xn=849.488 yn=245.82 w=0.476 l=0.072 fault_layer="Al_T01_Mod" $ $X=849416 $Y=245582... (2 Replies)
Discussion started by: naveen@
2 Replies

6. Shell Programming and Scripting

awk/sed for parsing file

Hi All, I have a log file like this E Mon Oct 06 00:17:08 2008 xxx2 cm:10614 fm_pi2_svc_iptv_purchase.c:149 1:pin_deferred_act:10601:11:169:1223245028:16 pi2_op_svc_iptv_purchase error <location=PIN_ERRLOC_FM:5 class=PIN_ERRCLASS_SYSTEM_DETERMINATE:1... (10 Replies)
Discussion started by: subin_bala
10 Replies

7. Shell Programming and Scripting

Parsing a file (sed/awk?)

Hello people, newbie question. I'm trying to parse these type of file 1 "CAR " " C1 " " " 6 0 C1 2 "CAR " " O1A" " " 8 0 O1A 3 "CAR " " O1B" " " 8 -1 O1B 4 "CAR " " C2 " " " 6 0 C2 5 "CAR " " C3 " " " 6 ... (10 Replies)
Discussion started by: aristegui
10 Replies

8. Shell Programming and Scripting

parsing xml with awk/sed

Hi people!, I need extract from the file (test-file.txt) the values between <context> and </context> tag's , the total are 7 lines,but i can only get 5 or 2 lines!!:confused: Please look my code: #awk '/context/{flag=1} /\/context/{flag=0} !/context/{ if (flag==1) p rint $0; }'... (3 Replies)
Discussion started by: ricgamch
3 Replies

9. Shell Programming and Scripting

awk sed parsing

hi , i would like to parse some file with the fallowing data : data data data "unwanted data" data data "unwanted data" data data data data #unwanted data. what i want it to have any coments between "" and after # to be erased using awk or/and sed. has anyone an idea? thanks. (3 Replies)
Discussion started by: Darsh
3 Replies

10. UNIX for Dummies Questions & Answers

A challenge for you sed/awk wizards...

Here's a challenge for you wizards... I have a file formatted as follows; $ What I need to output is; 87654321 Bobby One 12345678 Bobby One 09876543 Bobby One 1107338 Bobby! Two Any Ideas how I can do this? I've tried sed but I'm not sure if perl might be a better way to... (2 Replies)
Discussion started by: th3g0bl1n
2 Replies

Featured Tech Videos