Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Parsing log with sed problem

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parsing log with sed problem
# 15  
Old 01-07-2011
The rest beside the grouping and the reversal, the biggest gain is matching against fixed strings, versus ranges of characters. Therefore Chublers code is fastest:
Code:
sed -n '/--SELECT/s/.*--//p;/--UPDATE/s/.*--//p;/--DELETE/s/.*--//p;/#STMT#/s/\].*//;/#STMT#/s/.*# \[//p' infile

This is the fastest I found so far that works with the sample provided:
Code:
sed -n '/Fine]/s/.*--//p;/#STMT#/{s/\].*//;s/.*# \[//p;}' infile

Last edited by Scrutinizer; 01-07-2011 at 06:12 PM..
# 16  
Old 01-07-2011
The first 's' can do the search once inside the replace. The second to third transition can be done without a search before with grouping:
Code:
 
sed -n 's/.*Fine].*--//p;/#STMT#/{
  s/\].*//
  s/.*# \[//p
 }
 ' infile
 
or even get the second search inside 's' and use a t to continue the trimming on hits:
 
sed -n '
  s/.*Fine].*--//p
  s/.* #STMT# //
  t x
  b
  :x
  s/\].*//p
 ' infile
 
Now, printing the pattern space does not delete the line, so to avoid considering the same line for both patterns:
 
sed -n '
  s/.*Fine].*--//
  t p
  s/.* #STMT# //
  t x
  b
  :x
  s/\].*//p
  b
  :p
  p
 ' infile

I guess it depends on whether branching is quick, but a lot of sed speed tuning is avoiding regex matching at the same data twice!

It's sad and amazing \(\) is so slow!
Last edited by DGPickett; 01-10-2011 at 11:20 AM..
# 17  
Old 01-07-2011
Ha I was thinking the same thing about the single search Smilie . Yes that grouping really has an impact, as we also found out some time ago..., but also character ranges/classes..
# 18  
Old 01-10-2011
Sometimes I put the sed processing in many piped together sed sessions, one line per session, so the load is distributed and I get to busy more CPUs and save time. But I also moved many slow things into simple C character-at-a-time or line-at-a-time state machine filters, like trimming white space from delimited fields. This runs orders faster than sed or awk, and the C program is still general, and so a very reusable shell accessory.

I edited in some additional potential speedup tweaks above.
Last edited by DGPickett; 01-10-2011 at 11:21 AM..
# 19  
Old 01-10-2011
Quote:
Originally Posted by Scrutinizer
The slowless is probably partly due to the grouping part, the part in parentheses... Does this make difference?
Code:
sed -n '/.*#STMT#/{s/[^[]*\[//;s/\].*//;p;};/--[^[:lower:]]*$/s/.*--//p' infile

reversing the two statements is faster:
Code:
sed -n '/--[^[:lower:]]*$/s/.*--//p;/.*#STMT#/{s/[^[]*\[//;s/\].*//;p;}' infile

Thank you. Yes, this runs faster, but again it would not finish 10,000 line log in 1 minute, I'm guessing usage of [:lower:] makes sed matching harder compared to a number of plain strings (not reg. expr) used in the original approach. Just guessing.
# 20  
Old 01-11-2011
Try combining the regex in front of the s into the one after the s, when possible, so it does not eval two regex.

What percentage of line match neither? A line-filtering sed or egrep up front can reduce the sed s load, if there are many lines entirely discarded.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parsing via sed issue

sorry I messed up the last post with too many mistakes and corrections so I closed it and opening a new one which should be clear to everyone .my apologies to the admins. I am using sun solaris and Linux , what I want is SED to print any string (or output it to a file preferably) that does... (2 Replies)
Discussion started by: boncuk
2 Replies

2. UNIX for Dummies Questions & Answers

sed or Grep Parsing

I would like to parse two strings from lines in a file only when both strings appear on the same line. For example, if I have the following line: string1 string2 string3 string4 string5 string6 string7 string8 string9 I would like the output to be: string2: string7 Can someone give me... (5 Replies)
Discussion started by: ARBlue79
5 Replies

3. Shell Programming and Scripting

[SED] Parsing to get a single value

Hello guys, I guess you are fed up with sed command and parse questions, but after a while researching the forum, I could not get an answer to my doubt. I know it must be easy done with sed command, but unfortunately, I never get right syntax of this command OK, this is what I have in my... (3 Replies)
Discussion started by: manolain
3 Replies

4. Shell Programming and Scripting

Sed special parsing

What is the shortest & right way to remove the string "" with a sed statement ? echo 'whateverwhatever' | sed ........ ? :) (2 Replies)
Discussion started by: ctsgnb
2 Replies

5. Shell Programming and Scripting

sed (parsing value)

All, Can somebody provide me with some sed expertise on how to parse the following line. 27-MAR-2011 10:28:01 * (CONNECT_DATA=(SID=dmart)(CID=(PROGRAM=sqlplus)(HOST=mtasnprod1)(USER=mtasnord))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.197.7.47)(PORT=54881)) * establish * dmart * 0 I would like... (3 Replies)
Discussion started by: BeefStu
3 Replies

6. Shell Programming and Scripting

Parsing cron with sed

Hello I want to convert my cron list into a csv Can you please help me with sed ? eg: Convert #06,21,36,51 * * 1,2 * (. ~/.profile ; timex /some/path/script -30 -15) >> /some/path/logfile2 2>&1 * * * * * (. ~/.profile ; timex /some/path/script2) > /some/path/logfile2 To:... (1 Reply)
Discussion started by: drbiloukos
1 Replies

7. Shell Programming and Scripting

Parsing with awk or sed

I want to delete corrupt records from a file through awk or sed. Can anyone help me with this Thanks Striker Change subject to a descriptive one, ty. (1 Reply)
Discussion started by: Rahul_us
1 Replies

8. Shell Programming and Scripting

Another parsing line awk or sed problem

Hi, After looking on different forums, I'm still in trouble to parse a parameters line received in KSH. $* is equal to "/AAA:111 /BBB:222 /CCC:333 /DDD:444" I would like to parse it and be able to access anyone from his name in my KSH after. like echo myArray => display 111 ... (1 Reply)
Discussion started by: RickTrader
1 Replies

9. Shell Programming and Scripting

Sed parsing error

I'm having a problem with a sed script. A programmer needs to change columns 942,943,944 to blank spaces only where it has the number 999 in every line. I didn't have a copy of the data file to test with originally so made my own up with a bunch of x's and put 999 in columns 5-7. The sed... (1 Reply)
Discussion started by: gravy26
1 Replies

10. Shell Programming and Scripting

awk sed parsing

hi , i would like to parse some file with the fallowing data : data data data "unwanted data" data data "unwanted data" data data data data #unwanted data. what i want it to have any coments between "" and after # to be erased using awk or/and sed. has anyone an idea? thanks. (3 Replies)
Discussion started by: Darsh
3 Replies
Login or Register to Ask a Question