Shell Programming and Scripting

@ygemici: There is no fundamental difference between having a loop that you go through twice and two or more consecutive search and replace commands and now you come with one with 3 search and replace commands. I have been trying to explain that many times in this thread. One more example: I could write this:

Code:

sed -r ':a;s/(,|^)  *(,|$)/\1\2/g;ta'

or this:

Code:

sed -r 's/(,|^)  *(,|$)/\1\2/g;s/(,|^)  *(,|$)/\1\2/g'

It does not matter: they are practically equivalent. The first one is shorter though.

A loop saves pasting it in a second time, runtime versus very minor coding effort. Each pass is relatively expensive, so if 1 is not enough and 2 is enough, looping an extra time to 3 tries is silly (and you always have a dry pass). Actually, sometimes 2 is silly:

Code:

$ sed -r '
  s/(,|^)  *(,|$)/,,/g
  t n
  b
  :n
  s/(,|^)  *(,|$)/,,/g
 ' <<! | cat -e
aaa,   bbb,ccc   ,    ,    ,dddd
    ,ee  ee,   
!
aaa,   bbb,ccc   ,,,dddd$
,,ee  ee,,$
 
$

Of course, I have embraced some very fat data sets!

At the third time there are no further matches (dry pass as you call it) and so that is a relatively cheap pass. I found only a small performance difference between one or the other (<8%). Another consideration is that using the loop means short code that is easy to understand.

---------- Post updated at 21:22 ---------- Previous update was at 21:05 ----------

We could simplify the two statements since we can do the first field in the first run and the last field in the second run:

Code:

sed -r 's/(,|^)  *,/\1,/g;s/,  *(,|$)/,\1/g'

This runs 20% faster.

---------- Post updated at 21:30 ---------- Previous update was at 21:22 ----------

OK, this really is lots and lots faster; it only takes 1/5th of the time !!

Code:

sed -r 's/^  *,/,/;s/,  *,/,,/g;s/,  *,/,,/g;s/,  *$/,/'

but it is not because of the absence of the loop, since:

Code:

sed -r 's/^  *,/,/;:a;s/,  *,/,,/g;ta;s/,  *$/,/'

this is only 10% slower than this fast solution.

Obviously we can now drop the -r:

Code:

sed 's/^  *,/,/;s/,  *,/,,/g;s/,  *,/,,/g;s/,  *$/,/'

Code:

sed 's/^  *,/,/;:a;s/,  *,/,,/g;ta;s/,  *$/,/'

But that made no difference in performance.

==========
Apparently it is the lack of capturing groups, alternation and back references that makes all the difference!!
==========

I created some extra , with that last one. Yes, lack of iteration speeds things up! Does the t n b :n improve things over just two passes?

Code:

$ sed '
  s/^  *,/,/
  s/,  *$/,/
  s/,  *,/,,/g
  t n
  b
  :n
  s/,  *,/,,/g
 ' <<! |cat -e
aaa,   bbb,ccc   ,    ,    ,dddd
    ,ee  ee,   
!
aaa,   bbb,ccc   ,,,dddd$
,ee  ee,$ 
$

PS: How big did you expand the data set for the timings?

DG, not lack of iteration, but lack of capturing groups, alternation and back references is what is speeding things up dramatically is what I am finding.

The "t n b :n" bit in your suggestion is on par with my last part with the ta bit, i.e. 10% slower than without the loop.

I used 128K lines

Iteration in the alternation? Yes, the (|) looks like a time sponge. So, are back references implicitly slow?

Not even iteration in the alternation. The real culprit appears to be the use of capturing groups! To test this I used this:

Code:

sed -r 's/^  *,/,/;s/,  *,/,,/g;s/,  *,/,,/g;s/,  *$/,/'

and changed it to this:

Code:

sed -r 's/(^)  *,/,/;s/(,)  *,/,,/g;s/(,)  *,/,,/g;s/,  *($)/,/'

So only capturing groups without alternation or back references.
=======
The latter took 5x as long!!
=======