Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

sed or awk editing help

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sed or awk editing help
# 29  
Old 11-02-2010
Quote:
Originally Posted by Scrutinizer
@ygemici: There is no fundamental difference between having a loop that you go through twice and two or more consecutive search and replace commands and now you come with one with 3 search and replace commands. I have been trying to explain that many times in this thread. One more example: I could write this:
Code:
sed -r ':a;s/(,|^)  *(,|$)/\1\2/g;ta'

or this:
Code:
sed -r 's/(,|^)  *(,|$)/\1\2/g;s/(,|^)  *(,|$)/\1\2/g'

It does not matter: they are practically equivalent. The first one is shorter though.
I know already this..But for me , important issue is correct results and at the same time how fast so not shorter..
# 30  
Old 11-02-2010
And you did gather from this thread that there is not a real speed difference between the two?
# 31  
Old 11-02-2010
I tried this with the big file like size 5-6 Mb..Smilie
As a result, each of two methods run properly Smilie

Thanks for your suggestions Smilie

Regards
ygemici - @sed lover
# 32  
Old 11-02-2010
Good, additionally we found a method that is 5x as fast:
Code:
sed 's/^  *,/,/;s/,  *,/,,/g;s/,  *,/,,/g;s/,  *$/,/'

It turns out this is not at all due to the lack of loops, but apparently because no grouping with ( ) or \( \) is being used.
# 33  
Old 11-02-2010
Quote:
Originally Posted by Scrutinizer
Good, additionally we found a method that is 5x as fast:
Code:
sed 's/^  *,/,/;s/,  *,/,,/g;s/,  *,/,,/g;s/,  *$/,/'

It turns out this is not at all due to the lack of loops, but apparently because no grouping with ( ) or \( \) is being used.
Of course this will more fast so because of in this not used buffers but we have compare --loop with group-- and --non-loop with group--

I tested it again..

with loop with group
Code:
real    0m4.140s
user    0m3.005s
sys     0m0.899s

with non-loop with group
Code:
real    0m3.851s
user    0m2.368s
sys     0m0.918s

with non-loop with non-group
Code:
real    0m2.184s
user    0m0.678s
sys     0m0.650s

regards
ygemici
# 34  
Old 11-02-2010
Which is consistent with my findings. There is not much difference between using several replace statement and using a loop, but there is a significant gain when groups are not being used.

You state this is obvious ("of course") why then didn't you use it in the first place, especially since your main concern is speed?

I do not think it was obvious at all and I cannot think of a good reason why grouping would have to take up so much time. I am glad I found out that it does though.
# 35  
Old 11-02-2010
Quote:
Originally Posted by Scrutinizer
Which is consistent with my findings. There is not much difference between using several replace statement and using a loop, but there is a significant gain when groups are not being used.

You state this is obvious ("of course") why then didn't you use it in the first place, especially since your main concern is speed?

I do not think it was obvious at all and I cannot think of a good reason why grouping would have to take up so much time. I am glad I found out that it does though.
Maybe the parsing Step, indeed, when ambiguous grouping is specified, it goes through a kind of "auto completion" step.
Also maybe using a memory copy instead of a memory mapping?.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Editing files with sed or something similar

{ "AFafa": "FAFA","AFafa": "FAFA" "baseball":"soccer","wrestling":"dancing" "rhinos":"crocodiles","roles":"foodchain" } I need to insert a new line before the closing brackets "}" so that the final output looks like this: { "AFafa": "FAFA","AFafa": "FAFA"... (6 Replies)
Discussion started by: SkySmart
6 Replies

2. Shell Programming and Scripting

editing file with awk cut and sed

HI All, I am new to unix. I have a file would like to do some editing by using awk, cut and sed. Could anyone help? This file contain 100 lines. There are one line for example: 2,"102343454",5060,"579668","579668","579668","SIP",,,"825922","035885221283026",1,268,"00:59:00.782 APR 17... (2 Replies)
Discussion started by: mimilaw
2 Replies

3. UNIX for Dummies Questions & Answers

sed help finding and editing

With sed 1. I need to find a line that contains "DVM" and "73069". 2. I need to insert a double quote at the beginning of the first line of the file. These two have been driving me crazy for the last 45 minutes. Any help would be greatly appreciated. Thanks (3 Replies)
Discussion started by: nlassiter
3 Replies

4. UNIX for Dummies Questions & Answers

sed editing help....

Hello all, I need some help with sed. seems like i cant get through it. So here is what i am trying. when i do ps -ef|grep bla blah ...like below...i get /u01/app/oracle/11g/bin/tnslsnr .... but i want to replace that string with something using sed. So basically i want to get rid of... (3 Replies)
Discussion started by: abdul.irfan2
3 Replies

5. Shell Programming and Scripting

Line/Variable Editing for Awk sed Cut

Hello, i have a file, i open the file and read the line, i want to get the first item in the csv file and also teh third+6 item and wirte it to a new csv file. only problem is that using echo it takes TOO LONG: please help a newbie. below is my code: WorkingDir=$1 FileName=`cut -d ',' -f... (2 Replies)
Discussion started by: limamichelle
2 Replies

6. Shell Programming and Scripting

Comparison and editing of files using awk.(And also a possible bug in awk for loop?)

I have two files which I would like to compare and then manipulate in a way. File1: pictures.txt 1.1 1.3 dance.txt 1.2 1.4 treehouse.txt 1.3 1.5 File2: pictures.txt 1.5 ref2313 1.4 ref2345 1.3 ref5432 1.2 ref4244 dance.txt 1.6 ref2342 1.5 ref2352 1.4 ref0695 1.3 ref5738 1.2... (1 Reply)
Discussion started by: linuxkid
1 Replies

7. Shell Programming and Scripting

problem in using sed command in editing a file

Hi all, I have a conf file, i want to update some entries in that conf file. Below is the code for that using a temporary file. sed '/workgroup=/ c\workgroup=Workgroup' /usr/local/netx.conf > /usr/local/netx.conf.tmp mv -f /usr/local/netx.conf.tmp /usr/local/netx.conf Sample contents of... (9 Replies)
Discussion started by: ranj14r
9 Replies

8. Homework & Coursework Questions

String editing using sed? awk?

1. The problem statement, all variables and given/known data: Problem Statement for project: When an account is created on the CS Unix network, a public html directory is created in the account's home directory. A default web page is put into that directory. Some users replace or... (13 Replies)
Discussion started by: peage1475
13 Replies

9. Shell Programming and Scripting

Editing Commas in a textfile using sed

Hi guys task removing the last commas of 5th and 6th columns. The bug in the script is causing effect because of whitespaces around commas. I tried to delete white spaces first and running the above script. but still some where getting the results wrong. I already have a script to do this... (12 Replies)
Discussion started by: repinementer
12 Replies

10. Shell Programming and Scripting

Editing File using awk/sed

Hello Awk Gurus, Can anyone of you help me with the below problem. I have got a file having data in below format pmFaultyTransportBlocks ----------------------- 9842993 pmFrmNoOfDiscRachFrames ----------------------- NULL pmNoRecRandomAccSuccess -----------------------... (4 Replies)
Discussion started by: Mohammed
4 Replies
Login or Register to Ask a Question